Giving AI Agents Lakehouse Memory: How We Built an Apache Iceberg Connector for Cognee

Description and use

Built an Apache Iceberg connector for Cognee that ingests lakehouse metadata, builds a knowledge graph, and enables AI agents to query lakehouse memory without raw data leaks, while preventing ghost knowledge.

Problem

AI agents lack semantic memory of enterprise lakehouses, leading to blind operations and data leakage when accessing raw table data.

Outcome

The connector automatically discovers tables, parses metadata, populates a knowledge graph with 42 entities and 68 relations in 1.24s, and cleans orphaned nodes on table drop, enabling agents to query lakehouse memory without raw data leaks.

Workflow

• Configure Iceberg catalog source in Cognee
• Set document-mode marker on dlt source
• Run full-snapshot sync with write_disposition="replace"
• Cognee cognify() extracts entities and relations
• Orphan cleanup removes dropped tables
• Agents query lakehouse memory via graph completion

Evidence

Agents are increasingly tasked with diagnosing failed ETL pipelines, performing data governance audits, and answering developer queries like…

Source

The connector reads table metadata: Full table identifier, column definitions, partition specifications, snapshot commit history, and table properties…

Source

Cognee's internal orphan_cleanup detects the vanished entity and reconciles it out of both the knowledge graph and vector indices.

Source