Giving AI Agents Lakehouse Memory: How We Built an Apache Iceberg Connector for Cognee

Описание и применение

Создан коннектор Apache Iceberg для Cognee, позволяющий ИИ-агентам хранить и использовать метаданные lakehouse без загрузки сырых данных, обеспечивая точную память и предотвращая «призрачные» знания.

Задача

AI agents lack semantic memory of enterprise lakehouses, leading to blind operations and data leakage when accessing raw table data.

Результат

The connector automatically discovers tables, parses metadata, populates a knowledge graph with 42 entities and 68 relations in 1.24s, and cleans orphaned nodes on table drop, enabling agents to query lakehouse memory without raw data leaks.

Рабочий процесс

• Configure Iceberg catalog source in Cognee
• Set document-mode marker on dlt source
• Run full-snapshot sync with write_disposition="replace"
• Cognee cognify() extracts entities and relations
• Orphan cleanup removes dropped tables
• Agents query lakehouse memory via graph completion

Доказательства

Agents are increasingly tasked with diagnosing failed ETL pipelines, performing data governance audits, and answering developer queries like…

Источник

The connector reads table metadata: Full table identifier, column definitions, partition specifications, snapshot commit history, and table properties…

Источник

Cognee's internal orphan_cleanup detects the vanished entity and reconciles it out of both the knowledge graph and vector indices.

Источник