Giving AI Agents Pipeline Lineage Memory: How We Built an OpenLineage Connector for Cognee

说明与用途

AI 代理现在通过 OpenLineage 连接器将实时管道谱系信息导入 Cognee，从而加速数据故障诊断。

问题

AI agents lack operational lineage memory, preventing them from diagnosing data pipeline failures.

结果

The connector enabled AI agents to access real‑time pipeline lineage, reducing diagnosis time for data failures and preventing stale knowledge.

工作流

• 1. Query Marquez catalog for jobs, runs, datasets via REST API.
• 2. Generate high‑density Markdown projections summarizing job metadata and recent runs.
• 3. Ingest Markdown into Cognee using the dlt document mode, tagging source as "openlineage".
• 4. Cognee’s cognify engine extracts entities and relationships (READS_FROM, WRITES_TO, HAS_RUN).
• 5. Apply full‑snapshot replacement (write_disposition="replace") to avoid ghost knowledge.
• 6. Agents query the updated knowledge graph to diagnose failures.

证据

Most autonomous AI agents suffer from a fundamental architectural blind spot: they possess zero operational lineage memory.

来源

We tag the dlt source with Cognee's internal marker: ... setattr(source, DOCUMENT_SOURCE_ATTR, "openlineage")

来源

To solve this, our connector enforces a full‑snapshot replacement strategy: write_disposition="replace"

来源