Giving AI Agents Pipeline Lineage Memory: How We Built an OpenLineage Connector for Cognee

Description and use

AI agents now access real‑time pipeline lineage via an OpenLineage connector to Cognee, speeding up data failure diagnostics.

Problem

AI agents lack operational lineage memory, preventing them from diagnosing data pipeline failures.

Outcome

The connector enabled AI agents to access real‑time pipeline lineage, reducing diagnosis time for data failures and preventing stale knowledge.

Workflow

• 1. Query Marquez catalog for jobs, runs, datasets via REST API.
• 2. Generate high‑density Markdown projections summarizing job metadata and recent runs.
• 3. Ingest Markdown into Cognee using the dlt document mode, tagging source as "openlineage".
• 4. Cognee’s cognify engine extracts entities and relationships (READS_FROM, WRITES_TO, HAS_RUN).
• 5. Apply full‑snapshot replacement (write_disposition="replace") to avoid ghost knowledge.
• 6. Agents query the updated knowledge graph to diagnose failures.

Evidence

Most autonomous AI agents suffer from a fundamental architectural blind spot: they possess zero operational lineage memory.

Source

We tag the dlt source with Cognee's internal marker: ... setattr(source, DOCUMENT_SOURCE_ATTR, "openlineage")

Source

To solve this, our connector enforces a full‑snapshot replacement strategy: write_disposition="replace"

Source