Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

Giving AI Agents Pipeline Lineage Memory: How We Built an OpenLineage Connector for Cognee

Описание и применение

AI‑агенты теперь могут получать оперативную информацию о пайплайнах, используя OpenLineage‑коннектор к Cognee, что ускоряет диагностику сбоев данных.

Задача

AI agents lack operational lineage memory, preventing them from diagnosing data pipeline failures.

Результат

The connector enabled AI agents to access real‑time pipeline lineage, reducing diagnosis time for data failures and preventing stale knowledge.

Рабочий процесс

  • 1. Query Marquez catalog for jobs, runs, datasets via REST API.
  • 2. Generate high‑density Markdown projections summarizing job metadata and recent runs.
  • 3. Ingest Markdown into Cognee using the dlt document mode, tagging source as "openlineage".
  • 4. Cognee’s cognify engine extracts entities and relationships (READS_FROM, WRITES_TO, HAS_RUN).
  • 5. Apply full‑snapshot replacement (write_disposition="replace") to avoid ghost knowledge.
  • 6. Agents query the updated knowledge graph to diagnose failures.

Доказательства

Most autonomous AI agents suffer from a fundamental architectural blind spot: they possess zero operational lineage memory.

Источник

We tag the dlt source with Cognee's internal marker: ... setattr(source, DOCUMENT_SOURCE_ATTR, "openlineage")

Источник

To solve this, our connector enforces a full‑snapshot replacement strategy: write_disposition="replace"

Источник

  • Дата записи: 2026-10-09T22:41:12.786Z