Building AI Agent Memory for Prefect Workflows with Cognee and dlt

Description and use

Connecting Prefect to Cognee securely syncs orchestration metadata, giving AI agents instant context on flows and failures.

Problem

AI assistants lack contextual memory of orchestration topology, making debugging and failure analysis difficult.

Outcome

The connector securely syncs Prefect metadata into Cognee, enabling AI agents to query pipeline states, detect failures, and perform root‑cause analysis without leaking secrets.

Workflow

• Install cognee and connector via pip
• Configure PREFECT_API_URL (and PREFECT_API_KEY for Cloud)
• Create a prefect_source with desired resources and states
• Run cognee.remember to ingest data
• Query Cognee with cognee.search for insights

Evidence

Strict Zero-Secret Parameter Policy: The connector parses parameters on deployments and runs, strictly rendering parameter keys only (e.g. Parameters (keys only): batch_size, target_table), ensuring zero passwords or API keys are ever committed to memory.

Source

Merge Disposition with Tombstones: Configured with write_disposition="merge" and columns={"_deleted": {"data_type": "bool", "hard_delete": True}}. When an entity is deleted upstream, a tombstone row is emitted, enabling Cognee's orphan_cleanup to purge it from relational, graph, and vector stores without wiping the dataset.

Source

42 passed in 22.84s

Source