Provenance in the Primary Key: 54 Facts In, 49 Stored, and a Dedupe That Has Never Fired

Description and use

The Analysis Engine’s primary key includes collector and locator, preventing cross‑collector deduplication and causing inconsistent data in downstream graph and reports.

Problem

The Analysis Engine’s deduplication logic, based on a composite primary key that includes collector and locator, fails to merge duplicate facts from different collectors, leading to inconsistent data in downstream graph construction.

Outcome

The deduplication logic never fired for cross‑collector duplicates because the primary key included collector and locator, causing each source to be treated as a separate fact. The graph layer’s merge logic inadvertently masked the issue, but the weaker data persisted in reports.

Workflow

• 1. Collect observations from repo collector.
• 2. Store observations with composite primary key.
• 3. Deduplicate by comparing confidence on key collisions.
• 4. Build SystemGraph by merging component observations by key.
• 5. Canonicalise provider resources into logical names.
• 6. Generate reports from the graph.

Evidence

Line 82 of my Analysis Engine's fact base says it deduplicates and keeps the most confident copy.

Source

The five collisions were all config rows for HTTP routes, each the same line of services/api/main.py reported twice by the same scan at the same confidence, 0.85 against 0.85.

Source

For PostHog it returns the 0.60 observation from web/app.tsx:2, not the 0.75 one from web/package.json.

Source

The analyzer builds a SystemGraph from the component observations, and add_component() merges by key: the first copy becomes the node, later copies fill in any attribute the node is missing, and their evidence locators are appended to a list.

Source