Provenance in the Primary Key: 54 Facts In, 49 Stored, and a Dedupe That Has Never Fired

Описание и применение

В системе анализа фактов ключ первичного ключа включает источник и локатор, из‑за чего дублирующие данные из разных сборщиков не объединяются, что приводит к неконсистентным результатам в графе и отчетах.

Задача

The Analysis Engine’s deduplication logic, based on a composite primary key that includes collector and locator, fails to merge duplicate facts from different collectors, leading to inconsistent data in downstream graph construction.

Результат

The deduplication logic never fired for cross‑collector duplicates because the primary key included collector and locator, causing each source to be treated as a separate fact. The graph layer’s merge logic inadvertently masked the issue, but the weaker data persisted in reports.

Рабочий процесс

• 1. Collect observations from repo collector.
• 2. Store observations with composite primary key.
• 3. Deduplicate by comparing confidence on key collisions.
• 4. Build SystemGraph by merging component observations by key.
• 5. Canonicalise provider resources into logical names.
• 6. Generate reports from the graph.

Доказательства

Line 82 of my Analysis Engine's fact base says it deduplicates and keeps the most confident copy.

Источник

The five collisions were all config rows for HTTP routes, each the same line of services/api/main.py reported twice by the same scan at the same confidence, 0.85 against 0.85.

Источник

For PostHog it returns the 0.60 observation from web/app.tsx:2, not the 0.75 one from web/package.json.

Источник

The analyzer builds a SystemGraph from the component observations, and add_component() merges by key: the first copy becomes the node, later copies fill in any attribute the node is missing, and their evidence locators are appended to a list.

Источник