Fixing False Green Health Verdicts in Incident Agent

Описание и применение

Исправлена ошибка, при которой агент инцидентов сообщал о состоянии 'Здорово' на основе кэшированных данных Fivetran при отсутствии живых телеметрических данных. Внедрено правило: статус 'Здорово' требует подтверждения живых данных.

Задача

The incident investigation agent reported a pipeline as 'Healthy' based on cached Fivetran evidence when live telemetry was unavailable, leading to incorrect downstream investigation paths.

Результат

Fixed a silent correctness bug where cached pipeline status incorrectly resulted in a 'Healthy' verdict. Implemented a strict invariant that 'Healthy' requires live evidence, preventing false positives in incident investigation.

Рабочий процесс

• Reproduce the false green verdict via /api/investigate endpoint.
• Add custom Sentry span 'pipeline.decision' to capture runtime state (evidence mode, live status, MCP status, verdict).
• Inspect Sentry trace to identify contradiction between non-live evidence and Healthy verdict.
• Use Seer to analyze the failing trace and identify missing decision constraint (evidence currentness).
• Verify hypothesis against source code: Healthy verdict should require live evidence, not just MCP success.
• Implement fix in evaluateFivetranHealth helper to enforce hasLiveEvidence check.
• Verify fix ensures cached evidence does not produce Healthy verdict.

Доказательства

Current confidence requires current evidence.

Источник

Healthy requires live evidence

Источник

Pre-fix Sentry decision span showing cached Fivetran evidence, live evidence false, MCP unavailable, connected and on-schedule status, and a Healthy pipeline verdict

Источник