Fixing False Green Health Verdicts in Incident Agent

Description and use

Fixed a bug where the incident agent reported 'Healthy' status based on cached Fivetran evidence when live telemetry was unavailable. Enforced a rule that 'Healthy' status requires live evidence confirmation.

Problem

The incident investigation agent reported a pipeline as 'Healthy' based on cached Fivetran evidence when live telemetry was unavailable, leading to incorrect downstream investigation paths.

Outcome

Fixed a silent correctness bug where cached pipeline status incorrectly resulted in a 'Healthy' verdict. Implemented a strict invariant that 'Healthy' requires live evidence, preventing false positives in incident investigation.

Workflow

• Reproduce the false green verdict via /api/investigate endpoint.
• Add custom Sentry span 'pipeline.decision' to capture runtime state (evidence mode, live status, MCP status, verdict).
• Inspect Sentry trace to identify contradiction between non-live evidence and Healthy verdict.
• Use Seer to analyze the failing trace and identify missing decision constraint (evidence currentness).
• Verify hypothesis against source code: Healthy verdict should require live evidence, not just MCP success.
• Implement fix in evaluateFivetranHealth helper to enforce hasLiveEvidence check.
• Verify fix ensures cached evidence does not produce Healthy verdict.

Evidence

Current confidence requires current evidence.

Source

Healthy requires live evidence

Source

Pre-fix Sentry decision span showing cached Fivetran evidence, live evidence false, MCP unavailable, connected and on-schedule status, and a Healthy pipeline verdict

Source