Bugs Are Innocent Until Reproduced: Building Verdict, an Evidence-First Agent Harness

Description and use

Verdict transforms a GitHub issue into a reproducible investigation, recording every run and producing verifiable evidence that can be independently checked.

Problem

Flaky bug reports often end as "cannot reproduce" or patches that cannot be proven fixed, leading to uncertainty and wasted effort.

Outcome

Verdict produced a reproducible record for TrueForge issue #417, demonstrating 10/10 failures under a stalled endpoint and 0/10 under a responsive endpoint, with a verifiable hash and a draft PR awaiting maintainer approval.

Workflow

• Create a GitHub issue for the bug
• Configure Verdict with approved commands, knobs, and budgets
• Hunter searches the condition matrix and records all runs
• Surgeon narrows the reproduced condition to the smallest suspect range
• Insurance converts the reproduction into a regression plan and draft PR
• Maintainer reviews the evidence and approves publication
• Verifier re-runs the evidence to confirm reproducibility

Evidence

Verdict turns a GitHub issue into a bounded investigation. It runs an approved command repeatedly under approved conditions, keeps every observation and refuses to claim a reproduction unless the evidence crosses a deterministic threshold.

Source

The verifier returns: { "verdict": "REPRODUCED", "stalledRuns": 10, "responsiveControls": 10, ... }

Source