Bugs Are Innocent Until Reproduced: Building Verdict, an Evidence-First Agent Harness

Description and use

Verdict turns a bug into a reproducible record, using evidence and strict thresholds to reduce uncertainty in debugging flaky issues.

Problem

Flaky bug reports often end as "cannot reproduce" or patches that cannot be proven fixed, leaving maintainers uncertain about the true state of a defect.

Outcome

Verdict provides a reproducible, evidence-based record of flaky bugs, enabling maintainers to verify fixes with deterministic thresholds and reducing uncertainty in bug triage.

Workflow

• Create a GitHub issue for the bug.
• Configure Verdict with approved commands, knobs, and budgets.
• Hunter searches the condition matrix and records all runs.
• Surgeon narrows the reproduced condition to the smallest suspect range.
• Insurance converts the reproduction into a regression plan and drafts a PR.
• Maintainer reviews the evidence and approves the PR.
• Verdict publishes the reproducible record and evidence hash.
• Reproduction can be recomputed with pnpm --filter @verdict/agent verify:runtime-evidence.
• CI runs the same verification steps to ensure consistency.

Evidence

Verdict turns a GitHub issue into a bounded investigation. It runs an approved command repeatedly under approved conditions, keeps every observation and refuses to claim a reproduction unless the evidence crosses a deterministic threshold.

Source

The verifier returns: {"verdict":"REPRODUCED",...}

Source