Bugs Are Innocent Until Reproduced: Building Verdict, an Evidence-First Agent Harness

Описание и применение

Verdict превращает проблему в воспроизводимую запись, используя доказательства и строгие пороги, чтобы уменьшить неопределённость при отладке непостоянных ошибок.

Задача

Flaky bug reports often end as "cannot reproduce" or patches that cannot be proven fixed, leaving maintainers uncertain about the true state of a defect.

Результат

Verdict provides a reproducible, evidence-based record of flaky bugs, enabling maintainers to verify fixes with deterministic thresholds and reducing uncertainty in bug triage.

Рабочий процесс

• Create a GitHub issue for the bug.
• Configure Verdict with approved commands, knobs, and budgets.
• Hunter searches the condition matrix and records all runs.
• Surgeon narrows the reproduced condition to the smallest suspect range.
• Insurance converts the reproduction into a regression plan and drafts a PR.
• Maintainer reviews the evidence and approves the PR.
• Verdict publishes the reproducible record and evidence hash.
• Reproduction can be recomputed with pnpm --filter @verdict/agent verify:runtime-evidence.
• CI runs the same verification steps to ensure consistency.

Доказательства

Verdict turns a GitHub issue into a bounded investigation. It runs an approved command repeatedly under approved conditions, keeps every observation and refuses to claim a reproduction unless the evidence crosses a deterministic threshold.

Источник

The verifier returns: {"verdict":"REPRODUCED",...}

Источник