Don't trust "Done": a two-model review gate for AI coding agents

Описание и применение

Двухмодельный контроль завершения кода, использующий контракт с хешем, выявил все реальные дефекты и предотвратил принятие неполных исправлений, потребовав всего пять минут времени рецензента за четыре раунда.

Задача

AI coding agents claim completion without reliable verification, leading to undetected bugs and false confidence.

Результат

The two-model gate successfully identified all real defects, prevented acceptance of incomplete fixes, and required only about five minutes of reviewer time across four rounds, with the author’s repair effort exceeding reviewer effort.

Рабочий процесс

• 1. Define binary acceptance criteria and not-done conditions in CONTRACT.md and lock with SHA-256 hash.
• 2. Author generates code and writes a completion report claiming DONE.
• 3. Reviewer model runs in read-only sandbox, computes contract hash, evaluates each criterion, cites file:line evidence, and lists findings.
• 4. Reviewer reports findings and overall verdict (DONE/NOT DONE).
• 5. Author addresses findings, updates code and completion report.
• 6. Reviewer re-evaluates in next round, focusing only on previously raised findings.
• 7. Repeat until reviewer returns DONE or escalation threshold is reached.

Доказательства

The reviewer runs with a read-only sandbox and no network.

Источник

The reviewer returned DONE. It did not take my word for the fix: it reported building in-memory tab lists for three cases...

Источник

Four review rounds plus the control came to roughly 790k tokens, about 77% of them cache hits...

Источник