Don't trust "Done": a two-model review gate for AI coding agents

Description and use

A two-model completion gate using a hash-locked contract identified all real defects and prevented acceptance of incomplete fixes, requiring only five minutes of reviewer time over four rounds.

Problem

AI coding agents claim completion without reliable verification, leading to undetected bugs and false confidence.

Outcome

The two-model gate successfully identified all real defects, prevented acceptance of incomplete fixes, and required only about five minutes of reviewer time across four rounds, with the author’s repair effort exceeding reviewer effort.

Workflow

• 1. Define binary acceptance criteria and not-done conditions in CONTRACT.md and lock with SHA-256 hash.
• 2. Author generates code and writes a completion report claiming DONE.
• 3. Reviewer model runs in read-only sandbox, computes contract hash, evaluates each criterion, cites file:line evidence, and lists findings.
• 4. Reviewer reports findings and overall verdict (DONE/NOT DONE).
• 5. Author addresses findings, updates code and completion report.
• 6. Reviewer re-evaluates in next round, focusing only on previously raised findings.
• 7. Repeat until reviewer returns DONE or escalation threshold is reached.

Evidence

The reviewer runs with a read-only sandbox and no network.

Source

The reviewer returned DONE. It did not take my word for the fix: it reported building in-memory tab lists for three cases...

Source

Four review rounds plus the control came to roughly 790k tokens, about 77% of them cache hits...

Source