Agent Patches Pass Their Own Tests. Here’s the Three-Layer Gate That Doesn’t Trust Them

Description and use

A three-layer gate—property checks, curated fixtures, and a flaky freeze—breaks the correlation between agent-generated patches and tests, catching real faults before deployment.

Problem

Agent-generated patches often pass tests written by the same model, so a green test suite only confirms self-consistency, not real correctness.

Outcome

The three-layer gate effectively uncovers bugs that agent-generated tests miss, ensuring patches meet real-world specifications and production behavior.

Workflow

• 1. Replace example-based tests with property checks that assert invariants over a large input space.
• 2. Add curated fixtures derived from production incidents, written and reviewed by humans.
• 3. Implement a flaky freeze that removes unreliable tests from the signal until they are fixed or expire.
• 4. Run the gate cheaply before merging patches. If any layer fails, reject or iterate the patch.

Evidence

Agent-generated patches pass their own tests for a boring reason: the same model wrote both sides.

Source

Property checks replace examples with invariants. Instead of asserting that parse_csv_line("a,b") returns ["a", "b"], you state a rule that must hold for every input in a large generated space.

Source

Curated fixtures… written by a human and reviewed on a schedule.

Source