Agent Patches Pass Their Own Tests. Here’s the Three-Layer Gate That Doesn’t Trust Them

说明与用途

三层门控（属性检查、人工策划的固定案例和不稳定测试冻结）打破了由同一模型生成的补丁和测试之间的相关性，在部署前捕获真实错误。

问题

Agent-generated patches often pass tests written by the same model, so a green test suite only confirms self-consistency, not real correctness.

结果

The three-layer gate effectively uncovers bugs that agent-generated tests miss, ensuring patches meet real-world specifications and production behavior.

工作流

• 1. Replace example-based tests with property checks that assert invariants over a large input space.
• 2. Add curated fixtures derived from production incidents, written and reviewed by humans.
• 3. Implement a flaky freeze that removes unreliable tests from the signal until they are fixed or expire.
• 4. Run the gate cheaply before merging patches. If any layer fails, reject or iterate the patch.

证据

Agent-generated patches pass their own tests for a boring reason: the same model wrote both sides.

来源

Property checks replace examples with invariants. Instead of asserting that parse_csv_line("a,b") returns ["a", "b"], you state a rule that must hold for every input in a large generated space.

来源

Curated fixtures… written by a human and reviewed on a schedule.

来源