Agent Patches Pass Their Own Tests. Here’s the Three-Layer Gate That Doesn’t Trust Them

Описание и применение

Три уровня проверки (свойства, ручные фикстуры, заморозка нестабильных тестов) устраняют корреляцию между патчами и тестами, генерируемыми одной моделью, и выявляют реальные ошибки до развертывания.

Задача

Agent-generated patches often pass tests written by the same model, so a green test suite only confirms self-consistency, not real correctness.

Результат

The three-layer gate effectively uncovers bugs that agent-generated tests miss, ensuring patches meet real-world specifications and production behavior.

Рабочий процесс

• 1. Replace example-based tests with property checks that assert invariants over a large input space.
• 2. Add curated fixtures derived from production incidents, written and reviewed by humans.
• 3. Implement a flaky freeze that removes unreliable tests from the signal until they are fixed or expire.
• 4. Run the gate cheaply before merging patches. If any layer fails, reject or iterate the patch.

Доказательства

Agent-generated patches pass their own tests for a boring reason: the same model wrote both sides.

Источник

Property checks replace examples with invariants. Instead of asserting that parse_csv_line("a,b") returns ["a", "b"], you state a rule that must hold for every input in a large generated space.

Источник

Curated fixtures… written by a human and reviewed on a schedule.

Источник