I Built an Agent to Fix Bad Tests. I Found Eight Bugs in My Own Ruler.

Описание и применение

Разработчик создал агента, генерирующего тесты, которые убивают мутанты в Python-библиотеках, выявив восемь ошибок в измерительном инструменте.

Задача

Test suites often report high coverage but miss real bugs because they do not detect mutations. Mutation testing reveals this gap, but existing tools only report surviving mutants without guidance on fixing them.

Результат

The agent killed 9 of 15 mutants on two targets, outperforming baseline approaches. Eight bugs in the measurement instrument were identified, underscoring the need for rigorous instrument validation.

Рабочий процесс

• 1. Run mutation harness on target library to generate mutants and identify survivors.
• 2. Use the agent to generate tests with mutation diff context, execution gate, and retry logic.
• 3. Execute generated tests against clean code and mutants, capturing pytest output.
• 4. Gate tests: keep only those that pass on clean code and fail on mutants.
• 5. Repeat until desired kill rate or budget exhausted.
• 6. Validate results with determinism check (run each target three times serially).

Доказательства

The agent ran on 2 of 10 targets. The API budget ran out mid-run.

Nine retries fired and three succeeded.

Eight bugs in the instrument would have produced a confident, wrong, publishable number.