I Built an Agent to Fix Bad Tests. I Found Eight Bugs in My Own Ruler.

说明与用途

开发者构建了一个生成测试的代理，能够消除Python库中的突变体，并发现测量工具中的八个错误。

问题

Test suites often report high coverage but miss real bugs because they do not detect mutations. Mutation testing reveals this gap, but existing tools only report surviving mutants without guidance on fixing them.

结果

The agent killed 9 of 15 mutants on two targets, outperforming baseline approaches. Eight bugs in the measurement instrument were identified, underscoring the need for rigorous instrument validation.

工作流

• 1. Run mutation harness on target library to generate mutants and identify survivors.
• 2. Use the agent to generate tests with mutation diff context, execution gate, and retry logic.
• 3. Execute generated tests against clean code and mutants, capturing pytest output.
• 4. Gate tests: keep only those that pass on clean code and fail on mutants.
• 5. Repeat until desired kill rate or budget exhausted.
• 6. Validate results with determinism check (run each target three times serially).

证据

The agent ran on 2 of 10 targets. The API budget ran out mid-run.

Nine retries fired and three succeeded.

Eight bugs in the instrument would have produced a confident, wrong, publishable number.