I Built an Agent to Fix Bad Tests. I Found Eight Bugs in My Own Ruler.

Description and use

A developer built an agent that generates tests to kill mutants in Python libraries, uncovering eight bugs in the measurement instrument.

Problem

Test suites often report high coverage but miss real bugs because they do not detect mutations. Mutation testing reveals this gap, but existing tools only report surviving mutants without guidance on fixing them.

Outcome

The agent killed 9 of 15 mutants on two targets, outperforming baseline approaches. Eight bugs in the measurement instrument were identified, underscoring the need for rigorous instrument validation.

Workflow

• 1. Run mutation harness on target library to generate mutants and identify survivors.
• 2. Use the agent to generate tests with mutation diff context, execution gate, and retry logic.
• 3. Execute generated tests against clean code and mutants, capturing pytest output.
• 4. Gate tests: keep only those that pass on clean code and fail on mutants.
• 5. Repeat until desired kill rate or budget exhausted.
• 6. Validate results with determinism check (run each target three times serially).

Evidence

The agent ran on 2 of 10 targets. The API budget ran out mid-run.

Nine retries fired and three succeeded.

Eight bugs in the instrument would have produced a confident, wrong, publishable number.