Two Readers Broke My Checks. Not One of My Numbers Moved.

Description and use

A developer uncovered hidden bugs in a mutation‑testing harness, fixed them with byte‑size canaries and a negative control, and improved coverage accuracy to 95.7 %.

Problem

The evaluation harness for measuring test suite mutation coverage contained hidden bugs that allowed stale bytecode to be used, leading to false positives and unreliable metrics.

Outcome

The harness now accurately reports mutation coverage, with a 95.7% kill rate, and includes safeguards against stale bytecode and silent failures.

Workflow

• 1. Identify mutation bug: stale .pyc usage due to CPython mtime granularity.
• 2. Add byte‑size‑preserving canary to detect stale bytecode.
• 3. Verify no .pyc files are copied into temp directories.
• 4. Add negative control suite of vacuous tests expecting near‑zero score.
• 5. Detect and fix proxy test that relied on pyc count.
• 6. Re‑write test to assert observed outcome rather than proxy.
• 7. Document load‑bearing safeguards and environment variable handling.
• 8. Re‑run full mutation test suite and confirm kill rate of 95.7%.
• 9. Publish harness and results for community scrutiny.

Evidence

I added a byte-size-preserving canary alongside the original. Eight of twelve targets are eligible and all pass.

Source

It killed 7 of 51 mutants. 13.7%.

Source