Nine Bugs in My Own Evaluation Harness. Every One Made My Results Look Better.

Description and use

A developer found nine bugs in his evaluation harness that biased results favorably. By adding simple checks—a canary and a determinism gate—he uncovered hidden issues and improved the reliability of the metrics.

Problem

Evaluation harnesses can silently bias results by filtering out bugs that improve outcomes, leading to misleadingly favorable metrics.

Outcome

After adding canary and determinism checks, the harness produced more accurate mutation kill scores and identified previously hidden bugs, improving confidence in evaluation results.

Workflow

• 1. Build mutation‑testing harness and run suites.
• 2. Observe unexpectedly high kill scores.
• 3. Identify nine bugs that bias results.
• 4. Add canary check: overwrite source with garbage and assert failure.
• 5. Add determinism gate: run scoring thrice and compare byte‑identical output.
• 6. Re‑run harness; detect and fix hidden bugs.
• 7. Document checks and their intended guarantees.
• 8. Re‑evaluate metrics with checks in place.

Evidence

The canary passes happily while concurrency silently corrupts your results.

Source

Run the same scoring three times, serially, and require byte-identical output.

Source