Nine Bugs in My Own Evaluation Harness. Every One Made My Results Look Better.

说明与用途

开发者在评估工具中发现了九个偏向结果的错误。通过添加简单的检查——金丝雀和确定性门——他发现并修复了隐藏的缺陷，提高了评估指标的可靠性。

问题

Evaluation harnesses can silently bias results by filtering out bugs that improve outcomes, leading to misleadingly favorable metrics.

结果

After adding canary and determinism checks, the harness produced more accurate mutation kill scores and identified previously hidden bugs, improving confidence in evaluation results.

工作流

• 1. Build mutation‑testing harness and run suites.
• 2. Observe unexpectedly high kill scores.
• 3. Identify nine bugs that bias results.
• 4. Add canary check: overwrite source with garbage and assert failure.
• 5. Add determinism gate: run scoring thrice and compare byte‑identical output.
• 6. Re‑run harness; detect and fix hidden bugs.
• 7. Document checks and their intended guarantees.
• 8. Re‑evaluate metrics with checks in place.

证据

The canary passes happily while concurrency silently corrupts your results.

来源

Run the same scoring three times, serially, and require byte-identical output.

来源