Nine Bugs in My Own Evaluation Harness. Every One Made My Results Look Better.
Описание и применение
Разработчик обнаружил девять ошибок в своём инструменте оценки, которые искажали результаты в пользу более высоких метрик. Добавив простые проверки – канарейку и детерминированный шлюз – он смог выявить и исправить скрытые ошибки, повысив надёжность оценок.
Задача
Evaluation harnesses can silently bias results by filtering out bugs that improve outcomes, leading to misleadingly favorable metrics.
Результат
After adding canary and determinism checks, the harness produced more accurate mutation kill scores and identified previously hidden bugs, improving confidence in evaluation results.
Рабочий процесс
- 1. Build mutation‑testing harness and run suites.
- 2. Observe unexpectedly high kill scores.
- 3. Identify nine bugs that bias results.
- 4. Add canary check: overwrite source with garbage and assert failure.
- 5. Add determinism gate: run scoring thrice and compare byte‑identical output.
- 6. Re‑run harness; detect and fix hidden bugs.
- 7. Document checks and their intended guarantees.
- 8. Re‑evaluate metrics with checks in place.
Доказательства
The canary passes happily while concurrency silently corrupts your results.
Run the same scoring three times, serially, and require byte-identical output.
- Дата записи: 2026-09-15T20:45:24.441Z