Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

Two Readers Broke My Checks. Not One of My Numbers Moved.

Описание и применение

Разработчик обнаружил скрытые ошибки в системе измерения покрытия мутаций, исправил их, добавив байтовые канарейки и отрицательный контроль, и повысил точность результатов до 95,7 %.

Задача

The evaluation harness for measuring test suite mutation coverage contained hidden bugs that allowed stale bytecode to be used, leading to false positives and unreliable metrics.

Результат

The harness now accurately reports mutation coverage, with a 95.7% kill rate, and includes safeguards against stale bytecode and silent failures.

Рабочий процесс

  • 1. Identify mutation bug: stale .pyc usage due to CPython mtime granularity.
  • 2. Add byte‑size‑preserving canary to detect stale bytecode.
  • 3. Verify no .pyc files are copied into temp directories.
  • 4. Add negative control suite of vacuous tests expecting near‑zero score.
  • 5. Detect and fix proxy test that relied on pyc count.
  • 6. Re‑write test to assert observed outcome rather than proxy.
  • 7. Document load‑bearing safeguards and environment variable handling.
  • 8. Re‑run full mutation test suite and confirm kill rate of 95.7%.
  • 9. Publish harness and results for community scrutiny.

Доказательства

I added a byte-size-preserving canary alongside the original. Eight of twelve targets are eligible and all pass.

Источник

It killed 7 of 51 mutants. 13.7%.

Источник

  • Дата записи: 2026-09-17T00:30:20.738Z