Six Checks to Run Before You Trust a Number Your Own Harness Produced

Description and use

A six‑step checklist for measurement harnesses uncovered ten bugs, ensuring more reliable results.

Measurement harnesses can silently produce misleading results due to bugs that are not caught by code review.

The checklist uncovered ten bugs, improved harness reliability, and prevented biased measurement results.

1. Replace the target module with unparseable code and assert the harness fails (canary).
2. Run the scoring function three times and require byte‑identical output (determinism gate).
3. Create a vacuous test suite that should score near zero and assert it does (negative control).
4. Assert the actual property of interest rather than a proxy signal (property check).
5. Write a prediction of expected outcome before running diagnostics (prediction habit).
6. Install and run the harness in a fresh environment on an unrelated repository (external validation).

Replace the thing under test with something unparseable and assert that your harness notices.

Run the same scoring three times, serially, and require byte-identical output.

Build one case where the score should be near zero.