A Change That Does Nothing Should Score Zero. Mine Didn't.

Description and use

Control checks with known expected outcomes (e.g., zero for a no-op) uncovered hidden measurement bugs, ensuring the reliability of subsequent results.

Problem

Measurement pipelines can silently produce plausible but incorrect numbers, leading to unverified results.

Outcome

Implementing a simple control that verifies a no-op scores zero uncovered hidden measurement bugs, ensuring that subsequent results are trustworthy.

Workflow

• Run a pilot test to identify bugs in measurement code.
• Identify control cases where the expected outcome is known (e.g., no-op should score zero).
• Feed the control input to the pipeline and verify the output matches the expected value.
• If the output differs, investigate and fix measurement code.
• Repeat control checks before proceeding with the main experiment.
• Use controls to confirm instrument reliability throughout the study.
• Publish findings only after confirming controls pass.

Evidence

Build controls whose answer you already know.

Source

A change that does nothing should always score zero.

Source

Mine did not score zero. More than once.

Source

Before you trust a number about someone else's system, make your instrument prove it can produce a number it already knows.

Source