A Change That Does Nothing Should Score Zero. Mine Didn't.

说明与用途

使用已知预期结果（例如无操作应为零）的控制检查发现隐藏的测量错误，确保后续结果的可靠性。

问题

Measurement pipelines can silently produce plausible but incorrect numbers, leading to unverified results.

结果

Implementing a simple control that verifies a no-op scores zero uncovered hidden measurement bugs, ensuring that subsequent results are trustworthy.

工作流

• Run a pilot test to identify bugs in measurement code.
• Identify control cases where the expected outcome is known (e.g., no-op should score zero).
• Feed the control input to the pipeline and verify the output matches the expected value.
• If the output differs, investigate and fix measurement code.
• Repeat control checks before proceeding with the main experiment.
• Use controls to confirm instrument reliability throughout the study.
• Publish findings only after confirming controls pass.

证据

Build controls whose answer you already know.

来源

A change that does nothing should always score zero.

来源

Mine did not score zero. More than once.

来源

Before you trust a number about someone else's system, make your instrument prove it can produce a number it already knows.

来源