Freeze a Flake Only After a Clean Runner Reproduces the Same Class

说明与用途

在 CI 流水线中，只有在干净的运行器重现相同的失败类时，才冻结 flaky 测试，以避免本地环境偏差导致的误报。

问题

Determining whether a flaky test failure should be frozen requires confirmation from a clean runner to avoid false positives caused by local environment bias.

结果

The workflow ensures that only genuine flakes are frozen, preventing deterministic bugs from being masked and avoiding stalls on known flakes.

工作流

• Pin identities and required hashes before rerun; stop if any are missing.
• Classify the local failure as flake_shaped, deterministic, or environment; reject if not flake_shaped.
• Run the property cohort on unrelated inputs; a pass does not authorize a freeze.
• Start a clean runner that does not reuse the local workspace, replay the minimized input against the pinned fixture hash, and record remote class, fixture hash, input hash, and runner id.
• Compare local and remote data: require equal failure class, fixture hash, minimized-input hash, a passing property cohort, a completed clean run, and two distinct runner ids; write packet.json and decide eligibility.
• Expire the packet when oracle_version or patch_hash changes.

证据

A flake freeze is a recorded exception, not a pass.

来源

One local red bar is not enough. Agreement is the evidence; disagreement closes the freeze lane.

来源

The matrix below is a policy, not a measured benchmark.

来源