Freeze a Flake Only After a Clean Runner Reproduces the Same Class

Description and use

In CI pipelines, flakes are frozen only after a clean runner reproduces the same failure class to avoid false positives from local environment bias.

Problem

Determining whether a flaky test failure should be frozen requires confirmation from a clean runner to avoid false positives caused by local environment bias.

Outcome

The workflow ensures that only genuine flakes are frozen, preventing deterministic bugs from being masked and avoiding stalls on known flakes.

Workflow

• Pin identities and required hashes before rerun; stop if any are missing.
• Classify the local failure as flake_shaped, deterministic, or environment; reject if not flake_shaped.
• Run the property cohort on unrelated inputs; a pass does not authorize a freeze.
• Start a clean runner that does not reuse the local workspace, replay the minimized input against the pinned fixture hash, and record remote class, fixture hash, input hash, and runner id.
• Compare local and remote data: require equal failure class, fixture hash, minimized-input hash, a passing property cohort, a completed clean run, and two distinct runner ids; write packet.json and decide eligibility.
• Expire the packet when oracle_version or patch_hash changes.

Evidence

A flake freeze is a recorded exception, not a pass.

Source

One local red bar is not enough. Agreement is the evidence; disagreement closes the freeze lane.

Source

The matrix below is a policy, not a measured benchmark.

Source