Split the Flake Before You Freeze It

Описание и применение

Разделите флейки на три слоя и фиксируйте только хвост времени, сохраняя основной бизнес‑правило как обязательный тест.

Задача

Agent patches often make tests intermittent, but freezing the entire test hides the underlying business rule and envelope, leading to fragile CI.

Результат

Flaky tests are now split into deterministic core and freezeable timing tail, improving CI stability and test intent preservation.

Рабочий процесс

• Reproduce the flake once with a tight command to confirm intermittency.
• Classify the failing layer by reading the assertion: core, fixture, or timing tail.
• Extract the core into a separate property test to ensure determinism.
• Lock fixture inputs (hash or copy) and fail CI if agent commits change them without a ledger note.
• Write a freeze ledger entry only for the timing tail, including witness command, remaining property, and expiry.
• Fail the gate if a freeze marker lacks a live property test.
• Use a pytest hook to enforce that only timing tail freezes are allowed and that remaining_property is present.

Доказательства

A freeze on a whole test is a deleted oracle.

Источник

Freeze records how the flake was produced, under which patch, and which property remains in the gate.

Источник