Flaky tests aren't random: a field guide to finding and killing them

说明与用途

逐步识别和消除不稳定测试的流程：隔离、确定性重现、根本原因分析、修复和记录。

问题

Flaky tests pass and fail on the same code without changes, eroding confidence and leading teams to ignore failures.

结果

By systematically quarantining, reproducing, bisecting, and fixing flaky tests, teams reduce failure rates and prevent the "ignore red" habit.

工作流

• Quarantine flaky tests to prevent pipeline blockage while tracking them
• Reproduce failures deterministically by running the test 50 times locally
• Bisect the cause by isolating the test, disabling network, or freezing time
• Fix the root cause and de‑quarantine the test, documenting the reason

证据

A flaky test is one that passes and fails on the same code with no change in between.

来源

Fix: never sleep for a fixed time. Wait for a condition — an element state, a network response, a value.

来源

Fix: each test sets up and tears down its own state. No reliance on execution order.

来源

Fix: isolate data per test (unique keys), and stub the network at the boundary.

来源

Fix: inject the clock and the seed. Freeze time (freezegun, sinon), pin the seed, pin the locale.

来源

Quarantine, don't ignore. Move the flaky test to a separate suite that doesn't block the pipeline — but tracks it.

来源

Reproduce deterministically. Run it 50 times locally (pytest --count=50, or loop it).

来源