Flaky tests aren't random: a field guide to finding and killing them

Description and use

A step‑by‑step approach to identifying and eliminating flaky tests: quarantine, deterministic reproduction, root‑cause analysis, fix, and documentation.

Problem

Flaky tests pass and fail on the same code without changes, eroding confidence and leading teams to ignore failures.

Outcome

By systematically quarantining, reproducing, bisecting, and fixing flaky tests, teams reduce failure rates and prevent the "ignore red" habit.

Workflow

• Quarantine flaky tests to prevent pipeline blockage while tracking them
• Reproduce failures deterministically by running the test 50 times locally
• Bisect the cause by isolating the test, disabling network, or freezing time
• Fix the root cause and de‑quarantine the test, documenting the reason

Evidence

A flaky test is one that passes and fails on the same code with no change in between.

Source

Fix: never sleep for a fixed time. Wait for a condition — an element state, a network response, a value.

Source

Fix: each test sets up and tears down its own state. No reliance on execution order.

Source

Fix: isolate data per test (unique keys), and stub the network at the boundary.

Source

Fix: inject the clock and the seed. Freeze time (freezegun, sinon), pin the seed, pin the locale.

Source

Quarantine, don't ignore. Move the flaky test to a separate suite that doesn't block the pipeline — but tracks it.

Source

Reproduce deterministically. Run it 50 times locally (pytest --count=50, or loop it).

Source