When to Refactor, Rebuild, or Delete a Broken Test Automation Suite

Description and use

By auditing and refactoring the test suite, the team reduced flakiness from ~33% to 0–1 failures per run, cut execution time by ~20%, and improved test coverage. The approach avoided costly rebuilds and maintained the existing framework.

Problem

A test automation suite is heavily flaky (≈33% red tests) and its reliability is unclear, leading to uncertainty about whether to refactor, rebuild, or delete parts of it.

Outcome

By auditing and refactoring the test suite, the team reduced flakiness from ~33% to 0–1 failures per run, cut execution time by ~20%, and improved test coverage. The approach avoided costly rebuilds and maintained the existing framework.

Workflow

• 1. Audit the suite across four layers: test design/data, framework currency, execution infrastructure, ownership.
• 2. Score each layer; one failing layer → refactor, two or more → rebuild (unless a single architectural failure dominates).
• 3. Run three probes to distinguish test‑design vs tool failures: single‑spec run, repeated concurrent runs, and run against a clean database.
• 4. If test design is the culprit, refactor tests: generate users per run, use API for data setup/teardown, remove hard‑coded URLs, enable retries.
• 5. If framework is outdated, upgrade to the latest supported version or migrate to a maintained framework.
• 6. If execution infrastructure limits parallelism or CI feedback, reconfigure pipelines, enable sharding, and add performance tests.
• 7. Assign ownership and SLAs for fixing failures to prevent decay.
• 8. Document changes and monitor flakiness metrics to confirm improvement.

Evidence

Cognota went from roughly 120 failures per regression run to 0–1 without changing the framework at all.

Source

QIMA is the same verdict at a larger scale — three engineers since 2021, on an outdated Cypress version with no performance autotests and release testing taking 3–4 days.

Source

The audit before you decide — the four layers

Source