One Run in Two Hundred: Debugging a Silent Fixture Fallback

Описание и применение

Случайная ошибка в тестах CI скрывалась, пока не было явно зафиксировано, что файл фикстуры не удалось прочитать. Сделав обработку ошибок явной и собирая логи, удалось быстро выявить и исправить проблему.

Задача

A flaky test suite on CI silently fell back to default fixture values when a JSON fixture was partially overwritten, causing intermittent failures that were hard to detect.

Результат

Flaky failures were exposed by making silent fallbacks loud and capturing artifacts, enabling quick diagnosis and resolution.

Рабочий процесс

• 1. Count silent fallbacks by grepping for "except Exception" in code.
• 2. Refactor load_fixture to raise custom FixtureError on parse errors and log loudly.
• 3. Create a harness (flake_hunt.py) that runs tests multiple times with unique PYTHONHASHSEED and captures logs per attempt.
• 4. Run harness to collect 200 attempts, storing artifacts in .artifacts/flake.
• 5. Sort artifacts by size and analyze the smallest logs to identify the root cause.

Доказательства

The failure rate went from 1 in 200 to 200 in 200. That is progress.

Источник

DEBUG ingest.fixtures fixture unreadable, using defaults path=/tmp/fixture-shared.json

Источник