One Run in Two Hundred: Debugging a Silent Fixture Fallback

说明与用途

CI 测试中一个隐藏的错误导致测试在读取 JSON 夹具时默默回退到默认值。通过显式错误处理和捕获日志，快速定位并修复了问题。

问题

A flaky test suite on CI silently fell back to default fixture values when a JSON fixture was partially overwritten, causing intermittent failures that were hard to detect.

结果

Flaky failures were exposed by making silent fallbacks loud and capturing artifacts, enabling quick diagnosis and resolution.

工作流

• 1. Count silent fallbacks by grepping for "except Exception" in code.
• 2. Refactor load_fixture to raise custom FixtureError on parse errors and log loudly.
• 3. Create a harness (flake_hunt.py) that runs tests multiple times with unique PYTHONHASHSEED and captures logs per attempt.
• 4. Run harness to collect 200 attempts, storing artifacts in .artifacts/flake.
• 5. Sort artifacts by size and analyze the smallest logs to identify the root cause.

证据

The failure rate went from 1 in 200 to 200 in 200. That is progress.

来源

DEBUG ingest.fixtures fixture unreadable, using defaults path=/tmp/fixture-shared.json

来源