Agent Patches Touch Your Fixtures Too: Fingerprint the Setup Layer Before You Freeze a Flaky Test
说明与用途
脚本计算设置文件的 SHA256 摘要,pytest 插件记录 fixture 使用情况,帮助审查者发现并审查设置层的更改,避免被忽略。
问题
Agent patches often modify setup files (fixtures, factories, conftest.py, golden data) without reviewers noticing, causing tests to pass while the underlying state changes.
结果
The approach forces reviewers to examine changes in the setup layer, reducing flaky tests caused by unnoticed fixture modifications.
工作流
- 1. Run audit/setup_fingerprint.py with --write to create audit/setup_baseline.json.
- 2. Commit the baseline to version control.
- 3. Run the script without --write during CI to compare current fingerprints against the baseline; exit non-zero if changes are detected.
- 4. Use audit/fixture_usage_plugin.py as a pytest plugin to collect which tests use which fixtures during a normal run.
- 5. Generate a fixture usage map and review any tests that depend on changed fixtures.
- 6. Optionally add property assertions on fixtures to ensure they meet expected invariants.
证据
Most agent-patch review stops at the diff of src/ and tests/. That is one layer too high.
Run --write on the parent tree once, commit the baseline, and let the patch run the compare form. A non-zero exit means the review now has to look at setup files it would otherwise skim.
- 记录日期: 2026-10-17T05:41:05.458Z