Agent Patches Touch Your Fixtures Too: Fingerprint the Setup Layer Before You Freeze a Flaky Test

Description and use

A script computes SHA256 digests of setup files and a pytest plugin records fixture usage, enabling reviewers to detect and review changes to the setup layer that would otherwise be missed.

Problem

Agent patches often modify setup files (fixtures, factories, conftest.py, golden data) without reviewers noticing, causing tests to pass while the underlying state changes.

Outcome

The approach forces reviewers to examine changes in the setup layer, reducing flaky tests caused by unnoticed fixture modifications.

Workflow

• 1. Run audit/setup_fingerprint.py with --write to create audit/setup_baseline.json.
• 2. Commit the baseline to version control.
• 3. Run the script without --write during CI to compare current fingerprints against the baseline; exit non-zero if changes are detected.
• 4. Use audit/fixture_usage_plugin.py as a pytest plugin to collect which tests use which fixtures during a normal run.
• 5. Generate a fixture usage map and review any tests that depend on changed fixtures.
• 6. Optionally add property assertions on fixtures to ensure they meet expected invariants.

Evidence

Most agent-patch review stops at the diff of src/ and tests/. That is one layer too high.

Source

Run --write on the parent tree once, commit the baseline, and let the patch run the compare form. A non-zero exit means the review now has to look at setup files it would otherwise skim.

Source