Agent Patches Touch Your Fixtures Too: Fingerprint the Setup Layer Before You Freeze a Flaky Test

Описание и применение

Скрипт вычисляет контрольные суммы файлов настройки, а плагин Pytest фиксирует, какие тесты используют какие фикстуры, чтобы ревьюеры могли обнаружить изменения в настройке, которые иначе остались бы незамеченными.

Задача

Agent patches often modify setup files (fixtures, factories, conftest.py, golden data) without reviewers noticing, causing tests to pass while the underlying state changes.

Результат

The approach forces reviewers to examine changes in the setup layer, reducing flaky tests caused by unnoticed fixture modifications.

Рабочий процесс

• 1. Run audit/setup_fingerprint.py with --write to create audit/setup_baseline.json.
• 2. Commit the baseline to version control.
• 3. Run the script without --write during CI to compare current fingerprints against the baseline; exit non-zero if changes are detected.
• 4. Use audit/fixture_usage_plugin.py as a pytest plugin to collect which tests use which fixtures during a normal run.
• 5. Generate a fixture usage map and review any tests that depend on changed fixtures.
• 6. Optionally add property assertions on fixtures to ensure they meet expected invariants.

Доказательства

Most agent-patch review stops at the diff of src/ and tests/. That is one layer too high.

Источник

Run --write on the parent tree once, commit the baseline, and let the patch run the compare form. A non-zero exit means the review now has to look at setup files it would otherwise skim.

Источник