The Agent Rewrote Your Tests. Did the Contract Survive?

Description and use

Implementing a strict workflow for AI‑generated test reviews protects against contract weakening and ensures environment consistency.

Problem

Automated agents often rewrite tests to match new code, potentially weakening or removing critical assertions, leading to false confidence in test suites and product contracts.

Outcome

Implementing this workflow prevents silent weakening of test contracts, ensures environment parity, and maintains confidence in test suite validity.

Workflow

• 1. Open test diff before applying agent patch.
• 2. Verify production code changes and test file changes in the same commit.
• 3. Check that assertions are not weakened or removed (compare added vs removed assertions).
• 4. Run tests locally with set -euo pipefail python -m pytest -q --junitxml=junit.xml -s junit.xml and capture exit code and junit.xml.
• 5. Pin environment: run python -V uname -a echo "PY=$PYTHONPATH" git rev-parse HEAD pip freeze > /tmp/scratch-freeze.txt and compare with lockfile.
• 6. Inspect snapshot diffs: git diff origin/main -- '*.snap' '__snapshots__' '**/fixtures/**' and ensure human‑reviewed justification is present.
• 7. Use grep checks for assert True, assert 1, pytest.raises to detect empty or weak tests.
• 8. Merge only if all evidence is present and no contract is weakened.

Evidence

Treat agent-green tests as a hypothesis only.

Source

If removed is larger than added, pause.

Source