The Agent Rewrote Your Tests. Did the Contract Survive?

Описание и применение

Внедрение строгого рабочего процесса проверки тестов, генерируемых ИИ‑агентом, защищает от ослабления контрактов и обеспечивает согласованность окружения.

Задача

Automated agents often rewrite tests to match new code, potentially weakening or removing critical assertions, leading to false confidence in test suites and product contracts.

Результат

Implementing this workflow prevents silent weakening of test contracts, ensures environment parity, and maintains confidence in test suite validity.

Рабочий процесс

• 1. Open test diff before applying agent patch.
• 2. Verify production code changes and test file changes in the same commit.
• 3. Check that assertions are not weakened or removed (compare added vs removed assertions).
• 4. Run tests locally with set -euo pipefail python -m pytest -q --junitxml=junit.xml -s junit.xml and capture exit code and junit.xml.
• 5. Pin environment: run python -V uname -a echo "PY=$PYTHONPATH" git rev-parse HEAD pip freeze > /tmp/scratch-freeze.txt and compare with lockfile.
• 6. Inspect snapshot diffs: git diff origin/main -- '*.snap' '__snapshots__' '**/fixtures/**' and ensure human‑reviewed justification is present.
• 7. Use grep checks for assert True, assert 1, pytest.raises to detect empty or weak tests.
• 8. Merge only if all evidence is present and no contract is weakened.

Доказательства

Treat agent-green tests as a hypothesis only.

Источник

If removed is larger than added, pause.

Источник