The Agent Rewrote Your Tests. Did the Contract Survive?

说明与用途

实施严格的 AI 生成测试审核工作流程，防止合同弱化并确保环境一致性。

问题

Automated agents often rewrite tests to match new code, potentially weakening or removing critical assertions, leading to false confidence in test suites and product contracts.

结果

Implementing this workflow prevents silent weakening of test contracts, ensures environment parity, and maintains confidence in test suite validity.

工作流

• 1. Open test diff before applying agent patch.
• 2. Verify production code changes and test file changes in the same commit.
• 3. Check that assertions are not weakened or removed (compare added vs removed assertions).
• 4. Run tests locally with set -euo pipefail python -m pytest -q --junitxml=junit.xml -s junit.xml and capture exit code and junit.xml.
• 5. Pin environment: run python -V uname -a echo "PY=$PYTHONPATH" git rev-parse HEAD pip freeze > /tmp/scratch-freeze.txt and compare with lockfile.
• 6. Inspect snapshot diffs: git diff origin/main -- '*.snap' '__snapshots__' '**/fixtures/**' and ensure human‑reviewed justification is present.
• 7. Use grep checks for assert True, assert 1, pytest.raises to detect empty or weak tests.
• 8. Merge only if all evidence is present and no contract is weakened.

证据

Treat agent-green tests as a hypothesis only.

来源

If removed is larger than added, pause.

来源