The Assertion Got Weaker and the Build Went Green: Diffing Oracle Strength in Agent Patches

说明与用途

补丁可能会削弱测试断言，但CI仍然通过。使用静态分析脚本在补丁前后测量oracle强度，并阻止任何削弱。

问题

A patch can make a test weaker without failing the CI build, leading to silent regression.

结果

The workflow detects weakened assertions that would otherwise go unnoticed by CI, preventing silent regressions.

工作流

• 1. Run a static analysis script (oracle_ledger.py) on each test file to count assertions, comparisons, truthy-only checks, widest relative tolerance, broad exception raises, and skip markers.
• 2. Store the oracle metrics as a ledger entry per test.
• 3. After a patch, re‑run the script and compare new metrics to the stored ones.
• 4. If any metric indicates a weaker assertion (e.g., fewer comparisons, wider tolerance, broader exception), block the patch or raise a warning.
• 5. Review flagged tests manually to restore or strengthen assertions.

证据

A green build proves only that no assertion failed. It does not prove the assertions were still there, still compared anything, or still could fail.

来源

Below is a gate that measures oracle strength before and after a patch and blocks the weakening.

来源

The metric I use is deliberately crude and deterministic: how many assertions, how many of them are real comparisons, the widest numeric tolerance, how broad the expected exception is, and how many skip-family markers the test carries.

来源