The Assertion Got Weaker and the Build Went Green: Diffing Oracle Strength in Agent Patches

Description and use

A patch can weaken test assertions without causing CI failures. A static analysis script measures oracle strength before and after patches, blocking any weakening.

Problem

A patch can make a test weaker without failing the CI build, leading to silent regression.

Outcome

The workflow detects weakened assertions that would otherwise go unnoticed by CI, preventing silent regressions.

Workflow

• 1. Run a static analysis script (oracle_ledger.py) on each test file to count assertions, comparisons, truthy-only checks, widest relative tolerance, broad exception raises, and skip markers.
• 2. Store the oracle metrics as a ledger entry per test.
• 3. After a patch, re‑run the script and compare new metrics to the stored ones.
• 4. If any metric indicates a weaker assertion (e.g., fewer comparisons, wider tolerance, broader exception), block the patch or raise a warning.
• 5. Review flagged tests manually to restore or strengthen assertions.

Evidence

A green build proves only that no assertion failed. It does not prove the assertions were still there, still compared anything, or still could fail.

Source

Below is a gate that measures oracle strength before and after a patch and blocks the weakening.

Source

The metric I use is deliberately crude and deterministic: how many assertions, how many of them are real comparisons, the widest numeric tolerance, how broad the expected exception is, and how many skip-family markers the test carries.

Source