The Assertion Got Weaker and the Build Went Green: Diffing Oracle Strength in Agent Patches

Описание и применение

Патч может ослабить утверждения в тестах, но CI всё равно проходит. Используется статический анализ, чтобы измерить «силу оракла» тестов до и после патча и блокировать ослабление.

Задача

A patch can make a test weaker without failing the CI build, leading to silent regression.

Результат

The workflow detects weakened assertions that would otherwise go unnoticed by CI, preventing silent regressions.

Рабочий процесс

• 1. Run a static analysis script (oracle_ledger.py) on each test file to count assertions, comparisons, truthy-only checks, widest relative tolerance, broad exception raises, and skip markers.
• 2. Store the oracle metrics as a ledger entry per test.
• 3. After a patch, re‑run the script and compare new metrics to the stored ones.
• 4. If any metric indicates a weaker assertion (e.g., fewer comparisons, wider tolerance, broader exception), block the patch or raise a warning.
• 5. Review flagged tests manually to restore or strengthen assertions.

Доказательства

A green build proves only that no assertion failed. It does not prove the assertions were still there, still compared anything, or still could fail.

Источник

Below is a gate that measures oracle strength before and after a patch and blocks the weakening.

Источник

The metric I use is deliberately crude and deterministic: how many assertions, how many of them are real comparisons, the widest numeric tolerance, how broad the expected exception is, and how many skip-family markers the test carries.

Источник