9/9 Agent Tests Passed. The Trace Oracle Flagged 4 Behavior Changes

Description and use

Agent‑generated tests can pass while hidden behavioral changes remain undetected. The trace oracle compares old and new build traces, removes noise, and flags deltas not on an allowlist, uncovering four silent regressions.

Problem

Agent‑written unit tests can pass even when the patch introduces unintended behavioral changes, because the tests are derived from the same diff and only check the new behavior.

Outcome

The trace oracle identified four silent behavior changes that the agent’s unit tests missed, preventing a regression from being merged.

Workflow

• Generate property‑based seed corpus and freeze RNG seed
• Hash every fixture the binary touches to detect drift
• Run old build and capture golden.log
• Apply patch, run new build and capture patched.log
• Run trace_oracle.py to diff logs and classify deltas
• Merge only if flagged==0 and no fixture hash changed

Evidence

Agent-written tests are green because the patch wrote them. That is the entire problem in one sentence.

Source

The oracle still flags four silent behavior changes.

Source