9/9 Agent Tests Passed. The Trace Oracle Flagged 4 Behavior Changes

说明与用途

代理生成的测试可能通过，但隐藏的行为更改仍未被检测到。追踪或acular比较旧版和新版构建的跟踪，去除噪声，并标记未在允许列表中的差异，发现四个隐藏的回归。

问题

Agent‑written unit tests can pass even when the patch introduces unintended behavioral changes, because the tests are derived from the same diff and only check the new behavior.

结果

The trace oracle identified four silent behavior changes that the agent’s unit tests missed, preventing a regression from being merged.

工作流

• Generate property‑based seed corpus and freeze RNG seed
• Hash every fixture the binary touches to detect drift
• Run old build and capture golden.log
• Apply patch, run new build and capture patched.log
• Run trace_oracle.py to diff logs and classify deltas
• Merge only if flagged==0 and no fixture hash changed

证据

Agent-written tests are green because the patch wrote them. That is the entire problem in one sentence.

来源

The oracle still flags four silent behavior changes.

来源