We audited our own agent's ledger: 10% of it was one test that never fails

说明与用途

对 OpenAmer 的账本进行审计后发现，10% 的记录来自一个永不失败的测试，导致整体通过率被夸大。删除该测试后真实通过率降至 23.5%，暴露了代理自身生成错误的频率。

问题

An AI agent’s self‑verification ledger was inflated by a trivial check that never fails, masking real failures.

结果

The audit showed that a single trivial check inflated the overall pass rate by 4.5 percentage points. After excluding it, the true pass rate dropped to 23.5 %, exposing frequent failures caused by the agent’s own code.

工作流

• 1. Run OpenAmer and collect ledger entries as JSON lines.
• 2. Count rows, passed, and failed per goal using a short Python script.
• 3. Identify goals with 100 % pass rate and ≥20 rows.
• 4. Remove or flag trivial goals to obtain honest pass rates.
• 5. Analyze failure signatures (e.g., exit=1 Traceback, JSON decode errors).

证据

Measured just now (2026-10-13, 06:02 UTC), the ledger holds: rows (gate runs) 1,852, passed 579, failed 1,273, aggregate pass rate 31.3 %

来源

One goal owns 10 % of the ledger and has never failed

来源

The dominant failure mode of a self‑improving agent that writes its own verification code is its own generated code raising an exception

来源