We audited our own agent's ledger: 10% of it was one test that never fails

Description and use

An audit of OpenAmer’s ledger revealed that 10 % of entries came from a single test that never fails, inflating the overall pass rate. Removing it lowered the true pass rate to 23.5 %, exposing frequent agent‑generated errors.

Problem

An AI agent’s self‑verification ledger was inflated by a trivial check that never fails, masking real failures.

Outcome

The audit showed that a single trivial check inflated the overall pass rate by 4.5 percentage points. After excluding it, the true pass rate dropped to 23.5 %, exposing frequent failures caused by the agent’s own code.

Workflow

• 1. Run OpenAmer and collect ledger entries as JSON lines.
• 2. Count rows, passed, and failed per goal using a short Python script.
• 3. Identify goals with 100 % pass rate and ≥20 rows.
• 4. Remove or flag trivial goals to obtain honest pass rates.
• 5. Analyze failure signatures (e.g., exit=1 Traceback, JSON decode errors).

Evidence

Measured just now (2026-10-13, 06:02 UTC), the ledger holds: rows (gate runs) 1,852, passed 579, failed 1,273, aggregate pass rate 31.3 %

Source

One goal owns 10 % of the ledger and has never failed

Source

The dominant failure mode of a self‑improving agent that writes its own verification code is its own generated code raising an exception

Source