We audited our own agent's ledger: 10% of it was one test that never fails

Описание и применение

Аудит леджера OpenAmer выявил, что 10 % записей принадлежит одной тестовой задаче, которая никогда не падает, завышая средний показатель прохождения. Удалив её, реальный процент прохождения снизился до 23,5 %, что показало частые ошибки в коде агента.

Задача

An AI agent’s self‑verification ledger was inflated by a trivial check that never fails, masking real failures.

Результат

The audit showed that a single trivial check inflated the overall pass rate by 4.5 percentage points. After excluding it, the true pass rate dropped to 23.5 %, exposing frequent failures caused by the agent’s own code.

Рабочий процесс

• 1. Run OpenAmer and collect ledger entries as JSON lines.
• 2. Count rows, passed, and failed per goal using a short Python script.
• 3. Identify goals with 100 % pass rate and ≥20 rows.
• 4. Remove or flag trivial goals to obtain honest pass rates.
• 5. Analyze failure signatures (e.g., exit=1 Traceback, JSON decode errors).

Доказательства

Measured just now (2026-10-13, 06:02 UTC), the ledger holds: rows (gate runs) 1,852, passed 579, failed 1,273, aggregate pass rate 31.3 %

Источник

One goal owns 10 % of the ledger and has never failed

Источник

The dominant failure mode of a self‑improving agent that writes its own verification code is its own generated code raising an exception

Источник