Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

Our AI agents' "verified success" claims: 10 out of 10 failed independent recompute — including ours

Описание и применение

Независимый пересчёт показал, что все 10 заявленных успехов AI‑агентов были неверными, что привело к созданию системы сертификации с подписанными квитанциями и пометкой UNVERIFIABLE.

Задача

AI agents often report success based on self‑generated evidence, leading to false positives and unreliable benchmarks.

Результат

Independent recomputation exposed systematic failures in self‑reported AI success claims, leading to a robust certification process that requires external verification and signed receipts.

Рабочий процесс

  • Select 10 verified success logs from production
  • Apply three preregistered rules: recomputability, external flag authenticity, metadata consistency
  • Attempt to recompute scores from stored evidence
  • Check external_verified flag against external traces
  • Validate total_tokens against turn logs
  • Verify evidence items are self‑contained
  • Publish findings and rules
  • Cross‑check judge model on 47 samples for consistency
  • Identify and document false successes
  • Implement certification track with independent recompute and signed receipts
  • Publish scorecard and invite external recompute requests

Доказательства

All 10 failed.

Источник

47/47 inconsistent — below random baseline.

Источник

The verifier must recompute; it must never trust the subject's own status fields.

Источник

  • Дата записи: 2026-09-21T17:45:28.272Z