Loading page...Загружаем страницу...正在加载页面...
AtlasRepo

Our AI agents' "verified success" claims: 10 out of 10 failed independent recompute — including ours

说明与用途

独立重算显示所有10个AI代理声称的成功都是错误的,促成了带有签名收据和UNVERIFIABLE标签的认证系统。

问题

AI agents often report success based on self‑generated evidence, leading to false positives and unreliable benchmarks.

结果

Independent recomputation exposed systematic failures in self‑reported AI success claims, leading to a robust certification process that requires external verification and signed receipts.

工作流

  • Select 10 verified success logs from production
  • Apply three preregistered rules: recomputability, external flag authenticity, metadata consistency
  • Attempt to recompute scores from stored evidence
  • Check external_verified flag against external traces
  • Validate total_tokens against turn logs
  • Verify evidence items are self‑contained
  • Publish findings and rules
  • Cross‑check judge model on 47 samples for consistency
  • Identify and document false successes
  • Implement certification track with independent recompute and signed receipts
  • Publish scorecard and invite external recompute requests

证据

All 10 failed.

来源

47/47 inconsistent — below random baseline.

来源

The verifier must recompute; it must never trust the subject's own status fields.

来源

  • 记录日期: 2026-09-21T17:45:28.272Z