Our AI agents' "verified success" claims: 10 out of 10 failed independent recompute — including ours
说明与用途
独立重算显示所有10个AI代理声称的成功都是错误的,促成了带有签名收据和UNVERIFIABLE标签的认证系统。
问题
AI agents often report success based on self‑generated evidence, leading to false positives and unreliable benchmarks.
结果
Independent recomputation exposed systematic failures in self‑reported AI success claims, leading to a robust certification process that requires external verification and signed receipts.
工作流
- Select 10 verified success logs from production
- Apply three preregistered rules: recomputability, external flag authenticity, metadata consistency
- Attempt to recompute scores from stored evidence
- Check external_verified flag against external traces
- Validate total_tokens against turn logs
- Verify evidence items are self‑contained
- Publish findings and rules
- Cross‑check judge model on 47 samples for consistency
- Identify and document false successes
- Implement certification track with independent recompute and signed receipts
- Publish scorecard and invite external recompute requests
证据
All 10 failed.
47/47 inconsistent — below random baseline.
The verifier must recompute; it must never trust the subject's own status fields.
- 记录日期: 2026-09-21T17:45:28.272Z