We audited 110 aiAI usage tools for metering bugs - 45+ reproduced, every finding linkable

说明与用途

对110个AI计费工具进行审计，发现45+错误，修复25个，477个测试通过。

问题

AI usage tools frequently mis‑meter usage, leading to silent accounting errors such as retries billed as new operations, duplicate records after crashes, and cache hits priced as fresh tokens.

结果

Audit identified 45+ reproducible metering bugs across 110 tools, leading to 25 upstream fixes and 477 green tests.

工作流

• 1. Install agentmeasure via pip. 2. Run agentmeasure check against local session logs. 3. Inspect the output for duplicate records, retry chains, token anomalies, and missing evidence. 4. If a finding is flagged, use agentmeasure share to export sanitized summaries. 5. Submit a vector with agentmeasure conformance --new-vector <name> to pin semantics for CI. 6. Review audit index at https://roy-tong.github.io/AgentMeasure/claims.html for status and fixes.

证据

We kept running into the same class of silent accounting errors while metering AI agent usage — retries billed as new operations, duplicate records after crashes, cache hits priced as fresh tokens.

来源

The audit's vocabulary has five states and two of them are "we can't say": Verified Mismatch / Fixed · Re-verified / Conformance Pass / UNPROVABLE / Not-a-bug.

来源

Numbers as of 2026-10-14: 110 tools audited · 45+ reproduced metering bugs · 25 upstream fixes merged · 477 tests green.

来源