We audited 110 aiAI usage tools for metering bugs - 45+ reproduced, every finding linkable

Description and use

Audit of 110 AI metering tools uncovered 45+ bugs, 25 upstream fixes, and 477 green tests.

Problem

AI usage tools frequently mis‑meter usage, leading to silent accounting errors such as retries billed as new operations, duplicate records after crashes, and cache hits priced as fresh tokens.

Outcome

Audit identified 45+ reproducible metering bugs across 110 tools, leading to 25 upstream fixes and 477 green tests.

Workflow

• 1. Install agentmeasure via pip. 2. Run agentmeasure check against local session logs. 3. Inspect the output for duplicate records, retry chains, token anomalies, and missing evidence. 4. If a finding is flagged, use agentmeasure share to export sanitized summaries. 5. Submit a vector with agentmeasure conformance --new-vector <name> to pin semantics for CI. 6. Review audit index at https://roy-tong.github.io/AgentMeasure/claims.html for status and fixes.

Evidence

We kept running into the same class of silent accounting errors while metering AI agent usage — retries billed as new operations, duplicate records after crashes, cache hits priced as fresh tokens.

Source

The audit's vocabulary has five states and two of them are "we can't say": Verified Mismatch / Fixed · Re-verified / Conformance Pass / UNPROVABLE / Not-a-bug.

Source

Numbers as of 2026-10-14: 110 tools audited · 45+ reproduced metering bugs · 25 upstream fixes merged · 477 tests green.

Source