We audited 110 aiAI usage tools for metering bugs - 45+ reproduced, every finding linkable

Описание и применение

Проведён аудит 110 инструментов AI‑метеринга, выявлено 45+ ошибок, исправлено 25, 477 тестов успешно пройдены.

Задача

AI usage tools frequently mis‑meter usage, leading to silent accounting errors such as retries billed as new operations, duplicate records after crashes, and cache hits priced as fresh tokens.

Результат

Audit identified 45+ reproducible metering bugs across 110 tools, leading to 25 upstream fixes and 477 green tests.

Рабочий процесс

• 1. Install agentmeasure via pip. 2. Run agentmeasure check against local session logs. 3. Inspect the output for duplicate records, retry chains, token anomalies, and missing evidence. 4. If a finding is flagged, use agentmeasure share to export sanitized summaries. 5. Submit a vector with agentmeasure conformance --new-vector <name> to pin semantics for CI. 6. Review audit index at https://roy-tong.github.io/AgentMeasure/claims.html for status and fixes.

Доказательства

We kept running into the same class of silent accounting errors while metering AI agent usage — retries billed as new operations, duplicate records after crashes, cache hits priced as fresh tokens.

Источник

The audit's vocabulary has five states and two of them are "we can't say": Verified Mismatch / Fixed · Re-verified / Conformance Pass / UNPROVABLE / Not-a-bug.

Источник

Numbers as of 2026-10-14: 110 tools audited · 45+ reproduced metering bugs · 25 upstream fixes merged · 477 tests green.

Источник