tiktoken Undercounts Claude Tokens: 37 of 418 Requests Blew the Limit
说明与用途
使用 tiktoken 计数 Claude 令牌会低估 14%,导致请求失败和成本估算不准。解决方案是两步包装:先用惰性估算乘数快速本地估算,然后通过 Anthropic API 精确计数,必要时裁剪请求。
问题
Using tiktoken’s cl100k_base tokenizer to count Claude tokens caused a 14% undercount, leading to request failures and inaccurate cost estimates.
结果
The two‑pass packing approach eliminated all 37 request failures, aligned cost estimates with actual invoices, and provided a reliable method for token budgeting in Claude requests.
工作流
- 1. Sample 600 input chunks and compare tiktoken counts to Anthropic’s count_tokens to determine the median ratio (≈1.14).
- 2. Set a fast multiplier (FAST_RATIO=1.2) for the first local packing pass.
- 3. In the first pass, estimate token usage by multiplying tiktoken counts by FAST_RATIO and stop adding files when the estimate exceeds TARGET (194,000).
- 4. Assemble the final request with the selected files, system prompt, messages, and tool definitions.
- 5. Call Anthropic’s messages.count_tokens endpoint with the exact model, system prompt, messages, and tools to get the real input token count.
- 6. If the real count exceeds the 200K limit, trim the chosen files (e.g., remove the least relevant) and repeat step 5 until the request fits.
- 7. Send the final request to Claude.
- 8. Update cost dashboard to use the real token count for accurate billing.
证据
tiktoken undercounts Claude tokens because it implements OpenAI's BPE vocabularies, and Claude uses a different tokenizer that splits the same text into more pieces.
On my corpus it undercounted by a median of 14%: about 9% on prose, 16% on Python, 27% on JSON and lockfiles, and 38% on the worst single chunk.
Fix: do a cheap tiktoken pass with a multiplier, then verify the fully assembled request with Anthropic's free messages.count_tokens endpoint and trim until it fits.
- 记录日期: 2026-10-07T17:55:55.997Z