tiktoken Undercounts Claude Tokens: 37 of 418 Requests Blew the Limit

Description and use

Using tiktoken to count Claude tokens underestimates by 14%, causing request failures and inaccurate cost estimates. The fix is a two‑step packing: a fast local estimate with a pessimistic multiplier, then a precise count via Anthropic’s API, trimming if needed.

Problem

Using tiktoken’s cl100k_base tokenizer to count Claude tokens caused a 14% undercount, leading to request failures and inaccurate cost estimates.

Outcome

The two‑pass packing approach eliminated all 37 request failures, aligned cost estimates with actual invoices, and provided a reliable method for token budgeting in Claude requests.

Workflow

• 1. Sample 600 input chunks and compare tiktoken counts to Anthropic’s count_tokens to determine the median ratio (≈1.14).
• 2. Set a fast multiplier (FAST_RATIO=1.2) for the first local packing pass.
• 3. In the first pass, estimate token usage by multiplying tiktoken counts by FAST_RATIO and stop adding files when the estimate exceeds TARGET (194,000).
• 4. Assemble the final request with the selected files, system prompt, messages, and tool definitions.
• 5. Call Anthropic’s messages.count_tokens endpoint with the exact model, system prompt, messages, and tools to get the real input token count.
• 6. If the real count exceeds the 200K limit, trim the chosen files (e.g., remove the least relevant) and repeat step 5 until the request fits.
• 7. Send the final request to Claude.
• 8. Update cost dashboard to use the real token count for accurate billing.

Evidence

tiktoken undercounts Claude tokens because it implements OpenAI's BPE vocabularies, and Claude uses a different tokenizer that splits the same text into more pieces.

Source

On my corpus it undercounted by a median of 14%: about 9% on prose, 16% on Python, 27% on JSON and lockfiles, and 38% on the worst single chunk.

Source

Fix: do a cheap tiktoken pass with a multiplier, then verify the fully assembled request with Anthropic's free messages.count_tokens endpoint and trim until it fits.

Source