Stop dividing by 4. I counted 951 texts and the rule is wrong both ways

说明与用途

通过将常用的 “length / 4” 经验法替换为 tokenizer 或更精确的估算器，作者降低了 token 计数误差，防止了预算超支，并避免了上下文截断。

问题

Estimating LLM token counts with the common rule of one token per four characters leads to inaccurate cost and context‑limit calculations, especially for JSON, CSV, and non‑English text.

结果

Replacing the “length / 4” heuristic with tokenizer‑based or category‑specific estimators reduces token count errors, prevents budget overruns, and avoids context‑limit truncations.

工作流

• 1. Collected 951 text samples across categories (English blog, technical docs, Python, JSON, CSV, emoji, API keys, Chinese).
• 2. Tokenized each sample with OpenAI’s o200k_base tokenizer via tiktoken 0.14.0.
• 3. Calculated characters per token for each category and compared to “length / 4” predictions.
• 4. Built a custom estimator that counts words, digits, punctuation, and CJK characters separately, fitted to the samples.
• 5. Tested the estimator on 832 unseen files (README.md, package.json, index.js) and compared against “length / 4” and tokenx 2.2.0.
• 6. Evaluated accuracy: custom estimator undercounted package.json by 11.5%, tokenx within 1.4% on unseen files.
• 7. Recommended using the real tokenizer (gpt-tokenizer or js-tiktoken) on servers, tokenx or custom estimator in browsers with headroom for JSON/CSV/keys.
• 8. Provided guidance on dividing characters by 5 for plain English, 4.2 for code, 2 for JSON/CSV, 1.5 for keys, and 0.73 for Chinese.
• 9. Shared code and data on GitHub for community testing and improvement.

证据

Most token guides say one token is about four characters, and a lot of code runs Math.ceil(text.length / 4) right before an API call without anyone checking it.

来源

I ran 951 text samples through OpenAI's o200k_base tokenizer (GPT-4o and newer) with tiktoken 0.14.0.

来源

For plain English the rule guesses too high, so it errs on the safe side. For JSON, CSV, IDs and emoji it guesses about half.

来源

I built a better estimator, and it missed too

来源