Stop dividing by 4. I counted 951 texts and the rule is wrong both ways

Description and use

By replacing the common “length / 4” heuristic with a tokenizer or a more accurate estimator, the author reduced token count errors, prevented budget overruns, and avoided context truncation.

Problem

Estimating LLM token counts with the common rule of one token per four characters leads to inaccurate cost and context‑limit calculations, especially for JSON, CSV, and non‑English text.

Outcome

Replacing the “length / 4” heuristic with tokenizer‑based or category‑specific estimators reduces token count errors, prevents budget overruns, and avoids context‑limit truncations.

Workflow

• 1. Collected 951 text samples across categories (English blog, technical docs, Python, JSON, CSV, emoji, API keys, Chinese).
• 2. Tokenized each sample with OpenAI’s o200k_base tokenizer via tiktoken 0.14.0.
• 3. Calculated characters per token for each category and compared to “length / 4” predictions.
• 4. Built a custom estimator that counts words, digits, punctuation, and CJK characters separately, fitted to the samples.
• 5. Tested the estimator on 832 unseen files (README.md, package.json, index.js) and compared against “length / 4” and tokenx 2.2.0.
• 6. Evaluated accuracy: custom estimator undercounted package.json by 11.5%, tokenx within 1.4% on unseen files.
• 7. Recommended using the real tokenizer (gpt-tokenizer or js-tiktoken) on servers, tokenx or custom estimator in browsers with headroom for JSON/CSV/keys.
• 8. Provided guidance on dividing characters by 5 for plain English, 4.2 for code, 2 for JSON/CSV, 1.5 for keys, and 0.73 for Chinese.
• 9. Shared code and data on GitHub for community testing and improvement.

Evidence

Most token guides say one token is about four characters, and a lot of code runs Math.ceil(text.length / 4) right before an API call without anyone checking it.

Source

I ran 951 text samples through OpenAI's o200k_base tokenizer (GPT-4o and newer) with tiktoken 0.14.0.

Source

For plain English the rule guesses too high, so it errs on the safe side. For JSON, CSV, IDs and emoji it guesses about half.

Source

I built a better estimator, and it missed too

Source