The 4-Characters-Per-Token Rule Under-Counts Real Code by 27%

Description and use

Roomy provides accurate token budgeting for code prompts, preventing silent truncation and ensuring required files are included or an explicit error is raised.

Problem

Token counting for code prompts is inaccurate when using the 4‑characters‑per‑token rule, leading to prompt truncation or silent failures when budgets are exceeded.

Outcome

Roomy provides accurate token budgeting for code prompts, avoiding silent truncation and ensuring required files are included or an explicit error is raised. It outperforms the 4‑chars‑per‑token rule by 27% on a real repository and achieves 82.8% of optimal packing at 6000 tokens.

Workflow

• 1. Define a token budget and reserve for the model reply.
• 2. Provide a list of candidate files with priorities and sizes.
• 3. Run roomy.pack() to select files that fit within the usable budget.
• 4. Inspect the plan: included files, excluded files, and reasons for exclusion.
• 5. If a required file is excluded, handle the WontFit exception.
• 6. Use the resulting file set to construct the agent prompt.
• 7. Verify that the total token usage stays below the budget.
• 8. Optionally compare the Python and JavaScript implementations for consistency.

Evidence

The hard part is what happens after you are over budget. You have to drop something — and most tools drop silently.

Source

So I built roomy: it packs under an explicit budget and returns, for every candidate, either the reason it is in or the reason it is out.

Source

Measured over one real repository: characters 206,658; the 4-chars-per-token rule 51,664; roomy's estimate 65,859 (+27%).

Source

It ships no tokenizer, on purpose — vendoring one ties the tool to a single model family and is wrong for every other.

Source

A required item that does not fit raises WontFit.

Source

36 unit tests. 909 verifier asserts including the cross-language comparison. 0 failures.

Source