Python json.dumps ensure_ascii: 24% More Claude Tokens, Wrong Kanji

Description and use

Python’s json.dumps defaults to escaping non‑ASCII characters, inflating token counts and causing extraction errors in LLM prompts. Setting ensure_ascii=False (and equivalent options in pandas) reduces tokens by 24 % and improves extraction accuracy.

Problem

Using Python's json.dumps with its default ensure_ascii=True rewrites non‑ASCII characters into \uXXXX escapes, inflating token counts and causing LLM extraction errors when the escaped JSON is embedded directly into prompts.

Outcome

Switching to ensure_ascii=False reduced average input tokens from 1,180 to 1,461 per request (24% drop) and lowered wrong‑character extraction from 8.5% to 0.9% on Japanese tickets.

Workflow

• Identify JSON serialization points in the pipeline.
• Replace json.dumps calls with json.dumps(..., ensure_ascii=False).
• Adjust pandas.DataFrame.to_json(force_ascii=False).
• Add a test that fails if any \u escape appears in a rendered prompt.
• Run the 2,000‑ticket replay set to verify token counts and extraction accuracy.
• Deploy the updated code.
• Monitor token usage and extraction quality over time.

Evidence

Python's json.dumps defaults to ensure_ascii=True, which turns é into \u00fcnchen and 東 into \u6771\u4eac.

Source

On my 2,000-ticket replay set, that raised average input tokens from 1,180 to 1,461 per request (+24%).

Source

29 of 340 had at least one wrong character (8.5%) with escaped prompts vs 3 of 340 (0.9%) with raw UTF-8 prompts.

Source

Fix: json.dumps(obj, ensure_ascii=False), plus force_ascii=False in pandas to_json, plus a test that fails if \u shows up in a rendered prompt.

Source