Python json.dumps ensure_ascii: 24% More Claude Tokens, Wrong Kanji

说明与用途

Python 的 json.dumps 默认会转义非 ASCII 字符，导致令牌数量增加并在 LLM 提示中产生提取错误。将 ensure_ascii 设置为 False（以及 pandas 的相应选项）可将令牌数减少 24% 并提升提取准确率。

问题

Using Python's json.dumps with its default ensure_ascii=True rewrites non‑ASCII characters into \uXXXX escapes, inflating token counts and causing LLM extraction errors when the escaped JSON is embedded directly into prompts.

结果

Switching to ensure_ascii=False reduced average input tokens from 1,180 to 1,461 per request (24% drop) and lowered wrong‑character extraction from 8.5% to 0.9% on Japanese tickets.

工作流

• Identify JSON serialization points in the pipeline.
• Replace json.dumps calls with json.dumps(..., ensure_ascii=False).
• Adjust pandas.DataFrame.to_json(force_ascii=False).
• Add a test that fails if any \u escape appears in a rendered prompt.
• Run the 2,000‑ticket replay set to verify token counts and extraction accuracy.
• Deploy the updated code.
• Monitor token usage and extraction quality over time.

证据

Python's json.dumps defaults to ensure_ascii=True, which turns é into \u00fcnchen and 東 into \u6771\u4eac.

来源

On my 2,000-ticket replay set, that raised average input tokens from 1,180 to 1,461 per request (+24%).

来源

29 of 340 had at least one wrong character (8.5%) with escaped prompts vs 3 of 340 (0.9%) with raw UTF-8 prompts.

来源

Fix: json.dumps(obj, ensure_ascii=False), plus force_ascii=False in pandas to_json, plus a test that fails if \u shows up in a rendered prompt.

来源