Fixing Silent Prompt Truncation in Multilingual Ticket Classification

说明与用途

一名开发者发现，在切换到更便宜的端点后，工单分类服务在处理长非英语消息时出现故障。问题在于令牌计数不匹配，基于字符的截断移除了分类指令。解决方案是使用端点的分词器进行截断，并添加哨兵标记以验证响应完整性。

问题

A ticket-classification service began returning fallback labels for long, non-English messages after migrating to a cheaper inference endpoint. The issue was not model quality but a token-counting mismatch that silently truncated prompts before the classification instruction.

结果

The bug was resolved by aligning client-side truncation with the endpoint's tokenization logic and adding a sentinel marker to detect silent instruction loss. This prevented silent failures in multilingual long-context scenarios.

工作流

• Identify consistent failure patterns in long, non-English inputs despite valid HTTP responses.
• Isolate the variable by replaying failing tickets in a clean environment (e.g., using MonkeyCode for free model access).
• Log token counts instead of character counts to reveal discrepancies between client-side limits and endpoint tokenization.
• Replace character-based truncation with token-based truncation using the endpoint's specific tokenizer.
• Implement a sentinel instruction requiring a specific marker in the model's JSON output.
• Add validation logic to reject responses missing the sentinel marker.
• Create regression tests with long, multilingual fixtures to assert marker presence.

证据

The real culprit was a token-counting mismatch that silently truncated the prompt before the model ever saw the classification instruction.

来源

The tokenizer used by the new endpoint split German and Japanese text into roughly twice as many tokens per character as English.

来源

I replaced the character-based guard with a token-based guard that used the same tokenizer as the inference endpoint.

来源