Fixing Silent Prompt Truncation in Multilingual Ticket Classification

Description and use

A developer found that a ticket classification service failed on long, non-English messages after switching to a cheaper endpoint. The issue was a token-counting mismatch where character-based truncation removed classification instructions. The fix involved using the endpoint's tokenizer for truncation and adding a sentinel marker to validate response integrity.

Problem

A ticket-classification service began returning fallback labels for long, non-English messages after migrating to a cheaper inference endpoint. The issue was not model quality but a token-counting mismatch that silently truncated prompts before the classification instruction.

Outcome

The bug was resolved by aligning client-side truncation with the endpoint's tokenization logic and adding a sentinel marker to detect silent instruction loss. This prevented silent failures in multilingual long-context scenarios.

Workflow

• Identify consistent failure patterns in long, non-English inputs despite valid HTTP responses.
• Isolate the variable by replaying failing tickets in a clean environment (e.g., using MonkeyCode for free model access).
• Log token counts instead of character counts to reveal discrepancies between client-side limits and endpoint tokenization.
• Replace character-based truncation with token-based truncation using the endpoint's specific tokenizer.
• Implement a sentinel instruction requiring a specific marker in the model's JSON output.
• Add validation logic to reject responses missing the sentinel marker.
• Create regression tests with long, multilingual fixtures to assert marker presence.

Evidence

The real culprit was a token-counting mismatch that silently truncated the prompt before the model ever saw the classification instruction.

Source

The tokenizer used by the new endpoint split German and Japanese text into roughly twice as many tokens per character as English.

Source

I replaced the character-based guard with a token-based guard that used the same tokenizer as the inference endpoint.

Source