Fixing Silent Prompt Truncation in Multilingual Ticket Classification

Описание и применение

Разработчик обнаружил, что сервис классификации тикетов ошибочно обрабатывал длинные сообщения на немецком и японском языках после перехода на более дешевый эндпоинт. Проблема заключалась в несоответствии подсчета токенов: клиентская обрезка по символам удаляла инструкции из промпта. Решение включало использование токенизатора эндпоинта для обрезки и добавление маркера-сигнала в ответ модели для проверки целостности.

Задача

A ticket-classification service began returning fallback labels for long, non-English messages after migrating to a cheaper inference endpoint. The issue was not model quality but a token-counting mismatch that silently truncated prompts before the classification instruction.

Результат

The bug was resolved by aligning client-side truncation with the endpoint's tokenization logic and adding a sentinel marker to detect silent instruction loss. This prevented silent failures in multilingual long-context scenarios.

Рабочий процесс

• Identify consistent failure patterns in long, non-English inputs despite valid HTTP responses.
• Isolate the variable by replaying failing tickets in a clean environment (e.g., using MonkeyCode for free model access).
• Log token counts instead of character counts to reveal discrepancies between client-side limits and endpoint tokenization.
• Replace character-based truncation with token-based truncation using the endpoint's specific tokenizer.
• Implement a sentinel instruction requiring a specific marker in the model's JSON output.
• Add validation logic to reject responses missing the sentinel marker.
• Create regression tests with long, multilingual fixtures to assert marker presence.

Доказательства

The real culprit was a token-counting mismatch that silently truncated the prompt before the model ever saw the classification instruction.

Источник

The tokenizer used by the new endpoint split German and Japanese text into roughly twice as many tokens per character as English.

Источник

I replaced the character-based guard with a token-based guard that used the same tokenizer as the inference endpoint.

Источник