When to fine‑tune a small model for one narrow task

Описание и применение

Интерком заменил GPT‑4.1 на дообученную модель Qwen для задачи канонизации, сократив расходы на инференс до нуля.

Задача

High inference cost on a narrow, high‑volume ancillary task using a frontier LLM.

Результат

Intercom reduced inference spend on the summarization task from an estimated $250k/month to almost zero by swapping GPT‑4.1 for a fine‑tuned Qwen model.

Рабочий процесс

• 1. Identify ancillary, repeatable prompt that is close to a well‑prompted but untrained open model. 2. Run a production test with the untrained open model to confirm performance is acceptable. 3. Fine‑tune a small open‑weight model (e.g., Qwen 3) on the task. 4. Deploy the fine‑tuned model in place of the frontier model for that step. 5. Monitor resolution metrics and cost savings in production.

Доказательства

I don't remember exactly, but it could have been like a quarter million dollars a month on just that summarization task.

Источник

saved almost all of those, you know, several hundred thousand a month on inference just by doing that one thing.

Источник