When to fine‑tune a small model for one narrow task

Description and use

Intercom swapped GPT‑4.1 for a fine‑tuned Qwen model on a canonicalization task, cutting inference costs from ~$250k/month to almost zero.

Problem

High inference cost on a narrow, high‑volume ancillary task using a frontier LLM.

Outcome

Intercom reduced inference spend on the summarization task from an estimated $250k/month to almost zero by swapping GPT‑4.1 for a fine‑tuned Qwen model.

Workflow

• 1. Identify ancillary, repeatable prompt that is close to a well‑prompted but untrained open model. 2. Run a production test with the untrained open model to confirm performance is acceptable. 3. Fine‑tune a small open‑weight model (e.g., Qwen 3) on the task. 4. Deploy the fine‑tuned model in place of the frontier model for that step. 5. Monitor resolution metrics and cost savings in production.

Evidence

I don't remember exactly, but it could have been like a quarter million dollars a month on just that summarization task.

Source

saved almost all of those, you know, several hundred thousand a month on inference just by doing that one thing.

Source