The Small Model Was the Stock Answer

Description and use

Switching from SmolLM2 360M to Qwen2.5-3B-Instruct (4‑bit) enabled a voice agent to incorporate user context and generate relevant replies within milliseconds.

Problem

A voice agent using a small model (SmolLM2 360M) produced generic responses, failing to incorporate user context such as weather and time availability.

Outcome

Using a larger, quantized model enabled the voice agent to incorporate contextual information and produce more relevant responses within a few seconds.

Workflow

• 1. Prompt the voice agent with a user query containing contextual details (e.g., weather and free time).
• 2. The agent processes the query using the SmolLM2 360M model and outputs a generic reply.
• 3. Replace the small model with Qwen2.5-3B-Instruct (4‑bit quantized).
• 4. Re‑prompt the agent with the same query.
• 5. Observe a context‑aware reply generated in 3017 ms using 39 tokens.

Evidence

The voice model was SmolLM2 at 360 million parameters. A model that size can finish a short sentence. It cannot hold the detail you just said and do something with it, so the sentence comes out generic.

Source

Asked "I have two hours free and it is raining. What should I do?" it answered: "Since it's raining and you have two hours free, why not take this opportunity to read a book?" That was 39 tokens in 3017 ms.

Source