The Small Model Was the Stock Answer

说明与用途

将SmolLM2 360M替换为Qwen2.5-3B-Instruct（4‑bit）后，语音代理能够考虑上下文（天气和空闲时间），并在几毫秒内生成更相关的回复。

问题

A voice agent using a small model (SmolLM2 360M) produced generic responses, failing to incorporate user context such as weather and time availability.

结果

Using a larger, quantized model enabled the voice agent to incorporate contextual information and produce more relevant responses within a few seconds.

工作流

• 1. Prompt the voice agent with a user query containing contextual details (e.g., weather and free time).
• 2. The agent processes the query using the SmolLM2 360M model and outputs a generic reply.
• 3. Replace the small model with Qwen2.5-3B-Instruct (4‑bit quantized).
• 4. Re‑prompt the agent with the same query.
• 5. Observe a context‑aware reply generated in 3017 ms using 39 tokens.

证据

The voice model was SmolLM2 at 360 million parameters. A model that size can finish a short sentence. It cannot hold the detail you just said and do something with it, so the sentence comes out generic.

来源

Asked "I have two hours free and it is raining. What should I do?" it answered: "Since it's raining and you have two hours free, why not take this opportunity to read a book?" That was 39 tokens in 3017 ms.

来源