The Small Model Was the Stock Answer

Описание и применение

Использование более крупной модели Qwen2.5-3B-Instruct вместо SmolLM2 360M позволило голосовому агенту учитывать контекст (погода и свободное время) и выдавать более релевантные ответы за несколько секунд.

Задача

A voice agent using a small model (SmolLM2 360M) produced generic responses, failing to incorporate user context such as weather and time availability.

Результат

Using a larger, quantized model enabled the voice agent to incorporate contextual information and produce more relevant responses within a few seconds.

Рабочий процесс

• 1. Prompt the voice agent with a user query containing contextual details (e.g., weather and free time).
• 2. The agent processes the query using the SmolLM2 360M model and outputs a generic reply.
• 3. Replace the small model with Qwen2.5-3B-Instruct (4‑bit quantized).
• 4. Re‑prompt the agent with the same query.
• 5. Observe a context‑aware reply generated in 3017 ms using 39 tokens.

Доказательства

The voice model was SmolLM2 at 360 million parameters. A model that size can finish a short sentence. It cannot hold the detail you just said and do something with it, so the sentence comes out generic.

Источник

Asked "I have two hours free and it is raining. What should I do?" it answered: "Since it's raining and you have two hours free, why not take this opportunity to read a book?" That was 39 tokens in 3017 ms.

Источник