Building an Intelligent Model Router for Local LLMs

Описание и применение

Автоматический роутер выбирает оптимальную локальную модель LLM в зависимости от типа задачи, сложности и истории производительности, применяя правило 3B и переключаясь между 3B, 7B и 14B моделями, при необходимости используя облачные API.

Задача

Local LLMs have limited CPU resources; small models are fast but lack reasoning, large models are slow. Choosing the right model per task is challenging.

Результат

The router enforces a 3B minimum rule, dynamically switches between 3B, 7B, and 14B models based on task complexity, tracks performance to improve routing, and falls back to cloud APIs when local models are unsuitable, balancing quality and latency.

Рабочий процесс

• 1. Detect task specialty from the prompt using a predefined mapping.
• 2. Determine task complexity (simple, medium, complex).
• 3. Apply the 3B minimum rule: filter out models <3B for complex tasks.
• 4. Score available models based on specialty and preferred size.
• 5. Choose the highest‑scoring model that meets the size rule.
• 6. If no suitable local model, fall back to a cloud API (Groq, OpenRouter, Gemini).
• 7. Record the outcome (win/loss, latency) in model_performance.json for future routing decisions.

Доказательства

3B minimum rule prevents poor decisions from tiny models

Источник

Dynamic switching optimizes quality vs latency

Источник