Building an Intelligent Model Router for Local LLMs

Description and use

An automated router selects the best local LLM model based on task type, complexity, and performance history, enforcing a 3B minimum rule and switching between 3B, 7B, and 14B models, with a cloud fallback for reliability.

Problem

Local LLMs have limited CPU resources; small models are fast but lack reasoning, large models are slow. Choosing the right model per task is challenging.

Outcome

The router enforces a 3B minimum rule, dynamically switches between 3B, 7B, and 14B models based on task complexity, tracks performance to improve routing, and falls back to cloud APIs when local models are unsuitable, balancing quality and latency.

Workflow

• 1. Detect task specialty from the prompt using a predefined mapping.
• 2. Determine task complexity (simple, medium, complex).
• 3. Apply the 3B minimum rule: filter out models <3B for complex tasks.
• 4. Score available models based on specialty and preferred size.
• 5. Choose the highest‑scoring model that meets the size rule.
• 6. If no suitable local model, fall back to a cloud API (Groq, OpenRouter, Gemini).
• 7. Record the outcome (win/loss, latency) in model_performance.json for future routing decisions.

Evidence

3B minimum rule prevents poor decisions from tiny models

Source

Dynamic switching optimizes quality vs latency

Source