LLM Fallback Routing for Production Stability

Description and use

The author describes transitioning from a single expensive LLM for all tasks to a task-specific routing architecture. This improved stability, reduced costs, and avoided availability and latency issues in production.

Problem

Using a single expensive LLM for all tasks leads to latency spikes, timeouts, cost drift, and availability issues in production, especially when the model is overloaded or throttled.

Outcome

By implementing task-specific LLM routing with conditional fallbacks, the author achieved better stability, reduced costs, and improved reliability in production workflows, avoiding the pitfalls of a single-model architecture.

Workflow

• Identify distinct workloads: planning, extraction, review, and background jobs.
• Assign primary models based on task requirements (e.g., reasoning quality for planning, structured output for extraction).
• Implement routing using tools like OpenRouter or Portkey to manage provider selection.
• Configure fallback policies to trigger only on specific status codes (e.g., 429, 503) rather than general errors.
• Route non-urgent, async tasks to batch APIs (e.g., Gemini Batch) to reduce costs.
• Ensure prompt and schema compatibility across primary and fallback models to prevent output drift.
• Log the final provider and model for every step to aid debugging and cost analysis.

Evidence

The real upgrade was not a fancier model. It was routing.

Source

Portkey lets you define fallback chains and trigger them only for specific status codes like 429 or 503.

Source

Batch pricing is 50% cheaper than the standard path.

Source