Unified LLM Chatbot with Automatic Failover

Description and use

A unified LLM API layer provides failover and quality control for an edtech chatbot, keeping the frontend unchanged while routing requests across multiple providers.

Problem

Ensuring an in‑app edtech chatbot remains available and produces high‑quality enrichment when a primary LLM provider is rate‑limited or unavailable.

Outcome

The unified API reduces credential sprawl, simplifies billing, and keeps the frontend unchanged while providing reliable, quality‑controlled enrichment across multiple LLM providers.

Workflow

• 1. Frontend sends enrichment request with product text, target learner, and constrained output shape.
• 2. Unified API receives request and selects provider based on availability and policy.
• 3. API calls LLM with model="auto" to let gateway choose provider.
• 4. API validates response JSON for required fields; rejects if missing. 5. If response is missing or provider returns 429, retry once with exponential backoff and jitter. 6. If retry fails, fail over to next provider. 7. Return enriched JSON to frontend or error with actionable message.
• 8. Frontend displays enriched data or error to user.

Evidence

Keep the frontend talking to one contract, then route the request to OpenAI, Claude, or Gemini behind it.

Source

A 429 is not a quality score.

Source

Do not silently retry forever. One exponential backoff sequence is enough for a user‑facing request.

Source