Unified LLM Chatbot with Automatic Failover

说明与用途

统一的 LLM API 层为教育技术聊天机器人提供故障转移和质量控制，保持前端不变，同时在多个提供商之间路由请求。

问题

Ensuring an in‑app edtech chatbot remains available and produces high‑quality enrichment when a primary LLM provider is rate‑limited or unavailable.

结果

The unified API reduces credential sprawl, simplifies billing, and keeps the frontend unchanged while providing reliable, quality‑controlled enrichment across multiple LLM providers.

工作流

• 1. Frontend sends enrichment request with product text, target learner, and constrained output shape.
• 2. Unified API receives request and selects provider based on availability and policy.
• 3. API calls LLM with model="auto" to let gateway choose provider.
• 4. API validates response JSON for required fields; rejects if missing. 5. If response is missing or provider returns 429, retry once with exponential backoff and jitter. 6. If retry fails, fail over to next provider. 7. Return enriched JSON to frontend or error with actionable message.
• 8. Frontend displays enriched data or error to user.

证据

Keep the frontend talking to one contract, then route the request to OpenAI, Claude, or Gemini behind it.

来源

A 429 is not a quality score.

来源

Do not silently retry forever. One exponential backoff sequence is enough for a user‑facing request.

来源