Multiple LLM Providers: One API Key With Observable Candidate-Scoring Fallback

说明与用途

一家媒体公司使用单一 API 密钥将候选人评分请求路由到多个 LLM 提供商，但未验证响应合同，导致评分无效。通过强制执行可移植 JSON 合同、单独记录持久化评分以及使用幂等键进行安全回退，他们实现了有效、受限的评分和可靠的运营指标。

问题

A media company’s overnight batch hiring dashboard shows unranked applicants even though the API gateway reports 18,420 completed jobs and no obvious outage.

结果

After enforcing contract validation and separate score recording, the hiring dashboard displayed valid, bounded scores for all eligible candidates, and operational metrics accurately reflected scoring health.

工作流

• 1. Define a minimal JSON contract that all providers can honor.
• 2. Validate every model response against the contract and rubric version.
• 3. Record a durable score entry with candidate ID, rubric version, input digest, route, attempt, outcome, and timestamp.
• 4. Separate raw model responses from committed scores for audit and debugging.
• 5. Use deterministic idempotency keys to prevent duplicate commits.
• 6. Trigger fallback retries only for transport errors, not for schema or policy failures.
• 7. Monitor operational metrics: result freshness, valid completion rate, and outcome vocabulary.
• 8. Adjust routing policies based on validated outcomes rather than HTTP status alone.

证据

A single-key, multi-provider gateway can reduce integration sprawl, but it cannot define success for candidate scoring.

来源

Keep the score record separate from the raw model response. The durable record should include a candidate identifier, rubric version, input digest, model-route identifier, attempt number, validation outcome, and completion time.

来源

Use a deterministic idempotency key derived from the candidate, rubric version, and input digest. Commit with a uniqueness constraint on that key.

来源