Multiple LLM Providers: One API Key With Observable Candidate-Scoring Fallback

Описание и применение

Компания использовала один ключ для маршрутизации запросов к нескольким LLM‑провайдерам, но не проверяла контракты ответов, что приводило к недействительным оценкам кандидатов. Внедрив обязательную валидацию JSON‑контракта, отдельную запись результатов и idempotent‑ключи, они обеспечили корректные оценки и надёжные метрики.

Задача

A media company’s overnight batch hiring dashboard shows unranked applicants even though the API gateway reports 18,420 completed jobs and no obvious outage.

Результат

After enforcing contract validation and separate score recording, the hiring dashboard displayed valid, bounded scores for all eligible candidates, and operational metrics accurately reflected scoring health.

Рабочий процесс

• 1. Define a minimal JSON contract that all providers can honor.
• 2. Validate every model response against the contract and rubric version.
• 3. Record a durable score entry with candidate ID, rubric version, input digest, route, attempt, outcome, and timestamp.
• 4. Separate raw model responses from committed scores for audit and debugging.
• 5. Use deterministic idempotency keys to prevent duplicate commits.
• 6. Trigger fallback retries only for transport errors, not for schema or policy failures.
• 7. Monitor operational metrics: result freshness, valid completion rate, and outcome vocabulary.
• 8. Adjust routing policies based on validated outcomes rather than HTTP status alone.

Доказательства

A single-key, multi-provider gateway can reduce integration sprawl, but it cannot define success for candidate scoring.

Источник

Keep the score record separate from the raw model response. The durable record should include a candidate identifier, rubric version, input digest, model-route identifier, attempt number, validation outcome, and completion time.

Источник

Use a deterministic idempotency key derived from the candidate, rubric version, and input digest. Commit with a uniqueness constraint on that key.

Источник