Inside the eval gate: how we decide a cheaper model can take over a task

Описание и применение

Платформа Operant использует «eval gate» для безопасного замещения дорогой модели более дешевой, проверяя качество через holdout‑данные, LLM‑жюри и ручное утверждение.

Задача

How to safely replace a costly model with a cheaper one for repeated tasks while ensuring quality

Результат

A learned rule can be safely promoted to active routing after passing the eval gate and human ratification, reducing cost while maintaining quality

Рабочий процесс

• 1. Scribe learns a skill for a pattern and proposes a learned rule
• 2. Rule starts in shadow; no traffic changes
• 3. Gate splits pattern data into learn and holdout sets (default 25% holdout)
• 4. Gate replays each holdout conversation twice: baseline arm (original model) and candidate arm (target model + skill)
• 5. An LLM judge (default claude-opus-4-8) scores goal completion (parity 1.0, 0.5, 0.0) and writes a justification
• 6. Gate passes if mean parity ≥0.8
• 7. Human ratifies the rule
• 8. Rule becomes active and routes traffic to target model with skill

Доказательства

Operant can move a repeated task to a cheaper model. It does this only after a gate.

Источник

The gate grades the skill on the holdout set. It never grades the skill on the conversations that taught it.

Источник

The gate passes when the mean parity is 0.8 or more.

Источник