Inside the eval gate: how we decide a cheaper model can take over a task

说明与用途

Operant 通过 eval gate 在保持质量的前提下，将昂贵模型替换为更便宜的模型，评估过程包括 holdout 数据、LLM 判断和人工批准。

问题

How to safely replace a costly model with a cheaper one for repeated tasks while ensuring quality

结果

A learned rule can be safely promoted to active routing after passing the eval gate and human ratification, reducing cost while maintaining quality

工作流

• 1. Scribe learns a skill for a pattern and proposes a learned rule
• 2. Rule starts in shadow; no traffic changes
• 3. Gate splits pattern data into learn and holdout sets (default 25% holdout)
• 4. Gate replays each holdout conversation twice: baseline arm (original model) and candidate arm (target model + skill)
• 5. An LLM judge (default claude-opus-4-8) scores goal completion (parity 1.0, 0.5, 0.0) and writes a justification
• 6. Gate passes if mean parity ≥0.8
• 7. Human ratifies the rule
• 8. Rule becomes active and routes traffic to target model with skill

证据

Operant can move a repeated task to a cheaper model. It does this only after a gate.

来源

The gate grades the skill on the holdout set. It never grades the skill on the conversations that taught it.

来源

The gate passes when the mean parity is 0.8 or more.

来源