We asked a decision model to check our AI's work. It got good the day we stopped asking it to think.

说明与用途

使用 Decision‑1 作为轻量级检查器，降低了 AI 调用错误率至零，并将成本降至 300 次调用不到一分钱。

问题

Ensuring that AI-generated tool calls are correct before they are sent to users, without relying on expensive models for every check.

结果

Version 2.4 of the checker approved no wrong calls, fixed 20 of 96 wrong calls, and broke 0 of 201 correct calls, achieving the target metrics while keeping cost under a cent for 300 calls.

工作流

• 1. Code parses the user’s request and extracts relevant facts (e.g., city name, list length).
• 2. Decision‑1 receives a single factual question and returns a probability.
• 3. If probability exceeds a threshold, the call is APPROVED. If below, the cheap model attempts a repair.
• 4. The repaired call is re‑checked by Decision‑1.
• 5. Final status (APPROVE, REJECT, UNSURE) and a dossier of checks are returned.

证据

On a relevance question the same input moved by as much as 0.17 between calls.

来源

When the question was about a fact, it was steady. "Does the user's message contain this exact city name?" came back the same every time, and came back right.

来源

Version 2.4 fixed the order of those two checks, stopped the repair model from rescaling numbers, and added one quiet rule: a repair is kept only if the checker approves it.

来源

The whole 300‑call test cost under a cent, and on 1,417 more recorded calls it ran at about two thousandths of a cent per call.

来源