We asked a decision model to check our AI's work. It got good the day we stopped asking it to think.

Описание и применение

Внедрение Decision‑1 как лёгкого проверяющего для AI‑запросов позволило снизить количество ошибок до нуля и уменьшить расходы до менее чем цента за 300 запросов.

Задача

Ensuring that AI-generated tool calls are correct before they are sent to users, without relying on expensive models for every check.

Результат

Version 2.4 of the checker approved no wrong calls, fixed 20 of 96 wrong calls, and broke 0 of 201 correct calls, achieving the target metrics while keeping cost under a cent for 300 calls.

Рабочий процесс

• 1. Code parses the user’s request and extracts relevant facts (e.g., city name, list length).
• 2. Decision‑1 receives a single factual question and returns a probability.
• 3. If probability exceeds a threshold, the call is APPROVED. If below, the cheap model attempts a repair.
• 4. The repaired call is re‑checked by Decision‑1.
• 5. Final status (APPROVE, REJECT, UNSURE) and a dossier of checks are returned.

Доказательства

On a relevance question the same input moved by as much as 0.17 between calls.

Источник

When the question was about a fact, it was steady. "Does the user's message contain this exact city name?" came back the same every time, and came back right.

Источник

Version 2.4 fixed the order of those two checks, stopped the repair model from rescaling numbers, and added one quiet rule: a repair is kept only if the checker approves it.

Источник

The whole 300‑call test cost under a cent, and on 1,417 more recorded calls it ran at about two thousandths of a cent per call.

Источник