We asked a decision model to check our AI's work. It got good the day we stopped asking it to think.

Description and use

Deploying Decision‑1 as a lightweight checker for AI calls reduced errors to zero and cut costs to under a cent for 300 calls.

Problem

Ensuring that AI-generated tool calls are correct before they are sent to users, without relying on expensive models for every check.

Outcome

Version 2.4 of the checker approved no wrong calls, fixed 20 of 96 wrong calls, and broke 0 of 201 correct calls, achieving the target metrics while keeping cost under a cent for 300 calls.

Workflow

• 1. Code parses the user’s request and extracts relevant facts (e.g., city name, list length).
• 2. Decision‑1 receives a single factual question and returns a probability.
• 3. If probability exceeds a threshold, the call is APPROVED. If below, the cheap model attempts a repair.
• 4. The repaired call is re‑checked by Decision‑1.
• 5. Final status (APPROVE, REJECT, UNSURE) and a dossier of checks are returned.

Evidence

On a relevance question the same input moved by as much as 0.17 between calls.

Source

When the question was about a fact, it was steady. "Does the user's message contain this exact city name?" came back the same every time, and came back right.

Source

Version 2.4 fixed the order of those two checks, stopped the repair model from rescaling numbers, and added one quiet rule: a repair is kept only if the checker approves it.

Source

The whole 300‑call test cost under a cent, and on 1,417 more recorded calls it ran at about two thousandths of a cent per call.

Source