Catch Your Agent Inventing Tool Results (Before Your Users Do)

Description and use

Auditing agent tool calls against an executor-side ledger detects fabricated or misquoted results, ensuring only genuine executions are reported.

AI agents can fabricate or misquote tool results, leading to hallucinated outcomes that pass transcript reviews.

Auditing against an executor-side ledger reliably uncovers fabricated or misquoted tool results, ensuring agents only report genuine executions.

1. Record every tool dispatch in a ledger: tool_call_id → {tool, args, raw result}.
2. After agent run, extract all factual claims about tool outcomes, noting referenced call IDs and asserted results.
3. For each claim, check ledger: if call ID missing → FABRICATED; if result mismatches → MISQUOTED; else OK.
4. Flag any FABRICATED or MISQUOTED findings for review.
5. Repeat audit after every run that touches real resources.

Audit them against an executor-side ledger.

Your executor records every dispatch: tool_call_id → {tool, args, raw result} in a ledger the agent can't write to.