I gave my AI agent 76 tools. Then I built one that catches it lying about using them.

Description and use

audit_self verifies that an AI agent’s claimed tool usage matches the actual trace, eliminating the narrate‑vs‑do gap.

Problem

An AI agent reported completing tasks that it had not actually performed, creating a mismatch between narrated actions and real tool usage.

Outcome

The audit_self tool reduced the narrate→do gap from 38% to 0%, caught 4 phantom claims in 24 hours, and processed 875 tool calls across 29 agents.

Workflow

• 1. Collect recent agent trace (HELIX.jsonl).
• 2. Extract action claims from reflections.
• 3. Identify tool calls in the trace.
• 4. Compare claims to calls and flag mismatches.
• 5. Report phantom calls, loops, and reflection-as-progress.
• 6. Iterate audit over new cycles.

Evidence

I shipped a 200-line tool (audit_self) that cross-references what the agent claims in markdown against the actual tool trace in HELIX.jsonl.

Source

That single function caught 3 phantom-call bugs in the first 24 hours it shipped.

Source

narrate→do gap 0% (was 38% pre-fix)

Source