I Tracked Every AI Hallucination for a Week — The Numbers Were Worse Than I Thought

Description and use

The author tested 200 AI outputs, finding 18 % errors, and built a fast verification layer that checks citations, code, leaks, and arguments, reducing manual debugging.

Problem

AI agents frequently produce hallucinations—confidently wrong outputs, fabricated citations, tool argument errors, and prompt leaks—leading to wasted token budgets and debugging effort.

Outcome

The verification layer runs in under 100 ms on CPU, is model-agnostic, and free, enabling developers to audit agent outputs daily and uncover hidden hallucinations.

Workflow

• Generate output with AI agent
• Pass output to verification layer
• Citation checker validates references
• Code validator checks syntax and logic
• Safety leak detector scans for prompt leaks
• Argument verifier ensures tool call parameters match schemas
• Coherence scorer compares output to original prompt
• If all checks pass, output is accepted; otherwise, flag for review

Evidence

Out of 200 outputs across Claude, GPT, and DeepSeek: 36 were confidently wrong (18%)

Source

All runs in under 100ms on CPU. Model-agnostic. Free.

Source