I Tracked Every AI Hallucination for a Week — The Numbers Were Worse Than I Thought

Description and use

The author found that 18 % of AI outputs were hallucinations. He built a verification layer that detects fabricated citations, code errors, and prompt leaks in real time, cutting debugging time.

Problem

AI agents frequently produce hallucinations—confidently wrong statements, fabricated citations, tool misuse, and prompt leaks—leading to wasted token budgets and debugging effort.

Outcome

The verification layer runs in under 100 ms on CPU, is model-agnostic, and free, significantly reducing hallucination-related token waste and debugging time.

Workflow

• Generate output with AI agent
• Pass output through verification layer
• 1. Citation checker validates references
• 2. Code validator checks syntax and logic
• 3. Safety leak detector identifies leaked prompts
• 4. Argument verifier checks tool call parameters
• 5. Coherence scorer compares output to original prompt
• If any check fails, flag output for review; otherwise, forward to codebase

Evidence

Out of 200 outputs across Claude, GPT, and DeepSeek: 36 were confidently wrong (18%)

Source