I Tracked Every AI Hallucination for a Week — The Numbers Were Worse Than I Thought

Description and use

The author tracked AI hallucinations over a week, finding nearly 20 % were incorrect. He built a verification layer that automatically checks citations, code, leaks, and arguments, reducing manual effort.

Problem

AI agents frequently produce hallucinations—confidently incorrect outputs, fabricated citations, tool misuse, and prompt leaks—leading to wasted token budget and debugging effort.

Outcome

The verification layer runs under 100 ms on CPU, is model-agnostic, and free, helping developers catch hallucinations early and save token budget.

Workflow

• Generate output from LLM
• Run verification layer: citation checker, code validator, safety leak detector, argument verifier, coherence scorer
• If verification passes, forward output to codebase; otherwise flag for review

Evidence

Out of 200 outputs across Claude, GPT, and DeepSeek: 36 were confidently wrong (18%)

Source