I Tracked Every AI Hallucination for a Week — The Numbers Were Worse Than I Thought

Description and use

The author tracked 200 AI outputs, finding 18% errors, including fabricated citations and prompt leaks. He built a verification layer that checks citations, code, safety, tool arguments, and coherence, reducing hallucinations and saving token budget.

Problem

AI agents frequently produce hallucinated outputs, consuming token budget and requiring manual correction.

Outcome

The verification layer reduced hallucinations, catching 18% confident errors, fabricated citations, tool argument issues, and prompt leaks, saving token budget and debugging time.

Workflow

• Generate output from LLM
• Run verification layer: citation checker, code validator, safety leak detector, argument verifier, coherence scorer
• If verification passes, forward output to codebase; otherwise flag for manual review

Evidence

Out of 200 outputs across Claude, GPT, and DeepSeek: 36 were confidently wrong (18%)

Source

All runs in under 100ms on CPU. Model-agnostic. Free.

Source