I Tracked Every AI Hallucination for a Week — The Numbers Were Worse Than I Thought

Description and use

The author tracked 200 LLM outputs and found 18 % hallucinations. Built a lightweight verification layer that checks citations, code, leaks, arguments, and coherence, reducing errors.

Problem

AI agents frequently produce hallucinated outputs, wasting token budget and requiring manual correction.

Outcome

The verification layer runs in under 100 ms on CPU, is model‑agnostic, and free, reducing hallucination impact and saving token budget.

Workflow

• Generate output with LLM
• Run verification layer: citation checker, code validator, safety leak detector, argument verifier, coherence scorer
• If any check fails, flag for manual review; otherwise, pass output to codebase

Evidence

Out of 200 outputs across Claude, GPT, and DeepSeek: 36 were confidently wrong (18%)

Source

All runs in under 100ms on CPU. Model-agnostic. Free.

Source