I Tracked Every AI Hallucination for a Week — The Numbers Were Worse Than I Thought

Description and use

The author ran a week-long experiment with LLMs, finding 18 % of outputs were hallucinated. He built a lightweight verification layer that checks citations, code, safety leaks, tool arguments, and coherence, cutting token waste and debugging time.

Problem

AI agents frequently produce hallucinated outputs, consuming token budget and requiring manual correction.

Outcome

The verification layer runs in under 100 ms on CPU, is model‑agnostic, and free, reducing hallucination-related token waste and debugging time.

Workflow

• Generate output with AI agent
• Run verification layer: citation checker, code validator, safety leak detector, argument verifier, coherence scorer
• If verification passes, forward output to codebase; otherwise flag for manual review

Evidence

Out of 200 outputs across Claude, GPT, and DeepSeek: 36 were confidently wrong (18%)

Source