I Tracked Every AI Hallucination for a Week — The Numbers Were Worse Than I Thought

Description and use

The author found that about 18 % of AI outputs were wrong. He built a verification layer that automatically checks citations, code, leaks, and arguments, saving tokens and debugging time.

Problem

AI agents frequently produce hallucinated outputs, wasting token budget and causing debugging effort.

Outcome

The verification layer reduced hallucinations, saving nearly 20% of token budget and decreasing manual debugging effort.

Workflow

• Run AI model to generate output
• Pass output to verification layer
• Citation checker validates references
• Code validator checks syntax and logic
• Safety leak detector identifies leaked prompts
• Argument verifier checks tool call parameters
• Coherence scorer compares output to prompt
• If all checks pass, output is accepted; otherwise, it is flagged for review

Evidence

Out of 200 outputs across Claude, GPT, and DeepSeek: 36 were confidently wrong (18%)

Source

All runs in under 100ms on CPU. Model-agnostic. Free.

Source