I Tracked Every AI Hallucination for a Week — The Numbers Were Worse Than I Thought

说明与用途

作者测试了200个AI输出，发现18%错误，并构建了一个快速验证层，检查引用、代码、泄露和参数，减少手动调试。

问题

AI agents frequently produce hallucinations—confidently wrong outputs, fabricated citations, tool argument errors, and prompt leaks—leading to wasted token budgets and debugging effort.

结果

The verification layer runs in under 100 ms on CPU, is model-agnostic, and free, enabling developers to audit agent outputs daily and uncover hidden hallucinations.

工作流

• Generate output with AI agent
• Pass output to verification layer
• Citation checker validates references
• Code validator checks syntax and logic
• Safety leak detector scans for prompt leaks
• Argument verifier ensures tool call parameters match schemas
• Coherence scorer compares output to original prompt
• If all checks pass, output is accepted; otherwise, flag for review

证据

Out of 200 outputs across Claude, GPT, and DeepSeek: 36 were confidently wrong (18%)

来源

All runs in under 100ms on CPU. Model-agnostic. Free.

来源