I Tracked Every AI Hallucination for a Week — The Numbers Were Worse Than I Thought

Описание и применение

Автор проверил 200 выводов LLM и обнаружил 18 % ошибок. Создал быстрый слой проверки, который проверяет цитаты, код, утечки, аргументы и согласованность, уменьшая количество ошибок.

Задача

AI agents frequently produce hallucinated outputs, wasting token budget and requiring manual correction.

Результат

The verification layer runs in under 100 ms on CPU, is model‑agnostic, and free, reducing hallucination impact and saving token budget.

Рабочий процесс

• Generate output with LLM
• Run verification layer: citation checker, code validator, safety leak detector, argument verifier, coherence scorer
• If any check fails, flag for manual review; otherwise, pass output to codebase

Доказательства

Out of 200 outputs across Claude, GPT, and DeepSeek: 36 were confidently wrong (18%)

Источник

All runs in under 100ms on CPU. Model-agnostic. Free.

Источник