I Tracked Every AI Hallucination for a Week — The Numbers Were Worse Than I Thought
Описание и применение
Автор провёл эксперимент, выявив, что около 18 % выводов ИИ содержат ошибки. Он создал слой проверки, который автоматически проверяет цитаты, код, утечки и аргументы, экономя токены и время отладки.
Задача
AI agents frequently produce hallucinated outputs, wasting token budget and causing debugging effort.
Результат
The verification layer reduced hallucinations, saving nearly 20% of token budget and decreasing manual debugging effort.
Рабочий процесс
- Run AI model to generate output
- Pass output to verification layer
- Citation checker validates references
- Code validator checks syntax and logic
- Safety leak detector identifies leaked prompts
- Argument verifier checks tool call parameters
- Coherence scorer compares output to prompt
- If all checks pass, output is accepted; otherwise, it is flagged for review
Доказательства
Out of 200 outputs across Claude, GPT, and DeepSeek: 36 were confidently wrong (18%)
All runs in under 100ms on CPU. Model-agnostic. Free.
- Дата записи: 2026-09-06T22:07:18.951Z