AI Makes Up Statistics: We Checked 186 of Its Claims

Описание и применение

AI часто генерирует статистику, приписывая достоверные источники, что приводит к непроверенным утверждениям в публикациях. В тесте 20 статей, созданных DeepSeek V4 Flash, 56,5 % проверяемых утверждений не имели подтверждения, 4,3 % противоречили источникам, а только 39,2 % были подтверждены. Результат подчёркивает необходимость ручной проверки и удаления неподтверждённых цифр.

Задача

AI-generated content frequently invents statistics, attaching plausible numbers to real or trusted sources, leading to unverified claims in published articles.

Результат

The test showed that 105 of 186 checkable claims (56.5%) could not be traced to any source, 8 were contradicted, and every article contained at least one unverified claim. Only two of seven manually traced claims held up, both with minor issues.

Рабочий процесс

• Generate 20 blog posts with AI using the specified prompt.
• Extract each checkable claim from the drafts.
• Search for a source that supports each claim using automated lookup services.
• Mark claims as confirmed, no source found, or contradicted.
• Flag unconfirmed claims for editor review.
• Manually trace a sample of flagged claims to verify authenticity.
• Adjust or remove unverified statistics before publishing.

Доказательства

Six in ten. That is how many of the checkable claims in 20 AI-written blog posts I could not confirm, even with a fact-checking pipeline built for exactly that job.

Источник

105 of 186 checkable claims (56.5%) could not be traced to any source, 8 more were contradicted by one, and every one of the 20 articles contained at least one of them.

Источник

The model rarely invents opinions; it invents figures.

Источник

Highlight every number, date and named source in the draft. Search the exact figure alongside the source's name — if the only results are other blogs repeating it, you have a zombie stat.

Источник