AI Makes Up Statistics: We Checked 186 of Its Claims

Description and use

AI frequently fabricates statistics by attaching plausible numbers to real or trusted sources, resulting in unverified claims in published content. In a test of 20 articles generated by DeepSeek V4 Flash, 56.5 % of checkable claims had no source, 4.3 % were contradicted, and only 39.2 % were confirmed. The outcome underscores the need for manual verification and removal of unverifiable figures.

Problem

AI-generated content frequently invents statistics, attaching plausible numbers to real or trusted sources, leading to unverified claims in published articles.

Outcome

The test showed that 105 of 186 checkable claims (56.5%) could not be traced to any source, 8 were contradicted, and every article contained at least one unverified claim. Only two of seven manually traced claims held up, both with minor issues.

Workflow

• Generate 20 blog posts with AI using the specified prompt.
• Extract each checkable claim from the drafts.
• Search for a source that supports each claim using automated lookup services.
• Mark claims as confirmed, no source found, or contradicted.
• Flag unconfirmed claims for editor review.
• Manually trace a sample of flagged claims to verify authenticity.
• Adjust or remove unverified statistics before publishing.

Evidence

Six in ten. That is how many of the checkable claims in 20 AI-written blog posts I could not confirm, even with a fact-checking pipeline built for exactly that job.

Source

105 of 186 checkable claims (56.5%) could not be traced to any source, 8 more were contradicted by one, and every one of the 20 articles contained at least one of them.

Source

The model rarely invents opinions; it invents figures.

Source

Highlight every number, date and named source in the draft. Search the exact figure alongside the source's name — if the only results are other blogs repeating it, you have a zombie stat.

Source