Hallucinating Castles: The Risks of Unverified AI Data Extraction

说明与用途

一名开发者尝试使用AI（Claude）从不结构化文本中自动提取意大利城堡的历史数据。在对三个条目的测试中，发现了11处错误，包括日期和事实的扭曲。AI经常自信地验证其自身的幻觉。因此，开发者决定在最终的大全中不包含AI生成的内容，强调了严格人工验证的必要性。

问题

Automated extraction of historical and structural data from unstructured text sources using AI leads to subtle but critical hallucinations, including misattribution of dates, conflation of entities, and false confidence in incorrect data.

结果

The experiment revealed that AI hallucinations are not just random errors but often plausible-sounding distortions of truth. In a test of three castles, 11 errors were found, including misinterpreting 'preserved' as 'restored' and conflating town founding dates with castle construction dates. The developer concluded that AI-generated content must be treated as a draft requiring rigorous human verification, and ultimately decided to exclude AI-generated information from the final atlas.

工作流

• Define data fields to extract: location (from Wikidata), construction date, commissioner, current condition.
• Select a small test set of three castles: Walls of Cittadella, Castello Inferiore, and Castello Superiore.
• Prompt AI to read unstructured text from municipal websites and encyclopedias.
• Extract specific facts and format them into structured data.
• Implement automated checks for data consistency (e.g., date formats).
• Manually review the AI's output against original sources.
• Identify discrepancies such as misattributed dates, altered quotes, and conflated entities.
• Conclude that AI output requires line-by-line human verification and cannot be trusted as final data.

证据

Eleven mistakes, across three entries. I found almost none of them myself: Michele did, by rereading the same three entries over and over

来源

I turned that into 'that lord commissioned the castle' — the lord was connected to the town, not necessarily to the building.

来源

The project won't contain any AI-generated information.

来源