Hallucinating Castles: The Risks of Unverified AI Data Extraction

Description and use

A developer attempted to use AI (Claude) to automatically extract historical data about Italian castles from unstructured text. Testing on three entries revealed 11 errors, including distorted dates and facts. The AI often validated its own hallucinations with confidence. Consequently, the developer decided not to include AI-generated content in the final atlas, emphasizing the need for rigorous manual verification.

Problem

Automated extraction of historical and structural data from unstructured text sources using AI leads to subtle but critical hallucinations, including misattribution of dates, conflation of entities, and false confidence in incorrect data.

Outcome

The experiment revealed that AI hallucinations are not just random errors but often plausible-sounding distortions of truth. In a test of three castles, 11 errors were found, including misinterpreting 'preserved' as 'restored' and conflating town founding dates with castle construction dates. The developer concluded that AI-generated content must be treated as a draft requiring rigorous human verification, and ultimately decided to exclude AI-generated information from the final atlas.

Workflow

• Define data fields to extract: location (from Wikidata), construction date, commissioner, current condition.
• Select a small test set of three castles: Walls of Cittadella, Castello Inferiore, and Castello Superiore.
• Prompt AI to read unstructured text from municipal websites and encyclopedias.
• Extract specific facts and format them into structured data.
• Implement automated checks for data consistency (e.g., date formats).
• Manually review the AI's output against original sources.
• Identify discrepancies such as misattributed dates, altered quotes, and conflated entities.
• Conclude that AI output requires line-by-line human verification and cannot be trusted as final data.

Evidence

Eleven mistakes, across three entries. I found almost none of them myself: Michele did, by rereading the same three entries over and over

Source

I turned that into 'that lord commissioned the castle' — the lord was connected to the town, not necessarily to the building.

Source

The project won't contain any AI-generated information.

Source