Hallucinating Castles: The Risks of Unverified AI Data Extraction

Описание и применение

Разработчик попытался использовать ИИ (Claude) для автоматического извлечения исторических данных об итальянских замках из неструктурированных текстов. В ходе тестирования на трех объектах было обнаружено 11 ошибок, включая искажение дат и фактов. ИИ часто подтверждал свои галлюцинации с уверенностью. В результате разработчик решил не использовать сгенерированный ИИ контент в финальном атласе, подчеркнув необходимость тщательной ручной проверки.

Задача

Automated extraction of historical and structural data from unstructured text sources using AI leads to subtle but critical hallucinations, including misattribution of dates, conflation of entities, and false confidence in incorrect data.

Результат

The experiment revealed that AI hallucinations are not just random errors but often plausible-sounding distortions of truth. In a test of three castles, 11 errors were found, including misinterpreting 'preserved' as 'restored' and conflating town founding dates with castle construction dates. The developer concluded that AI-generated content must be treated as a draft requiring rigorous human verification, and ultimately decided to exclude AI-generated information from the final atlas.

Рабочий процесс

• Define data fields to extract: location (from Wikidata), construction date, commissioner, current condition.
• Select a small test set of three castles: Walls of Cittadella, Castello Inferiore, and Castello Superiore.
• Prompt AI to read unstructured text from municipal websites and encyclopedias.
• Extract specific facts and format them into structured data.
• Implement automated checks for data consistency (e.g., date formats).
• Manually review the AI's output against original sources.
• Identify discrepancies such as misattributed dates, altered quotes, and conflated entities.
• Conclude that AI output requires line-by-line human verification and cannot be trusted as final data.

Доказательства

Eleven mistakes, across three entries. I found almost none of them myself: Michele did, by rereading the same three entries over and over

Источник

I turned that into 'that lord commissioned the castle' — the lord was connected to the town, not necessarily to the building.

Источник

The project won't contain any AI-generated information.

Источник