Checking what an LLM writes costs more than the writing

Описание и применение

Проверка контента, созданного LLM, занимает больше времени, чем сам процесс написания. Скачивание источников, проверка их целостности и ограничение модели цитированием только локальных файлов значительно повышают точность цитат и снижают количество ошибок.

Задача

LLM‑generated content often contains inaccurate or fabricated quotes, making post‑generation verification the most time‑consuming part of content creation.

Результат

Implementing a pre‑download and verification workflow reduced inaccurate quotes from ~35% to 5% and increased verified content to 95% of biographies, saving significant manual checking time.

Рабочий процесс

• Download source files with curl before model access
• Verify each file’s integrity and content
• Provide files to the writer and instruct it not to quote external URLs
• Model returns claims, URLs, and keywords; a script extracts matching sentences from downloaded HTML
• Run a second model as a judge to confirm each claim is supported by its quote
• Human reviewers double‑check flagged content and a second agent refutes findings
• Iterate until verification threshold is met

Доказательства

I measured it a few times. Asking a small model for literal quotes, between 24% and 35% of them survived a check against the source.

Источник

With that, 36 of 39 research quotes came out literal, and the thinker biographies ended with 95% of their quotes verifying against the source.

Источник

Download the sources with curl before any model sees them.

Источник