Checking what an LLM writes costs more than the writing

说明与用途

验证LLM生成的内容比写作更耗时。下载源文件、检查文件完整性，并限制模型仅引用本地文件，可显著提高引用准确性并减少错误。

问题

LLM‑generated content often contains inaccurate or fabricated quotes, making post‑generation verification the most time‑consuming part of content creation.

结果

Implementing a pre‑download and verification workflow reduced inaccurate quotes from ~35% to 5% and increased verified content to 95% of biographies, saving significant manual checking time.

工作流

• Download source files with curl before model access
• Verify each file’s integrity and content
• Provide files to the writer and instruct it not to quote external URLs
• Model returns claims, URLs, and keywords; a script extracts matching sentences from downloaded HTML
• Run a second model as a judge to confirm each claim is supported by its quote
• Human reviewers double‑check flagged content and a second agent refutes findings
• Iterate until verification threshold is met

证据

I measured it a few times. Asking a small model for literal quotes, between 24% and 35% of them survived a check against the source.

来源

With that, 36 of 39 research quotes came out literal, and the thinker biographies ended with 95% of their quotes verifying against the source.

来源

Download the sources with curl before any model sees them.

来源