How I stopped my research agent from inventing its sources

说明与用途

研究代理经常生成带链接的声明，但细节可能不准确。作者添加了一个脚本，下载引用页面，验证引用文本中的每个单词和数字是否确实出现在页面上，并检查声明中的数字是否与引用中的数字匹配。未通过检查的声明将被标记为“未机器检查”，提高了报告的可靠性。

问题

Research agents often generate claims with linked sources that are not accurate, leading to misinformation.

结果

The agent now reliably flags inaccurate claims, preventing fabricated quotes and mismatched numbers, and clearly indicates which claims are not machine‑checked.

工作流

• 1. Researcher searches and fetches pages, outputs a citations file.
• 2. Writer drafts the brief from citations, no web access.
• 3. Fact‑Checker fetches cited pages but cannot search, reads each linked page.
• 4. Fact‑Checker verifies each claim by matching the quoted text against the page using a case‑insensitive, punctuation‑agnostic comparison.
• 5. If a quote is missing or altered, the report refuses to build the claim or marks it as not machine‑checked.
• 6. Added a numbers check to ensure figures in the claim appear in the quote, with tolerance for formatting and rounding.
• 7. Extended support to PDFs and allowed quote fragments in any order.
• 8. Claims that pass all checks are marked verified; otherwise they are marked partial or unverified.

证据

The source often said something slightly different. A different number. A different year. "Experimental" where the brief said "default".

来源

The Fact‑Checker having no search tool is deliberate. If it could search, it would "verify" a claim by finding some page that agrees.

来源

If a quote isn’t on the page, the report refuses to build. If a page can’t be downloaded, the claim is marked not machine‑checked instead of quietly passing.

来源

v1.2 adds a numbers check: every figure in a "verified" claim must also appear in its quote.

来源