How I stopped my research agent from inventing its sources

Description and use

Research agents often produce claims with linked sources that are inaccurate. The author added a script that downloads cited pages, verifies that every word and number in the quoted text appears on the page, and checks that figures in the claim match those in the quote. Claims that fail are marked not machine‑checked, improving report reliability.

Problem

Research agents often generate claims with linked sources that are not accurate, leading to misinformation.

Outcome

The agent now reliably flags inaccurate claims, preventing fabricated quotes and mismatched numbers, and clearly indicates which claims are not machine‑checked.

Workflow

• 1. Researcher searches and fetches pages, outputs a citations file.
• 2. Writer drafts the brief from citations, no web access.
• 3. Fact‑Checker fetches cited pages but cannot search, reads each linked page.
• 4. Fact‑Checker verifies each claim by matching the quoted text against the page using a case‑insensitive, punctuation‑agnostic comparison.
• 5. If a quote is missing or altered, the report refuses to build the claim or marks it as not machine‑checked.
• 6. Added a numbers check to ensure figures in the claim appear in the quote, with tolerance for formatting and rounding.
• 7. Extended support to PDFs and allowed quote fragments in any order.
• 8. Claims that pass all checks are marked verified; otherwise they are marked partial or unverified.

Evidence

The source often said something slightly different. A different number. A different year. "Experimental" where the brief said "default".

Source

The Fact‑Checker having no search tool is deliberate. If it could search, it would "verify" a claim by finding some page that agrees.

Source

If a quote isn’t on the page, the report refuses to build. If a page can’t be downloaded, the claim is marked not machine‑checked instead of quietly passing.

Source

v1.2 adds a numbers check: every figure in a "verified" claim must also appear in its quote.

Source