claimcheck: Gemma 4 fact-checks my AI blog drafts against their own sources

Description and use

claimcheck uses Gemma 4 to fact‑check AI‑generated drafts against their own sources, generating a report and blocking publication when issues are found.

Problem

AI‑generated drafts can contain factual errors that mislead readers, and existing fact‑checking tools are often black boxes or require paid APIs.

Outcome

claimcheck detected a false claim about Python 3.15, flagged a false alarm about model token limits, and showed that many claims in a popular article were unsupported due to poor linking. The tool prevented publishing of the erroneous draft and highlighted areas needing better source links.

Workflow

• Extract claims from the draft using Gemma 4, ensuring each claim quotes the draft.
• Retrieve every linked source page and split it into passages.
• For each claim, use a lexical index to select the closest passages and ask Gemma 4 to judge the claim as supported, contradicted, or not_found, returning an evidence sentence.
• Verify the evidence sentence exists verbatim in the source and that all numbers in the claim appear in the evidence or surrounding text.
• Generate a report listing problems first, show draft text next to source text, and exit with code 1 if any claim is not supported.
• Use the exit code to block CI or publishing steps.
• Optional: run claimcheck on old posts to re‑check as sources change.

Evidence

The stdlib now returns frozendict from EnumType.__members__ and errno.errorcode instead of MappingProxyType or dict.

Source

Gemma 4 lists the checkable claims in the draft (numbers, versions, limits, defaults, "X supports Y") and has to quote the draft for each one.

Source

A full article took 3 to 5 minutes and about 70k input tokens. At DigitalOcean's listed Gemma 4 serverless rate ($0.18 per 1M input tokens and $0.50 per 1M output tokens as of October 2026), that is about 1.4 cents per article.

Source