claimcheck: Gemma 4 fact-checks my AI blog drafts against their own sources

Описание и применение

claimcheck использует модель Gemma 4 для проверки фактов в AI‑генерируемых черновиках, сравнивая их с источниками, указанными в тексте, и блокирует публикацию при обнаружении ошибок.

Задача

AI‑generated drafts can contain factual errors that mislead readers, and existing fact‑checking tools are often black boxes or require paid APIs.

Результат

claimcheck detected a false claim about Python 3.15, flagged a false alarm about model token limits, and showed that many claims in a popular article were unsupported due to poor linking. The tool prevented publishing of the erroneous draft and highlighted areas needing better source links.

Рабочий процесс

• Extract claims from the draft using Gemma 4, ensuring each claim quotes the draft.
• Retrieve every linked source page and split it into passages.
• For each claim, use a lexical index to select the closest passages and ask Gemma 4 to judge the claim as supported, contradicted, or not_found, returning an evidence sentence.
• Verify the evidence sentence exists verbatim in the source and that all numbers in the claim appear in the evidence or surrounding text.
• Generate a report listing problems first, show draft text next to source text, and exit with code 1 if any claim is not supported.
• Use the exit code to block CI or publishing steps.
• Optional: run claimcheck on old posts to re‑check as sources change.

Доказательства

The stdlib now returns frozendict from EnumType.__members__ and errno.errorcode instead of MappingProxyType or dict.

Источник

Gemma 4 lists the checkable claims in the draft (numbers, versions, limits, defaults, "X supports Y") and has to quote the draft for each one.

Источник

A full article took 3 to 5 minutes and about 70k input tokens. At DigitalOcean's listed Gemma 4 serverless rate ($0.18 per 1M input tokens and $0.50 per 1M output tokens as of October 2026), that is about 1.4 cents per article.

Источник