A publish gate for AI-assisted posts: check every claim before it goes out

Description and use

A publish gate using a facts file and an LLM prevents the spread of unverified claims in AI‑generated posts.

Problem

AI‑generated posts can contain unverified numbers, unmeasured claims, or stock phrases that mislead readers.

Outcome

The gate caught unverified numbers, unmeasured claims, and stock phrases, preventing potentially misleading posts from being published. It also identified a publisher bug when a Show HN post was incorrectly marked as published.

Workflow

• Create facts.md with evidence for every claim and a not‑claimable section
• Define deterministic rules for numbers, phrases, structure, and links
• Run deterministic checks before model involvement
• Send draft and facts.md to an LLM for claim verdicts (SUPPORTED/UNSUPPORTED/CONTRADICTED)
• If all claims are SUPPORTED, write a sidecar file with SHA‑256 of the post body
• Publisher script calls require_verified() to ensure sidecar matches and LLM step ran
• Publish only if verification passes; otherwise refuse and log the failure

Evidence

Every claim and every number a post may use lives in facts.md, each with its evidence: a file path, a command and its output, or a URL.

Source

The last check sends the draft and the facts file to a model and asks for a verdict per claim: SUPPORTED, UNSUPPORTED or CONTRADICTED.

Source

A PASS writes a small sidecar file with the SHA-256 of the post body.

Source