Ban Naked Numbers: A Token Gate for AI-Drafted Docs

Description and use

The article presents an automation recipe that blocks naked numeric values in AI‑generated documentation using a token gate and a fact registry.

Problem

AI‑generated documentation often inserts incorrect numeric constants, causing costly errors for users.

Outcome

All numeric claims in AI‑drafted docs are now traceable to a verified source, eliminating accidental mis‑documentation.

Workflow

• 1. Create a facts.json registry with value, unit, source, verified_at, and owner for each numeric claim.
• 2. Prompt the model to emit only tokens like {{fact:queue.backoff_ms}} instead of raw numbers.
• 3. Run claimcheck.py in two stages: draft and rendered, to ensure every token resolves to a registered fact and no naked numbers remain.
• 4. Fail the PR if any unregistered numeric claim is detected.

Evidence

A docs PR landed in March with one changed sentence: the queue client's retry backoff was now "250 ms".

Source

The prompt contract is short: write Markdown, never write a number, version, or URL directly — emit a token instead.

Source

claimcheck.py is stdlib‑only and runs in two stages. Stage one scans the raw draft; stage two scans the rendered page and confirms every surviving claim traces back to a registered value.

Source