Ban Naked Numbers: A Token Gate for AI-Drafted Docs

Описание и применение

В статье описывается автоматизированный рецепт, позволяющий блокировать «голые» числовые значения в документации, генерируемой ИИ, через токен‑гейт и реестр фактов.

Задача

AI‑generated documentation often inserts incorrect numeric constants, causing costly errors for users.

Результат

All numeric claims in AI‑drafted docs are now traceable to a verified source, eliminating accidental mis‑documentation.

Рабочий процесс

• 1. Create a facts.json registry with value, unit, source, verified_at, and owner for each numeric claim.
• 2. Prompt the model to emit only tokens like {{fact:queue.backoff_ms}} instead of raw numbers.
• 3. Run claimcheck.py in two stages: draft and rendered, to ensure every token resolves to a registered fact and no naked numbers remain.
• 4. Fail the PR if any unregistered numeric claim is detected.

Доказательства

A docs PR landed in March with one changed sentence: the queue client's retry backoff was now "250 ms".

Источник

The prompt contract is short: write Markdown, never write a number, version, or URL directly — emit a token instead.

Источник

claimcheck.py is stdlib‑only and runs in two stages. Stage one scans the raw draft; stage two scans the rendered page and confirms every surviving claim traces back to a registered value.

Источник