Make the Draft Refuse: An Abstention Ledger for Model-Written Docs

Описание и применение

В workflow LLM генерирует документацию, вставляя маркеры для утверждений, которые нельзя проверить. Скрипт парсит эти маркеры и выявляет скрытые утверждения, создавая таблицу маршрутизации для проверки.

Задача

Model‑written docs often contain silent failures such as incorrect version pins or rate limits that reviewers miss because they look like fluent prose.

Результат

The workflow turns invisible risks into countable objects, enabling systematic review and reducing silent failures in documentation.

Рабочий процесс

• 1. Define a stable marker in the system prompt: <!-- NEEDS id="<short-slug>" owner="<team-or-person>" question="<one question>" -->
• 2. Run the LLM to draft prose, structure, and links, allowing it to emit markers when it cannot cite a fact.
• 3. Parse the draft with a Python script that extracts markers and detects silent assertions (versions, commands, URLs, quantities).
• 4. Generate a routing table mapping each marker to its line number, owner, and question.
• 5. Reviewers act on the table, filling in facts or correcting assertions.
• 6. Re‑run the draft if necessary to eliminate remaining markers.

Доказательства

Instead of asking the model to be cautious in prose, you give it a syntactic obligation: when a claim depends on a fact it cannot cite from your repository, it must emit a marker and stop.

Источник

The parser does two jobs: it extracts every marker, and it finds silent assertions — version numbers, shell commands, URLs, and unit-bearing figures that appear without a marker.

Источник