Your Docs Gate Has Never Failed, So It Proves Nothing: Mutation-Testing a Model-Drafted Page

Описание и применение

Гейт проверки документации, который всегда выдаёт «пройдено», не выявляет дефекты, добавляемые AI. В статье описывается метод мутационного тестирования, который вводит известные ошибки в прошедший черновик и проверяет, что гейт отклоняет каждый вариант. Это позволяет обнаружить слабые места правил гейта и гарантировать, что критический контент не удаляется без проверки.

Задача

Documentation gates that only report pass fail silently miss defects introduced by AI drafting, allowing erroneous content to ship.

Результат

The mutation‑testing approach exposes blind spots in documentation gates, ensuring that AI‑drafted pages cannot silently delete critical content or introduce unverified changes.

Рабочий процесс

• 1. Define ownership split: mark human‑owned blocks and encode machine‑checkable rules.
• 2. Create a facts file listing tested commands, versions, and required human phrases.
• 3. Implement a lightweight Python gate that checks fenced code blocks, command coverage, version registration, human‑owned block presence, and required phrases.
• 4. Write a mutation harness that applies one defect at a time (e.g., delete a safety caveat, swap a version, empty a human block).
• 5. Run the gate on each mutated draft and assert a non‑zero exit for every mutation.
• 6. Use the mutation sheet to identify which gate rules need strengthening.
• 7. Iterate: update facts or gate logic until all mutations fail.
• 8. Integrate the gate into the review queue so that AI‑drafted pages must pass the mutation‑tested gate before human review.

Доказательства

A documentation gate that has only ever printed pass is not evidence of quality; it is evidence that nobody has tried to break it.

Источник

Mutation testing gives you a cheap way to measure that blindness.

Источник

The gate reads one draft page and one facts file, and it does not care which model wrote the prose.

Источник