Your Docs Gate Has Never Failed, So It Proves Nothing: Mutation-Testing a Model-Drafted Page

说明与用途

一个始终通过的文档门无法检测 AI 草稿引入的缺陷。文章介绍了一种突变测试方法，将已通过的草稿注入已知错误，并验证门是否对每个突变失败，从而发现门规则的盲点，确保关键内容不会被悄悄删除。

问题

Documentation gates that only report pass fail silently miss defects introduced by AI drafting, allowing erroneous content to ship.

结果

The mutation‑testing approach exposes blind spots in documentation gates, ensuring that AI‑drafted pages cannot silently delete critical content or introduce unverified changes.

工作流

• 1. Define ownership split: mark human‑owned blocks and encode machine‑checkable rules.
• 2. Create a facts file listing tested commands, versions, and required human phrases.
• 3. Implement a lightweight Python gate that checks fenced code blocks, command coverage, version registration, human‑owned block presence, and required phrases.
• 4. Write a mutation harness that applies one defect at a time (e.g., delete a safety caveat, swap a version, empty a human block).
• 5. Run the gate on each mutated draft and assert a non‑zero exit for every mutation.
• 6. Use the mutation sheet to identify which gate rules need strengthening.
• 7. Iterate: update facts or gate logic until all mutations fail.
• 8. Integrate the gate into the review queue so that AI‑drafted pages must pass the mutation‑tested gate before human review.

证据

A documentation gate that has only ever printed pass is not evidence of quality; it is evidence that nobody has tried to break it.

来源

Mutation testing gives you a cheap way to measure that blindness.

来源

The gate reads one draft page and one facts file, and it does not care which model wrote the prose.

来源