Your Generated Docs Passed Lint. Did Any Example Actually Run?

说明与用途

一种工作流，提取、执行并变异文档中的每个代码块，确保示例可运行并具有断言。

问题

Documentation pipelines often pass linting and link checks but fail to verify that code examples actually run and produce expected results.

结果

All code examples are verified to run and assert something, reducing runtime failures in released documentation.

工作流

• Draft prose and examples with a free AI model (MonkeyCode).
• Extract every fenced code block from the markdown.
• Execute each block in a clean subprocess with a timeout.
• Mutate each block by changing an assignment and re‑run to ensure it fails if no assertion.
• Have a human review claims that automation cannot verify (e.g., version pins, security statements).
• Publish the documentation with a gate report summarizing execution results.

证据

Execute every fenced code block in a Markdown file, then mutate each block to prove the example actually asserts something.

来源

The expensive part is the human review, and that is where the value lives.

来源