Your Generated Docs Passed Lint. Did Any Example Actually Run?

Описание и применение

Пайплайн проверяет, что все примеры кода в документации действительно работают и делают что‑то, а не просто компилируются.

Задача

Documentation pipelines often pass linting and link checks but fail to verify that code examples actually run and produce expected results.

Результат

All code examples are verified to run and assert something, reducing runtime failures in released documentation.

Рабочий процесс

• Draft prose and examples with a free AI model (MonkeyCode).
• Extract every fenced code block from the markdown.
• Execute each block in a clean subprocess with a timeout.
• Mutate each block by changing an assignment and re‑run to ensure it fails if no assertion.
• Have a human review claims that automation cannot verify (e.g., version pins, security statements).
• Publish the documentation with a gate report summarizing execution results.

Доказательства

Execute every fenced code block in a Markdown file, then mutate each block to prove the example actually asserts something.

Источник

The expensive part is the human review, and that is where the value lives.

Источник