Treat Model Comments as Named Hypotheses in an OSS Review

Description and use

Integrating AI into code review requires constraining its output to testable hypotheses, preventing unverified changes and speeding human oversight.

Problem

Chat‑authored model diffs introduce unverified changes, hidden helpers, and missing tests, causing reviewers to waste time proving patches are real and leading to stalled reviews.

Outcome

The workflow enforces that AI model contributions are hypothesis‑driven, testable, and confined to a small, allowlisted set of files, eliminating unverified changes and speeding human review.

Workflow

• Detach at a recorded upstream SHA and record base SHA
• Pin the public issue URL and allowlist specific paths
• Write a failing test on the frozen SHA and record repro exit code
• Author minimal production edits by hand and record allowlisted diff
• Re‑run repro and smoke commands, recording exit codes
• Build a secret‑stripped review bundle and verify no credentials
• Ask the model for hypotheses only, writing output to hypotheses.md
• Accept, convert, or discard each hypothesis based on allowlist and testability
• Open the pull request with the contract, hypothesis log, and minimal changes

Evidence

Treat every model comment as a named hypothesis. Require a frozen path and a testable claim.

Source

The model never writes files in the working tree. The model never receives tokens or private mail.

Source