Pin Your AI Review Configuration Like a Dependency

Description and use

Pin AI review configuration as a dependency: store model, prompt, sampling parameters, and diff context in the repo so each run is reproducible, and keep artifacts in CI for audit.

Problem

AI-assisted code reviews lose trust when the model, prompt, and diff cannot be reproduced, making it hard to audit or compare results across model updates.

Outcome

By pinning AI review configurations, teams gain an audit trail, can detect regressions caused by model updates, and make informed merge decisions based on reproducible evidence.

Workflow

• 1. Create a pin file (JSON) with model, prompt_version, temperature, max_tokens, and rules.
• 2. Store the pin file alongside the code in the repo.
• 3. Run the review harness (e.g., review_harness.sh) passing the diff and pin file.
• 4. Capture the review output artifact (JSON) and fingerprint.
• 5. Store the artifact in CI for audit and future comparison.
• 6. Re-run the harness after model or prompt changes to compare fingerprints and outputs.
• 7. Use the decision table to handle review comments based on reproducibility.

Evidence

AI-assisted code review only earns trust when the exact model, prompt, and diff can be replayed from a stored configuration.

Source

Store the configuration in the repository so the review is tied to the code it evaluated, not to a developer's memory.

Source