Your AI Reviewer Disagrees With Itself. Measure the Drift.

Description and use

The drift probe demonstrates AI reviewer instability, showing that identical PRs yield different findings, and recommends adding stability checks to CI.

Problem

AI code reviewers produce inconsistent findings for the same pull request, leading to flaky gates and missed bugs.

Outcome

The probe consistently shows non‑deterministic findings, with overlap rates between 60% and 90%, proving that free model endpoints are not stable and that CI gates must account for drift.

Workflow

• 1. Save the provided 15‑line Python script as review_drift.py.
• 2. Run the script twice against the same diff using an OpenAI‑compatible endpoint, supplying api‑url, api‑key, and model name.
• 3. Capture the number of findings from each run and compute the overlap percentage.
• 4. Analyze overlap (60‑90%) to assess reviewer stability.
• 5. Use the drift metric to decide whether to add redundancy or adjust thresholds in the CI pipeline.

Evidence

two runs, one diff, different findings. Happens more often than you want.

Source

my overlap rate between two runs lands between 60% and 90%. A whole slice of findings exists in only one run.

Source