Your AI Reviewer Disagrees With Itself. Measure the Drift.

说明与用途

漂移探针显示 AI 评审不稳定，说明相同 PR 产生不同结果，建议在 CI 中加入稳定性检查。

问题

AI code reviewers produce inconsistent findings for the same pull request, leading to flaky gates and missed bugs.

结果

The probe consistently shows non‑deterministic findings, with overlap rates between 60% and 90%, proving that free model endpoints are not stable and that CI gates must account for drift.

工作流

• 1. Save the provided 15‑line Python script as review_drift.py.
• 2. Run the script twice against the same diff using an OpenAI‑compatible endpoint, supplying api‑url, api‑key, and model name.
• 3. Capture the number of findings from each run and compute the overlap percentage.
• 4. Analyze overlap (60‑90%) to assess reviewer stability.
• 5. Use the drift metric to decide whether to add redundancy or adjust thresholds in the CI pipeline.

证据

two runs, one diff, different findings. Happens more often than you want.

来源

my overlap rate between two runs lands between 60% and 90%. A whole slice of findings exists in only one run.

来源