Your AI Reviewer Disagrees With Itself. Measure the Drift.

Описание и применение

Проба измеряет нестабильность AI‑ревьюера, показывая, что одни и те же PR‑ы дают разные результаты, и предлагает добавить проверку стабильности в CI.

Задача

AI code reviewers produce inconsistent findings for the same pull request, leading to flaky gates and missed bugs.

Результат

The probe consistently shows non‑deterministic findings, with overlap rates between 60% and 90%, proving that free model endpoints are not stable and that CI gates must account for drift.

Рабочий процесс

• 1. Save the provided 15‑line Python script as review_drift.py.
• 2. Run the script twice against the same diff using an OpenAI‑compatible endpoint, supplying api‑url, api‑key, and model name.
• 3. Capture the number of findings from each run and compute the overlap percentage.
• 4. Analyze overlap (60‑90%) to assess reviewer stability.
• 5. Use the drift metric to decide whether to add redundancy or adjust thresholds in the CI pipeline.

Доказательства

two runs, one diff, different findings. Happens more often than you want.

Источник

my overlap rate between two runs lands between 60% and 90%. A whole slice of findings exists in only one run.

Источник