Budget Calibration: How to Test Whether Your AI Reviewer Still Works Before You Trust It

Описание и применение

Проверка смещения LLM‑ревьюера: создайте набор меток, запустите скрипт, сравните результаты с базовым отчётом, чтобы вовремя обнаружить изменения поведения модели.

Задача

LLM-based code reviewers drift over time due to model updates and prompt changes, causing inconsistent patch approvals or rejections without developers noticing.

Результат

The calibration harness provides a cheap, repeatable sanity check that catches reviewer drift before it impacts merge queues, preventing costly backlogs and missed bugs.

Рабочий процесс

• Build a small labeled set of 10‑12 patches with gold approve/reject labels.
• Define a deterministic grader prompt that forces JSON output with a verdict field.
• Implement a Python script that sends each patch to the reviewer endpoint, runs each case three times, and aggregates majority verdicts.
• Run the script to generate a baseline report and commit it to the repository.
• Periodically rerun the harness after model or prompt changes to detect drift before merges are affected.

Доказательства

A calibration harness does to your reviewer what golden tests do to your generator: it feeds it known inputs and checks that it still produces known verdicts.

Источник

Ten to twelve patch snippets are enough for a smoke test.

Источник