Budget Calibration: How to Test Whether Your AI Reviewer Still Works Before You Trust It

Description and use

Calibrate your LLM code reviewer by running a small labeled set of patches against it, aggregating verdicts, and comparing to a baseline report to detect drift before merges are affected.

Problem

LLM-based code reviewers drift over time due to model updates and prompt changes, causing inconsistent patch approvals or rejections without developers noticing.

Outcome

The calibration harness provides a cheap, repeatable sanity check that catches reviewer drift before it impacts merge queues, preventing costly backlogs and missed bugs.

Workflow

• Build a small labeled set of 10‑12 patches with gold approve/reject labels.
• Define a deterministic grader prompt that forces JSON output with a verdict field.
• Implement a Python script that sends each patch to the reviewer endpoint, runs each case three times, and aggregates majority verdicts.
• Run the script to generate a baseline report and commit it to the repository.
• Periodically rerun the harness after model or prompt changes to detect drift before merges are affected.

Evidence

A calibration harness does to your reviewer what golden tests do to your generator: it feeds it known inputs and checks that it still produces known verdicts.

Source

Ten to twelve patch snippets are enough for a smoke test.

Source