Budget Calibration: How to Test Whether Your AI Reviewer Still Works Before You Trust It
说明与用途
使用标记集校准LLM代码审阅器,生成基线报告,定期比较以检测模型漂移,防止合并队列受影响。
问题
LLM-based code reviewers drift over time due to model updates and prompt changes, causing inconsistent patch approvals or rejections without developers noticing.
结果
The calibration harness provides a cheap, repeatable sanity check that catches reviewer drift before it impacts merge queues, preventing costly backlogs and missed bugs.
工作流
- Build a small labeled set of 10‑12 patches with gold approve/reject labels.
- Define a deterministic grader prompt that forces JSON output with a verdict field.
- Implement a Python script that sends each patch to the reviewer endpoint, runs each case three times, and aggregates majority verdicts.
- Run the script to generate a baseline report and commit it to the repository.
- Periodically rerun the harness after model or prompt changes to detect drift before merges are affected.
证据
A calibration harness does to your reviewer what golden tests do to your generator: it feeds it known inputs and checks that it still produces known verdicts.
Ten to twelve patch snippets are enough for a smoke test.
- 记录日期: 2026-09-09T10:45:15.453Z