Review the Reviewer: A Risk-Score Harness for Agent-Generated PRs

Описание и применение

Модельный инструмент автоматически классифицирует каждый блок изменений в PR, созданном AI‑агентом, как «доверять», «отменить» или «тестировать», предоставляя оценку риска и ускоряя ручной обзор.

Задача

AI agents generate many PRs, but their diffs can contain subtle errors such as tautological tests, silent fallbacks, or scope creep, making manual review time-consuming and error-prone.

Результат

The harness processes a diff in seconds, classifying each hunk and producing a risk score that helps reviewers prioritize manual effort, thereby scaling review capacity for agent-generated PRs.

Рабочий процесс

• 1. Run the provided Python script with a unified diff file and a local model endpoint (e.g., MonkeyCode free server).
• 2. The script splits the diff into hunks.
• 3. For each hunk, it sends a prompt to the model endpoint asking for a JSON classification (trust/revert/test) and a short reason.
• 4. The script aggregates counts of each classification to compute a simple risk score.
• 5. Reviewers use the score and reasons to decide whether to trust, revert, or manually test the hunk.

Доказательства

This article shows a lightweight meta-review harness that scores each hunk of an agent-generated diff and flags what to trust, what to revert, and what to test.

Источник

It uses a free model server from MonkeyCode so you can run it without a cloud budget.

Источник

I’m not going to quote token quotas or benchmark numbers because those change and I haven’t verified them. What I can say is that the free tier was enough to run this prototype against a few hundred hunks without hitting a wall in testing.

Источник