Review the Reviewer: A Risk-Score Harness for Agent-Generated PRs

Description and use

A lightweight harness uses an LLM to classify each diff hunk from an AI-generated PR as trust, revert, or test, producing a risk score that guides reviewers and scales review capacity.

Problem

AI agents generate many PRs, but their diffs can contain subtle errors such as tautological tests, silent fallbacks, or scope creep, making manual review time-consuming and error-prone.

Outcome

The harness processes a diff in seconds, classifying each hunk and producing a risk score that helps reviewers prioritize manual effort, thereby scaling review capacity for agent-generated PRs.

Workflow

• 1. Run the provided Python script with a unified diff file and a local model endpoint (e.g., MonkeyCode free server).
• 2. The script splits the diff into hunks.
• 3. For each hunk, it sends a prompt to the model endpoint asking for a JSON classification (trust/revert/test) and a short reason.
• 4. The script aggregates counts of each classification to compute a simple risk score.
• 5. Reviewers use the score and reasons to decide whether to trust, revert, or manually test the hunk.

Evidence

This article shows a lightweight meta-review harness that scores each hunk of an agent-generated diff and flags what to trust, what to revert, and what to test.

Source

It uses a free model server from MonkeyCode so you can run it without a cloud budget.

Source

I’m not going to quote token quotas or benchmark numbers because those change and I haven’t verified them. What I can say is that the free tier was enough to run this prototype against a few hundred hunks without hitting a wall in testing.

Source