A Noise Budget for Automated PR Reviews: Free Models, a Free Server, and a Protocol That Keeps Humans Listening

Description and use

The noise‑budget protocol limits an AI PR reviewer to blocker comments up to a cap, restoring trust and keeping human reviewers focused while incurring near‑zero cost.

Problem

Automated PR reviewers that comment on everything erode trust and overwhelm human reviewers, leading to rapid abandonment of the bot.

Outcome

The protocol reduces noise to a manageable level, preserves bot credibility, and keeps human reviewers engaged while incurring near‑zero cost.

Workflow

• 1. Deploy a free model on a free server (e.g., MonkeyCode).
• 2. Configure a noise budget: set a maximum number of blocker comments per PR (NOISE_CAP).
• 3. Run the GitHub Action on pull_request events.
• 4. The Action fetches the base branch, calls the review endpoint, and pipes output to the severity‑gated runner.
• 5. The runner filters comments by severity, caps blocker comments, and outputs them in a checklist format.
• 6. Lower‑severity findings are added to the PR description as a quiet summary.
• 7. The bot’s comments are prefixed with its identity for auditability.
• 8. Track false positives in logs to adjust the noise budget over time.

Evidence

The silent bot costs a token budget and nothing else; the chatty one trains a whole team to ignore every flag it raises, including the genuine blockers.

Source

The runner consumes the model's raw review as JSON and reduces it to a short, accountable list.

Source