How I built an AI code reviewer that knows when to shut up

Description and use

A concise AI code reviewer that reports only significant findings, ranks them by severity, caps low‑priority comments, and transparently reports suppressed items.

Problem

AI code reviewers often produce noisy, irrelevant feedback, causing developers to disable them and miss real bugs.

Outcome

The system reduces noise, surfaces real bugs first, and maintains transparency by reporting suppressed comments, improving developer trust and adoption.

Workflow

• 1. Receive GitHub webhook for pull_request
• 2. Verify signature and immediately return 200
• 3. In background: fetch diff, call Claude to generate JSON findings
• 4. Validate severity against whitelist and sanitize findings
• 5. Stable‑sort findings by severity (BUG>WARN>NIT>PRAISE)
• 6. Slice to MAX_COMMENTS, compute suppressedCount
• 7. Post inline comments; if anchoring fails, fallback to top‑level comment
• 8. Log errors and suppress failures silently

Evidence

GitHub wants a 2XX response within roughly 10 seconds of a webhook delivery.

Source

The model returns JSON findings, each with { path, line, severity, message }

Source