LLM Code Reviews Are Not What You Think - An Opinionated Take on AI-Assisted Quality

Описание и применение

LLM‑ревьюер выявил проблемы с логированием, транзакциями и PII в логах, но не смог обнаружить семантические ошибки API.

Задача

Human reviewers struggled to consistently detect repetitive, cross‑service code quality issues in a large microservices codebase.

Результат

The LLM reviewer identified log‑level mismatches, missing transaction boundaries, PII in logs, and dead code more consistently than human reviewers, while false positives were reduced by adding known‑pattern sections to the prompt.

Рабочий процесс

• PR opened on GitHub
• GitHub Action triggered
• Diff extraction (changed lines + 30 lines context)
• Prompt assembly (diff + review rules + file metadata)
• Send prompt to Claude via Bedrock
• Claude returns findings
• Post inline comments on PR
• Human reviewer accepts or dismisses comments

Доказательства

We introduced LLM reviewer to our Java/Spring Boot PR process several months ago.

Источник

The main design choices: 1. Diff only, not the whole file. 2. File information is valuable. 3. Review criteria are clear.

Источник

This was seen in multiple services. All were human-approved. All were detected by the LLM. There were no false positives here.

Источник

The bug was caught manually by a junior engineer on our team since she had worked on the downstream consumer previously.

Источник