LLM Code Reviews Are Not What You Think - An Opinionated Take on AI-Assisted Quality

Description and use

The LLM reviewer detected log‑level mismatches, missing @Transactional annotations, and PII in logs, but missed semantic contract issues such as JSON field name changes.

Problem

Human reviewers struggled to consistently detect repetitive, cross‑service code quality issues in a large microservices codebase.

Outcome

The LLM reviewer identified log‑level mismatches, missing transaction boundaries, PII in logs, and dead code more consistently than human reviewers, while false positives were reduced by adding known‑pattern sections to the prompt.

Workflow

• PR opened on GitHub
• GitHub Action triggered
• Diff extraction (changed lines + 30 lines context)
• Prompt assembly (diff + review rules + file metadata)
• Send prompt to Claude via Bedrock
• Claude returns findings
• Post inline comments on PR
• Human reviewer accepts or dismisses comments

Evidence

We introduced LLM reviewer to our Java/Spring Boot PR process several months ago.

Source

The main design choices: 1. Diff only, not the whole file. 2. File information is valuable. 3. Review criteria are clear.

Source

This was seen in multiple services. All were human-approved. All were detected by the LLM. There were no false positives here.

Source

The bug was caught manually by a junior engineer on our team since she had worked on the downstream consumer previously.

Source