LLM Code Reviews Are Not What You Think - An Opinionated Take on AI-Assisted Quality

说明与用途

LLM 评审发现日志级别不匹配、缺失 @Transactional 注解和日志中的 PII，但未能捕捉到 JSON 字段名更改等语义合同问题。

问题

Human reviewers struggled to consistently detect repetitive, cross‑service code quality issues in a large microservices codebase.

结果

The LLM reviewer identified log‑level mismatches, missing transaction boundaries, PII in logs, and dead code more consistently than human reviewers, while false positives were reduced by adding known‑pattern sections to the prompt.

工作流

• PR opened on GitHub
• GitHub Action triggered
• Diff extraction (changed lines + 30 lines context)
• Prompt assembly (diff + review rules + file metadata)
• Send prompt to Claude via Bedrock
• Claude returns findings
• Post inline comments on PR
• Human reviewer accepts or dismisses comments

证据

We introduced LLM reviewer to our Java/Spring Boot PR process several months ago.

来源

The main design choices: 1. Diff only, not the whole file. 2. File information is valuable. 3. Review criteria are clear.

来源

This was seen in multiple services. All were human-approved. All were detected by the LLM. There were no false positives here.

来源

The bug was caught manually by a junior engineer on our team since she had worked on the downstream consumer previously.

来源