48-Hour Field Notes: A Free AI Server as My PR Reviewer

说明与用途

作者测试了免费 AI 代码审查服务器，发现它因上下文限制而漏报错误，产生误报，并且延迟不稳定。他建议使用自定义错误列表和多次运行来评估其效果。

问题

Can a free AI coding server reliably review pull requests and catch real bugs in a repo?

结果

The AI reviewer detected 2/5 bugs in a small diff, 1/5 in a large refactor, and 3/5 with tests, but also produced false positives and suffered from context window limits and latency drift.

工作流

• 1. Seed a repo with known bugs and record their file:line locations.
• 2. Write a bash harness to send the same diff to the AI server five times, recording latency and response.
• 3. Score each run by matching reported bugs against the ground truth file.
• 4. Analyze hit/miss rates, false positives, and latency variations.
• 5. Document lessons learned and repeat the process for different diff sizes.

证据

The harness measured how fast the server responded and whether it found the seeded bug inside the changed lines.

来源

The third break was the quiet one: false confidence. On the run with tests attached, the server found three of five bugs and invented two more.

来源