48-Hour Field Notes: A Free AI Server as My PR Reviewer

Description and use

The author tested a free AI server for code review, finding it misses bugs due to context limits, produces false positives, and has variable latency. He recommends using a custom bug list and multiple runs to evaluate effectiveness.

Problem

Can a free AI coding server reliably review pull requests and catch real bugs in a repo?

Outcome

The AI reviewer detected 2/5 bugs in a small diff, 1/5 in a large refactor, and 3/5 with tests, but also produced false positives and suffered from context window limits and latency drift.

Workflow

• 1. Seed a repo with known bugs and record their file:line locations.
• 2. Write a bash harness to send the same diff to the AI server five times, recording latency and response.
• 3. Score each run by matching reported bugs against the ground truth file.
• 4. Analyze hit/miss rates, false positives, and latency variations.
• 5. Document lessons learned and repeat the process for different diff sizes.

Evidence

The harness measured how fast the server responded and whether it found the seeded bug inside the changed lines.

Source

The third break was the quiet one: false confidence. On the run with tests attached, the server found three of five bugs and invented two more.

Source