48-Hour Field Notes: A Free AI Server as My PR Reviewer

Описание и применение

Автор протестировал бесплатный AI-сервер для ревью кода, обнаружив, что он иногда пропускает ошибки из-за ограничений окна контекста, генерирует ложные срабатывания и имеет непостоянную задержку. Он рекомендует использовать собственный список ошибок и многократные тесты для оценки эффективности.

Задача

Can a free AI coding server reliably review pull requests and catch real bugs in a repo?

Результат

The AI reviewer detected 2/5 bugs in a small diff, 1/5 in a large refactor, and 3/5 with tests, but also produced false positives and suffered from context window limits and latency drift.

Рабочий процесс

• 1. Seed a repo with known bugs and record their file:line locations.
• 2. Write a bash harness to send the same diff to the AI server five times, recording latency and response.
• 3. Score each run by matching reported bugs against the ground truth file.
• 4. Analyze hit/miss rates, false positives, and latency variations.
• 5. Document lessons learned and repeat the process for different diff sizes.

Доказательства

The harness measured how fast the server responded and whether it found the seeded bug inside the changed lines.

Источник

The third break was the quiet one: false confidence. On the run with tests attached, the server found three of five bugs and invented two more.

Источник