I Benchmarked a Free AI Code Reviewer. It Caught 23 of 40 Bugs and Invented 6.

Описание и применение

Бесплатный AI‑ревьюер обнаружил 23 из 40 дефектов в контролируемом наборе тестов, но создал 6 ложных предупреждений. Добавление фильтра, проверяющего существование символов, устранило большинство ошибок без потери истинных находок.

Задача

Assessing the effectiveness of a free AI code reviewer on a controlled set of defects.

Результат

The free AI reviewer caught 23 of 40 injected defects (57%) and produced 6 hallucinated findings. Adding a symbol‑existence filter eliminated most hallucinations while preserving true positives. Latency averaged 9 s per review.

Рабочий процесс

• 1. Prepare 40 diffs each containing one injected defect in a single file. 2. Store defect metadata (id, diff path, line number) in defects.json. 3. Write a harness (bench_reviewer.py) that reads each diff, sends it to the AI model via MonkeyCode’s free tier, and collects findings. 4. Score each case by matching the reported line to the ground truth and count false positives. 5. Run the harness overnight on the free server. 6. Analyze results: 23 hits, 17 misses, 6 hallucinations. 7. Add a deterministic post‑filter that discards findings referencing non‑existent symbols. 8. Re‑run to confirm hallucinations are reduced without affecting true positives.

Доказательства

23 of 40. The free model was excellent at defects that look like typos.

Источник

I added one. A deterministic post-filter: every finding must reference a symbol that actually exists in the diff or the file.

Источник