I Benchmarked a Free AI Code Reviewer. It Caught 23 of 40 Bugs and Invented 6.

Description and use

A free AI code reviewer caught 23 of 40 injected defects but produced 6 hallucinations. Adding a symbol‑existence filter removed most false positives while preserving true positives.

Problem

Assessing the effectiveness of a free AI code reviewer on a controlled set of defects.

Outcome

The free AI reviewer caught 23 of 40 injected defects (57%) and produced 6 hallucinated findings. Adding a symbol‑existence filter eliminated most hallucinations while preserving true positives. Latency averaged 9 s per review.

Workflow

• 1. Prepare 40 diffs each containing one injected defect in a single file. 2. Store defect metadata (id, diff path, line number) in defects.json. 3. Write a harness (bench_reviewer.py) that reads each diff, sends it to the AI model via MonkeyCode’s free tier, and collects findings. 4. Score each case by matching the reported line to the ground truth and count false positives. 5. Run the harness overnight on the free server. 6. Analyze results: 23 hits, 17 misses, 6 hallucinations. 7. Add a deterministic post‑filter that discards findings referencing non‑existent symbols. 8. Re‑run to confirm hallucinations are reduced without affecting true positives.

Evidence

23 of 40. The free model was excellent at defects that look like typos.

Source

I added one. A deterministic post-filter: every finding must reference a symbol that actually exists in the diff or the file.

Source