I Benchmarked a Free AI Code Reviewer. It Caught 23 of 40 Bugs and Invented 6.

说明与用途

免费 AI 代码审查器在 40 个注入缺陷的基准测试中发现了 23 个缺陷，但产生了 6 个幻觉。添加符号存在性过滤器消除了大多数错误，同时保留了真实发现。

问题

Assessing the effectiveness of a free AI code reviewer on a controlled set of defects.

结果

The free AI reviewer caught 23 of 40 injected defects (57%) and produced 6 hallucinated findings. Adding a symbol‑existence filter eliminated most hallucinations while preserving true positives. Latency averaged 9 s per review.

工作流

• 1. Prepare 40 diffs each containing one injected defect in a single file. 2. Store defect metadata (id, diff path, line number) in defects.json. 3. Write a harness (bench_reviewer.py) that reads each diff, sends it to the AI model via MonkeyCode’s free tier, and collects findings. 4. Score each case by matching the reported line to the ground truth and count false positives. 5. Run the harness overnight on the free server. 6. Analyze results: 23 hits, 17 misses, 6 hallucinations. 7. Add a deterministic post‑filter that discards findings referencing non‑existent symbols. 8. Re‑run to confirm hallucinations are reduced without affecting true positives.

证据

23 of 40. The free model was excellent at defects that look like typos.

来源

I added one. A deterministic post-filter: every finding must reference a symbol that actually exists in the diff or the file.

来源