I Built a Benchmark That Grades AI Code Reviewers on Planted Defects

Description and use

A new benchmark, Red‑Pen Bench, was built to evaluate AI code reviewers on defect detection, localization, explanation, and false‑positive rates using planted defects across difficulty tiers.

Problem

Existing benchmarks do not isolate defect detection, localization, explanation, and false‑positive measurement for AI code reviewers.

Outcome

Gemini 3.7 Flash achieved the highest overall score (0.799), Claude Haiku 5.5 was most consistent across tiers, GPT‑5.4 mini showed a large hard‑versus‑easy gap, and DeepSeek‑R1 performed below the always‑no baseline.

Workflow

• 1. Define 44 planted defects from mutation‑testing operators (AOR, ROR, UOI, LCR, statement‑deletion, higher‑order mutants).
• 2. Create four variants per defect: original, disguised, fixed, clean.
• 3. Verify killability with automated tests that fail on defect variants and pass on fixed/clean.
• 4. Publish benchmark on Kaggle and run four AI models (Gemini 3.7 Flash, Claude Haiku 5.5, GPT‑5.4 mini, DeepSeek‑R1) plus baselines.
• 5. Score models automatically using detection rate, line‑level F1, keyword recall, and false‑positive rate.
• 6. Compute tier score as (detection + line‑F1 + keyword‑recall)/3 – false‑positive rate.
• 7. Report results and analyze performance across tiers.

Evidence

I spent the last month running fourteen independent audit rounds on a clean-room codebase.

Source

Red‑Pen Bench was created, containing 44 distinct planted defects across three difficulty tiers.

Source

Models must output three things: DEFECT: yes/no, LINES:, EXPLANATION:.

Source

Gemini 3.7 Flash led overall, Claude Haiku 5.5 was the most consistent across tiers, GPT‑5.4 mini showed the largest observed hard‑versus‑easy gap, and DeepSeek‑R1 finished below the always‑no baseline.

Source