I Built a Benchmark That Grades AI Code Reviewers on Planted Defects

说明与用途

构建了 Red‑Pen Bench 基准，用于评估 AI 代码审查者在检测、定位、解释人工植入缺陷以及误报率方面的表现。

问题

Existing benchmarks do not isolate defect detection, localization, explanation, and false‑positive measurement for AI code reviewers.

结果

Gemini 3.7 Flash achieved the highest overall score (0.799), Claude Haiku 5.5 was most consistent across tiers, GPT‑5.4 mini showed a large hard‑versus‑easy gap, and DeepSeek‑R1 performed below the always‑no baseline.

工作流

• 1. Define 44 planted defects from mutation‑testing operators (AOR, ROR, UOI, LCR, statement‑deletion, higher‑order mutants).
• 2. Create four variants per defect: original, disguised, fixed, clean.
• 3. Verify killability with automated tests that fail on defect variants and pass on fixed/clean.
• 4. Publish benchmark on Kaggle and run four AI models (Gemini 3.7 Flash, Claude Haiku 5.5, GPT‑5.4 mini, DeepSeek‑R1) plus baselines.
• 5. Score models automatically using detection rate, line‑level F1, keyword recall, and false‑positive rate.
• 6. Compute tier score as (detection + line‑F1 + keyword‑recall)/3 – false‑positive rate.
• 7. Report results and analyze performance across tiers.

证据

I spent the last month running fourteen independent audit rounds on a clean-room codebase.

来源

Red‑Pen Bench was created, containing 44 distinct planted defects across three difficulty tiers.

来源

Models must output three things: DEFECT: yes/no, LINES:, EXPLANATION:.

来源

Gemini 3.7 Flash led overall, Claude Haiku 5.5 was the most consistent across tiers, GPT‑5.4 mini showed the largest observed hard‑versus‑easy gap, and DeepSeek‑R1 finished below the always‑no baseline.

来源