I Built a Benchmark That Grades AI Code Reviewers on Planted Defects

Описание и применение

Создано новое бенчмарк‑средство Red‑Pen Bench, которое оценивает AI‑ревьюеров по обнаружению, локализации и объяснению искусственно введённых дефектов, учитывая ложные срабатывания.

Задача

Existing benchmarks do not isolate defect detection, localization, explanation, and false‑positive measurement for AI code reviewers.

Результат

Gemini 3.7 Flash achieved the highest overall score (0.799), Claude Haiku 5.5 was most consistent across tiers, GPT‑5.4 mini showed a large hard‑versus‑easy gap, and DeepSeek‑R1 performed below the always‑no baseline.

Рабочий процесс

• 1. Define 44 planted defects from mutation‑testing operators (AOR, ROR, UOI, LCR, statement‑deletion, higher‑order mutants).
• 2. Create four variants per defect: original, disguised, fixed, clean.
• 3. Verify killability with automated tests that fail on defect variants and pass on fixed/clean.
• 4. Publish benchmark on Kaggle and run four AI models (Gemini 3.7 Flash, Claude Haiku 5.5, GPT‑5.4 mini, DeepSeek‑R1) plus baselines.
• 5. Score models automatically using detection rate, line‑level F1, keyword recall, and false‑positive rate.
• 6. Compute tier score as (detection + line‑F1 + keyword‑recall)/3 – false‑positive rate.
• 7. Report results and analyze performance across tiers.

Доказательства

I spent the last month running fourteen independent audit rounds on a clean-room codebase.

Источник

Red‑Pen Bench was created, containing 44 distinct planted defects across three difficulty tiers.

Источник

Models must output three things: DEFECT: yes/no, LINES:, EXPLANATION:.

Источник

Gemini 3.7 Flash led overall, Claude Haiku 5.5 was the most consistent across tiers, GPT‑5.4 mini showed the largest observed hard‑versus‑easy gap, and DeepSeek‑R1 finished below the always‑no baseline.

Источник