A "decontaminated" dataset still had 52 of HumanEval's 164 problems in it

说明与用途

去污过滤器未能删除 52 个 HumanEval 问题，导致模型性能被高估。删除后 pass@1 降低 4.3 分。

问题

A code model trained on instruction data contained benchmark problems that were not removed by a decontamination filter, leading to inflated evaluation results.

结果

The decontamination filter failed to remove many benchmark problems because the dataset was built by rewriting seed problems, making exact‑match filtering ineffective. Removing the contaminated data lowered the model’s HumanEval performance by 4.3 points, demonstrating that practice on benchmark‑style problems can inflate results.

工作流

• 1. Train a 360M parameter code model on instruction data.
• 2. Evaluate on HumanEval, obtaining 26.8% pass@1.
• 3. Perform a two‑step scan: (a) match first 80 chars of HumanEval docstrings, (b) match 13‑token n‑grams of canonical solutions.
• 4. Identify 52 HumanEval problems present in the supposedly decontaminated dataset.
• 5. Remove contaminated examples and retrain the model.
• 6. Re‑evaluate on HumanEval, HumanEval+, and MBPP.
• 7. Compare results with baseline and seed‑controlled runs.
• 8. Publish findings and provide code and data for replication.

证据

Two checks against each instruction set: Problem text. The first 80 characters of each HumanEval docstring, whitespace normalized. Canonical solutions. Half or more of a solution's 13-token n-grams, with comments and docstrings stripped.

来源

The filtered redistribution differs from the original by 89 rows and still contains 52 of the 60 contaminated HumanEval problems. Switching to it would not have saved us.

来源

Decontaminated versus published on HumanEval: -4.3 points, 8 problems lost and 1 gained, McNemar p = 0.039.

来源