A "decontaminated" dataset still had 52 of HumanEval's 164 problems in it

Description and use

The decontamination filter failed to remove 52 HumanEval problems from the instruction data, inflating the model’s performance. Removing them lowered pass@1 by 4.3 points.

Problem

A code model trained on instruction data contained benchmark problems that were not removed by a decontamination filter, leading to inflated evaluation results.

Outcome

The decontamination filter failed to remove many benchmark problems because the dataset was built by rewriting seed problems, making exact‑match filtering ineffective. Removing the contaminated data lowered the model’s HumanEval performance by 4.3 points, demonstrating that practice on benchmark‑style problems can inflate results.

Workflow

• 1. Train a 360M parameter code model on instruction data.
• 2. Evaluate on HumanEval, obtaining 26.8% pass@1.
• 3. Perform a two‑step scan: (a) match first 80 chars of HumanEval docstrings, (b) match 13‑token n‑grams of canonical solutions.
• 4. Identify 52 HumanEval problems present in the supposedly decontaminated dataset.
• 5. Remove contaminated examples and retrain the model.
• 6. Re‑evaluate on HumanEval, HumanEval+, and MBPP.
• 7. Compare results with baseline and seed‑controlled runs.
• 8. Publish findings and provide code and data for replication.

Evidence

Two checks against each instruction set: Problem text. The first 80 characters of each HumanEval docstring, whitespace normalized. Canonical solutions. Half or more of a solution's 13-token n-grams, with comments and docstrings stripped.

Source

The filtered redistribution differs from the original by 89 rows and still contains 52 of the 60 contaminated HumanEval problems. Switching to it would not have saved us.

Source

Decontaminated versus published on HumanEval: -4.3 points, 8 problems lost and 1 gained, McNemar p = 0.039.

Source