A "decontaminated" dataset still had 52 of HumanEval's 164 problems in it

Описание и применение

Фильтр «деконтаминации» не удалил 52 задачи HumanEval из набора инструкций, что привело к завышенным оценкам модели. После удаления загрязненных примеров точность снизилась на 4,3 пункта.

Задача

A code model trained on instruction data contained benchmark problems that were not removed by a decontamination filter, leading to inflated evaluation results.

Результат

The decontamination filter failed to remove many benchmark problems because the dataset was built by rewriting seed problems, making exact‑match filtering ineffective. Removing the contaminated data lowered the model’s HumanEval performance by 4.3 points, demonstrating that practice on benchmark‑style problems can inflate results.

Рабочий процесс

• 1. Train a 360M parameter code model on instruction data.
• 2. Evaluate on HumanEval, obtaining 26.8% pass@1.
• 3. Perform a two‑step scan: (a) match first 80 chars of HumanEval docstrings, (b) match 13‑token n‑grams of canonical solutions.
• 4. Identify 52 HumanEval problems present in the supposedly decontaminated dataset.
• 5. Remove contaminated examples and retrain the model.
• 6. Re‑evaluate on HumanEval, HumanEval+, and MBPP.
• 7. Compare results with baseline and seed‑controlled runs.
• 8. Publish findings and provide code and data for replication.

Доказательства

Two checks against each instruction set: Problem text. The first 80 characters of each HumanEval docstring, whitespace normalized. Canonical solutions. Half or more of a solution's 13-token n-grams, with comments and docstrings stripped.

Источник

The filtered redistribution differs from the original by 89 rows and still contains 52 of the 60 contaminated HumanEval problems. Switching to it would not have saved us.

Источник

Decontaminated versus published on HumanEval: -4.3 points, 8 problems lost and 1 gained, McNemar p = 0.039.

Источник