SWE-bench Is Not Contaminated — But Its Test Files Were Public Years Before the Fixes

Description and use

The SWE‑bench audit revealed that while problem statements were not leaked, 97.8 % of test files were public before the base commit, indicating a significant contamination channel.

Problem

Assess whether SWE‑bench benchmark instances are contaminated by public GitHub data, particularly test files that could have been seen by language models.

Outcome

The study found zero verbatim leakage of problem statements but 97.8% of test files were publicly available at the base commit, revealing a major contamination channel. Recommendations include pinning benchmarks to earliest exposure, rotating test checks, and reporting exposure ratios.

Workflow

• Collect all 500 SWE‑bench‑Verified instances.
• For each instance, probe four GitHub history channels: C1a (issue public), C1b (verbatim leakage), C2 (test files public early), C3 (fix findable by issue search).
• Record results for each channel.
• Generate a table of contamination rates.
• Publish deterministic pipeline and evidence snapshots (500 JSON records).
• Publish reproducibility script (reproduce.sh).
• Publish findings and recommendations for benchmark design.

Evidence

0.0% of instances contain the literal problem statement

97.8% (489/500) of tests existed in the repo at base_commit or earlier

The benchmark is not verbatim contaminated. Zero instances leak the problem statement word-for-word

The check you actually need is not in the training data — it was in the repo all along.