I built an LLM benchmark you cannot train on

说明与用途

构建了一个动态 LLM 基准测试，使用随机种子在运行时生成任务，并通过提交-揭示和多种检查器来检测模型是否仅仅记住答案，确保真正的技能评估。

问题

LLM benchmarks suffer from contamination: models memorize questions that are publicly available, leading to inflated scores that do not reflect real-world performance.

结果

The benchmark achieved 99.7% accuracy with a deterministic solver and 93.7% with gpt‑oss‑120b, while a memorizer model was flagged due to a +100pp gap, demonstrating effective contamination detection.

工作流

• 1. Generate 225 deterministic task worlds from a random seed at run time.
• 2. Publish only the SHA‑256 hash of the seed before the run (commit).
• 3. Run candidate models against the generated tasks.
• 4. Collect aggregate reports; no individual question is revealed.
• 5. Reveal the seed after the run (reveal) to prove no swapping.
• 6. Use three checker kinds: formal, differential, and open (human A/B).
• 7. Measure contamination by mixing old tasks from a reference run into the current set and checking performance gaps.
• 8. Flag models with a gap >15pp as likely memorizers.
• 9. Publish results and provide reproducible scripts for independent runs.

证据

If the evaluation set does not exist until the run starts, there is nothing to memorize.

来源

gap > 15 pp is a reliable sign the model is optimized for the benchmark rather than solving it.

来源

Real models are called directly through provider APIs.

来源