I built an LLM benchmark you cannot train on

Описание и применение

Создан динамический бенчмарк LLM, генерирующий задачи из случайного семени во время выполнения, чтобы предотвратить запоминание вопросов. Используются commit‑reveal и три вида проверок, включая дифференциальную, чтобы выявлять модели, которые просто запоминают ответы.

Задача

LLM benchmarks suffer from contamination: models memorize questions that are publicly available, leading to inflated scores that do not reflect real-world performance.

Результат

The benchmark achieved 99.7% accuracy with a deterministic solver and 93.7% with gpt‑oss‑120b, while a memorizer model was flagged due to a +100pp gap, demonstrating effective contamination detection.

Рабочий процесс

• 1. Generate 225 deterministic task worlds from a random seed at run time.
• 2. Publish only the SHA‑256 hash of the seed before the run (commit).
• 3. Run candidate models against the generated tasks.
• 4. Collect aggregate reports; no individual question is revealed.
• 5. Reveal the seed after the run (reveal) to prove no swapping.
• 6. Use three checker kinds: formal, differential, and open (human A/B).
• 7. Measure contamination by mixing old tasks from a reference run into the current set and checking performance gaps.
• 8. Flag models with a gap >15pp as likely memorizers.
• 9. Publish results and provide reproducible scripts for independent runs.

Доказательства

If the evaluation set does not exist until the run starts, there is nothing to memorize.

Источник

gap > 15 pp is a reliable sign the model is optimized for the benchmark rather than solving it.

Источник

Real models are called directly through provider APIs.

Источник