I built an LLM benchmark you cannot train on

Description and use

A dynamic LLM benchmark that generates tasks from a random seed at runtime, using commit‑reveal and multiple checker types to detect memorization and ensure genuine skill assessment.

Problem

LLM benchmarks suffer from contamination: models memorize questions that are publicly available, leading to inflated scores that do not reflect real-world performance.

Outcome

The benchmark achieved 99.7% accuracy with a deterministic solver and 93.7% with gpt‑oss‑120b, while a memorizer model was flagged due to a +100pp gap, demonstrating effective contamination detection.

Workflow

• 1. Generate 225 deterministic task worlds from a random seed at run time.
• 2. Publish only the SHA‑256 hash of the seed before the run (commit).
• 3. Run candidate models against the generated tasks.
• 4. Collect aggregate reports; no individual question is revealed.
• 5. Reveal the seed after the run (reveal) to prove no swapping.
• 6. Use three checker kinds: formal, differential, and open (human A/B).
• 7. Measure contamination by mixing old tasks from a reference run into the current set and checking performance gaps.
• 8. Flag models with a gap >15pp as likely memorizers.
• 9. Publish results and provide reproducible scripts for independent runs.

Evidence

If the evaluation set does not exist until the run starts, there is nothing to memorize.

Source

gap > 15 pp is a reliable sign the model is optimized for the benchmark rather than solving it.

Source

Real models are called directly through provider APIs.

Source