I Built a Benchmark That Catches AI Models Cheating (And They All Failed)

Description and use

TwinBench is a benchmark that tests whether models truly follow rules or merely pattern‑match. Four twin pairs revealed systematic errors in three leading models.

Problem

AI models often appear to follow rules but actually rely on pattern matching, leading to incorrect answers when faced with counterfactual variations.

Outcome

Four twin pairs survived filtering; all three frontier models failed every run on two pairs (CR‑N5, CR‑N8) and nearly all runs on two others (CR‑N3, CR‑N6). Models were confident yet incorrect, and naive baselines scored zero, confirming the benchmark’s rigor.

Workflow

• 1. Design twin pairs that differ only in a single controlling field.
• 2. Run each model (Opus 5.5 Beta, Grok 4.6, GPT‑6.1 Sol) five times per pair.
• 3. Retain a pair if any model fails at least two runs.
• 4. Evaluate final benchmark items with exact‑match scoring.
• 5. Analyze failure patterns and model confidence.
• 6. Publish results and code.

Evidence

TwinBench, a benchmark that tests whether AI agents actually follow rules or just pattern‑match their way to plausible‑looking answers.

Source

Four pairs survived. All of them are brutal.

Source

CR‑N5 and CR‑N8: every model failed every single run.

Source

The models are confident and wrong.

Source