I Built a Benchmark That Catches AI Models Cheating (And They All Failed)

Описание и применение

TwinBench – это бенчмарк, проверяющий, действительно ли модели следуют правилам, а не просто сопоставляют шаблоны. Четыре пары тестов выявили системные ошибки в трёх ведущих моделях.

Задача

AI models often appear to follow rules but actually rely on pattern matching, leading to incorrect answers when faced with counterfactual variations.

Результат

Four twin pairs survived filtering; all three frontier models failed every run on two pairs (CR‑N5, CR‑N8) and nearly all runs on two others (CR‑N3, CR‑N6). Models were confident yet incorrect, and naive baselines scored zero, confirming the benchmark’s rigor.

Рабочий процесс

• 1. Design twin pairs that differ only in a single controlling field.
• 2. Run each model (Opus 5.5 Beta, Grok 4.6, GPT‑6.1 Sol) five times per pair.
• 3. Retain a pair if any model fails at least two runs.
• 4. Evaluate final benchmark items with exact‑match scoring.
• 5. Analyze failure patterns and model confidence.
• 6. Publish results and code.

Доказательства

TwinBench, a benchmark that tests whether AI agents actually follow rules or just pattern‑match their way to plausible‑looking answers.

Источник

Four pairs survived. All of them are brutal.

Источник

CR‑N5 and CR‑N8: every model failed every single run.

Источник

The models are confident and wrong.

Источник