I Built a Benchmark That Catches AI Models Cheating (And They All Failed)

说明与用途

TwinBench 是一个测试模型是否真正遵循规则而非仅仅进行模式匹配的基准。四个双胞胎对揭示了三种领先模型的系统性错误。

问题

AI models often appear to follow rules but actually rely on pattern matching, leading to incorrect answers when faced with counterfactual variations.

结果

Four twin pairs survived filtering; all three frontier models failed every run on two pairs (CR‑N5, CR‑N8) and nearly all runs on two others (CR‑N3, CR‑N6). Models were confident yet incorrect, and naive baselines scored zero, confirming the benchmark’s rigor.

工作流

• 1. Design twin pairs that differ only in a single controlling field.
• 2. Run each model (Opus 5.5 Beta, Grok 4.6, GPT‑6.1 Sol) five times per pair.
• 3. Retain a pair if any model fails at least two runs.
• 4. Evaluate final benchmark items with exact‑match scoring.
• 5. Analyze failure patterns and model confidence.
• 6. Publish results and code.

证据

TwinBench, a benchmark that tests whether AI agents actually follow rules or just pattern‑match their way to plausible‑looking answers.

来源

Four pairs survived. All of them are brutal.

来源

CR‑N5 and CR‑N8: every model failed every single run.

来源

The models are confident and wrong.

来源