Benchmaxing: Winning the Exam Is Not Doing Better Work

说明与用途

更高的基准分数并不保证更好的实际工作。对比 Opus 5 和 Fable 5 表明，基准结果并不总能反映实际效果。

问题

Higher benchmark scores do not guarantee better real-world performance, leading to potential misinterpretation of model capabilities.

结果

Opus 5 and Fable 5 perform similarly on benchmark-like synthetic tasks; neither model consistently outperforms the other in real-world relevant criteria. Benchmark scores alone cannot confirm practical superiority.

工作流

• 1. Reviewed literature on benchmaxing and Goodhart’s law.
• 2. Identified key phenomena: test familiarity, selective reporting, metric-work gap.
• 3. Designed a pilot test with five synthetic cases per model using Claude Code.
• 4. Ran Opus 5 and Fable 5 on identical prompts and constraints.
• 5. Collected qualitative assessments of core criteria per case.
• 6. Analyzed failures, especially in external effect duplication scenarios.
• 7. Compared results and drew conclusions about benchmark vs. real work.
• 8. Documented findings and made prompts, answers, and checks publicly available.

证据

I compared claude-opus-5 and claude-fable-5 through Claude Code on a Max subscription, at high effort and with the same 8192-output-token limit.

来源

Both models recognized much of the problem. They proposed stable keys, persistent states and stopping retries after a deadline.

来源