Benchmaxing: Winning the Exam Is Not Doing Better Work

Description and use

Higher benchmark scores do not guarantee better real-world performance. Comparing Opus 5 and Fable 5 shows that benchmark results do not always reflect practical effectiveness.

Problem

Higher benchmark scores do not guarantee better real-world performance, leading to potential misinterpretation of model capabilities.

Outcome

Opus 5 and Fable 5 perform similarly on benchmark-like synthetic tasks; neither model consistently outperforms the other in real-world relevant criteria. Benchmark scores alone cannot confirm practical superiority.

Workflow

• 1. Reviewed literature on benchmaxing and Goodhart’s law.
• 2. Identified key phenomena: test familiarity, selective reporting, metric-work gap.
• 3. Designed a pilot test with five synthetic cases per model using Claude Code.
• 4. Ran Opus 5 and Fable 5 on identical prompts and constraints.
• 5. Collected qualitative assessments of core criteria per case.
• 6. Analyzed failures, especially in external effect duplication scenarios.
• 7. Compared results and drew conclusions about benchmark vs. real work.
• 8. Documented findings and made prompts, answers, and checks publicly available.

Evidence

I compared claude-opus-5 and claude-fable-5 through Claude Code on a Max subscription, at high effort and with the same 8192-output-token limit.

Source

Both models recognized much of the problem. They proposed stable keys, persistent states and stopping retries after a deadline.

Source