I Turned the Reasoning Dial to 'High' on 4 Models. It Fixed One Thing and Billed Me for Everything.

Description and use

The reasoning_effort parameter in LLMs is often set intuitively, yet its impact on accuracy and cost is unknown. The Reasoning Dial benchmark showed that only the high setting improves accuracy in deduction tasks, while in other cases it raises cost without benefit.

Problem

Developers set the reasoning_effort parameter daily without knowing its impact on accuracy, token usage, or cost.

Outcome

High reasoning_effort improved accuracy from 15% to 97.5% on deduction tasks for gpt-5.4-mini, but in 7 of 12 model×task cells accuracy stayed the same while cost per correct answer rose 1.5–3.4×. The dial behaved inconsistently across models, with some rejecting the none level or ignoring it.

Workflow

• Define four task families (Deduce, Arith, Distract) with 20 items each.
• Create four Kaggle tasks differing only in reasoning_effort level (none, low, medium, high).
• Run each model on all tasks, recording accuracy, output tokens, and cost per correct answer.
• Analyze results with paired Wilcoxon tests, Holm correction, and bootstrap.
• Publish leaderboard and code on GitHub.
• Pre-register hypotheses and lock test items to ensure reproducibility.

Evidence

Over 1,800 graded calls on 4 models, high produced exactly one statistically real accuracy gain: gpt-5.4-mini on logic puzzles, 15% → 97.5%.

Source

In 7 of 12 model × task cells, accuracy did not change while cost per correct answer rose ×1.5 to ×3.4.

Source

The dial also isn't one instrument: one model spends ×14 more tokens at high, one rejects none with an HTTP 400, and one thinks at none anyway.

Source