I Turned the Reasoning Dial to 'High' on 4 Models. It Fixed One Thing and Billed Me for Everything.

说明与用途

LLM中的reasoning_effort参数通常根据直觉设置，但其对准确性和成本的影响未知。Reasoning Dial基准显示，只有高设置在推理任务中提高准确性，而在其他情况下会增加成本而无收益。

问题

Developers set the reasoning_effort parameter daily without knowing its impact on accuracy, token usage, or cost.

结果

High reasoning_effort improved accuracy from 15% to 97.5% on deduction tasks for gpt-5.4-mini, but in 7 of 12 model×task cells accuracy stayed the same while cost per correct answer rose 1.5–3.4×. The dial behaved inconsistently across models, with some rejecting the none level or ignoring it.

工作流

• Define four task families (Deduce, Arith, Distract) with 20 items each.
• Create four Kaggle tasks differing only in reasoning_effort level (none, low, medium, high).
• Run each model on all tasks, recording accuracy, output tokens, and cost per correct answer.
• Analyze results with paired Wilcoxon tests, Holm correction, and bootstrap.
• Publish leaderboard and code on GitHub.
• Pre-register hypotheses and lock test items to ensure reproducibility.

证据

Over 1,800 graded calls on 4 models, high produced exactly one statistically real accuracy gain: gpt-5.4-mini on logic puzzles, 15% → 97.5%.

来源

In 7 of 12 model × task cells, accuracy did not change while cost per correct answer rose ×1.5 to ×3.4.

来源

The dial also isn't one instrument: one model spends ×14 more tokens at high, one rejects none with an HTTP 400, and one thinks at none anyway.

来源