I Turned the Reasoning Dial to 'High' on 4 Models. It Fixed One Thing and Billed Me for Everything.

Описание и применение

Параметр reasoning_effort в LLM часто меняется интуитивно, но его влияние на точность и стоимость не измерено. Создано соревнование Reasoning Dial, которое показало, что только высокий уровень улучшает точность в задачах дедукции, а в остальных случаях повышает стоимость без выгоды.

Задача

Developers set the reasoning_effort parameter daily without knowing its impact on accuracy, token usage, or cost.

Результат

High reasoning_effort improved accuracy from 15% to 97.5% on deduction tasks for gpt-5.4-mini, but in 7 of 12 model×task cells accuracy stayed the same while cost per correct answer rose 1.5–3.4×. The dial behaved inconsistently across models, with some rejecting the none level or ignoring it.

Рабочий процесс

• Define four task families (Deduce, Arith, Distract) with 20 items each.
• Create four Kaggle tasks differing only in reasoning_effort level (none, low, medium, high).
• Run each model on all tasks, recording accuracy, output tokens, and cost per correct answer.
• Analyze results with paired Wilcoxon tests, Holm correction, and bootstrap.
• Publish leaderboard and code on GitHub.
• Pre-register hypotheses and lock test items to ensure reproducibility.

Доказательства

Over 1,800 graded calls on 4 models, high produced exactly one statistically real accuracy gain: gpt-5.4-mini on logic puzzles, 15% → 97.5%.

Источник

In 7 of 12 model × task cells, accuracy did not change while cost per correct answer rose ×1.5 to ×3.4.

Источник

The dial also isn't one instrument: one model spends ×14 more tokens at high, one rejects none with an HTTP 400, and one thinks at none anyway.

Источник