Telling a cheap model to "think a little" made it slower and sometimes silent

Описание и применение

Гейтвей, который перенаправляет запросы к нескольким моделям, ошибочно включал режим размышления для модели Gemini 2.5 Flash‑Lite, которая по умолчанию не размышляет. Это приводило к медленной работе и иногда к пустому ответу при ограниченном бюджете токенов. Исправление включало обновление регулярного выражения, исключающего Flash‑Lite, и добавление тестов, чтобы убедиться, что поле размышления не отправляется этим моделям.

Задача

A gateway that forwards requests to multiple model families mistakenly enabled reasoning on a model that does not think by default, causing slow or silent responses when a tight token budget is used.

Результат

By refining the regex to exclude Flash‑Lite variants and adding tests, the gateway no longer mistakenly enables reasoning on models that do not think by default, eliminating slow or silent responses under tight token budgets.

Рабочий процесс

• 1. Build a desktop app that can call a model via Ollama, user API key, or Cloudflare Worker gateway.
• 2. Gateway receives a virtual label and maps it to a real model name.
• 3. Gateway checks if the model name matches a regex for reasoning models.
• 4. If matched, it adds a reasoning field with effort "low" to the request body.
• 5. The model processes the request; if it thinks by default, it may spend all tokens on reasoning and return an empty reply.
• 6. The app observes slower or silent responses when max_tokens is tight.
• 7. Update the regex to exclude "-flash-lite" variants using a negative lookahead.
• 8. Add unit tests to confirm reasoning is not sent to Flash‑Lite and non‑reasoning models are untouched.
• 9. Adjust gateway to treat empty replies as failures rather than successful 200 responses.

Доказательства

The gateway checks the model name and, for models that reason, adds a cap.

Источник

The Gemini 2.5 models I had in mind think by default, so gemini-2\.5 looked right.

Источник

With a roomier budget the answer can still fit after the thinking, so the failure is quieter: the reply just arrives later than it should, from the model you picked because it was fast.

Источник

The fix was one negative lookahead: return /gpt-oss|deepseek-r1|qwq|magistral|gemini-2\.5(?!-flash-lite)/i.test(model);

Источник