Tokens arrived on time — the renderer didn't: measuring a 90x display lag in local-LLM streaming (and how Ollama fixed it)

Описание и применение

При использовании всех ядер процессора в локальном LLM IDE наблюдалось 90‑кратное увеличение задержки отображения, хотя скорость генерации токенов оставалась неизменной. Обновление Ollama до версии 0.40.2 устранило проблему, так как новый движок использует меньше потоков и не насыщает CPU.

Задача

Local-LLM IDEs can feel slower when the model uses all CPU cores, even though token generation speed is unchanged.

Результат

Using all CPU cores for local LLM inference caused a 90× increase in display lag and a 6× slower UI click response on Ollama 0.21.0. Upgrading to Ollama 0.40.2 reduced the lag dramatically because the new engine defaults to 3 threads and does not saturate the CPU. Thread caps and the think flag remain useful for guaranteeing consistent performance across versions.

Рабочий процесс

• Measure token arrival via a proxy between IDE and Ollama.
• Record DOM render time with a MutationObserver.
• Measure event-loop delay with a 100 ms setTimeout drift.
• Count effective FPS with requestAnimationFrame.
• Time a real UI click during generation.
• Run tests on a Ryzen 5 4500U with fixed model settings.
• Compare results for all 6 cores vs 4 cores on Ollama 0.21.0 and 0.40.2.
• Analyze runner process to understand thread usage.
• Deploy thread cap and think flag settings in Teaspoon IDE.
• Publish findings and update IDE settings accordingly.

Доказательства

On Ollama 0.21.0, running inference on all 6 cores vs. 4 produced the same token rate but 90x worse display lag (P95 181ms → 2ms) and 6x worse UI click response (929ms → 158ms).

Источник

On Ollama 0.40.2 the gap nearly vanished, because the runner internals changed.

Источник

The default changed: unspecified now means 3 threads — it never grabs all cores in the first place.

Источник