Tokens arrived on time — the renderer didn't: measuring a 90x display lag in local-LLM streaming (and how Ollama fixed it)

说明与用途

在本地 LLM IDE 中使用所有 CPU 核心导致显示延迟增加 90 倍，尽管生成速率保持不变。升级到 Ollama 0.40.2 后，新的引擎使用更少的线程并且不会饱和 CPU，从而解决了此问题。

问题

Local-LLM IDEs can feel slower when the model uses all CPU cores, even though token generation speed is unchanged.

结果

Using all CPU cores for local LLM inference caused a 90× increase in display lag and a 6× slower UI click response on Ollama 0.21.0. Upgrading to Ollama 0.40.2 reduced the lag dramatically because the new engine defaults to 3 threads and does not saturate the CPU. Thread caps and the think flag remain useful for guaranteeing consistent performance across versions.

工作流

• Measure token arrival via a proxy between IDE and Ollama.
• Record DOM render time with a MutationObserver.
• Measure event-loop delay with a 100 ms setTimeout drift.
• Count effective FPS with requestAnimationFrame.
• Time a real UI click during generation.
• Run tests on a Ryzen 5 4500U with fixed model settings.
• Compare results for all 6 cores vs 4 cores on Ollama 0.21.0 and 0.40.2.
• Analyze runner process to understand thread usage.
• Deploy thread cap and think flag settings in Teaspoon IDE.
• Publish findings and update IDE settings accordingly.

证据

On Ollama 0.21.0, running inference on all 6 cores vs. 4 produced the same token rate but 90x worse display lag (P95 181ms → 2ms) and 6x worse UI click response (929ms → 158ms).

来源

On Ollama 0.40.2 the gap nearly vanished, because the runner internals changed.

来源

The default changed: unspecified now means 3 threads — it never grabs all cores in the first place.

来源