Why Your Local LLM Gives Different Answers at Temperature 0

说明与用途

即使将温度设为0并固定种子，局部 LLM 仍可能产生不同的输出，原因是浮点运算顺序和引擎层面的非确定性。

问题

Setting temperature to 0 and fixing the seed does not guarantee identical completions from a local LLM.

结果

The article demonstrates that even with temperature 0 and a fixed seed, local LLMs can produce different completions due to floating‑point order effects and engine‑level nondeterminism. It provides a reproducible test setup and explains the underlying causes.

工作流

• 1. Start a local LLM server (e.g., llama.cpp) with desired model and parameters.
• 2. Run a benchmark (e.g., lm-evaluation-harness on gsm8k) once with single requests and once with concurrent requests.
• 3. Compare outputs token‑by‑token to detect differences.
• 4. Use the provided Python script to send a prompt alone and again with concurrent prompts, then compare the two answers.
• 5. Analyze causes such as floating‑point addition order, batching, prompt cache, speculative decoding, and kernel nondeterminism.

证据

One measured case: I ran lm-evaluation-harness on gsm8k (100 problems) against llama.cpp's server (build b11643, Qwen2.5-7B-Instruct Q8_0, -np 8), once with one request at a time and once with eight concurrent requests. Same model file, same settings, greedy decoding. The generated text differed on 41 of the 100 problems.

来源

Temperature and the seed control one step: picking the next token from the probabilities the model computed. At temperature 0 the engine takes the most probable token, and the seed stops mattering.

来源

A computer rounds after every floating-point addition, so (a + b) + c is not always equal to a + (b + c). Each logit is the end of millions of multiply-adds, and the order in which a GPU kernel adds them up decides the last bits.

来源

Batching and concurrent requests. The server computes your request together with whatever else it is doing. A matrix multiplication for one row and for eight rows can use different kernels with different reduction orders, so your row comes out slightly different depending on who else was in the batch.

来源