Why Your Local LLM Gives Different Answers at Temperature 0

Description and use

Setting temperature 0 and a fixed seed does not guarantee identical completions from local LLMs due to floating‑point order effects and engine‑level nondeterminism.

Problem

Setting temperature to 0 and fixing the seed does not guarantee identical completions from a local LLM.

Outcome

The article demonstrates that even with temperature 0 and a fixed seed, local LLMs can produce different completions due to floating‑point order effects and engine‑level nondeterminism. It provides a reproducible test setup and explains the underlying causes.

Workflow

• 1. Start a local LLM server (e.g., llama.cpp) with desired model and parameters.
• 2. Run a benchmark (e.g., lm-evaluation-harness on gsm8k) once with single requests and once with concurrent requests.
• 3. Compare outputs token‑by‑token to detect differences.
• 4. Use the provided Python script to send a prompt alone and again with concurrent prompts, then compare the two answers.
• 5. Analyze causes such as floating‑point addition order, batching, prompt cache, speculative decoding, and kernel nondeterminism.

Evidence

One measured case: I ran lm-evaluation-harness on gsm8k (100 problems) against llama.cpp's server (build b11643, Qwen2.5-7B-Instruct Q8_0, -np 8), once with one request at a time and once with eight concurrent requests. Same model file, same settings, greedy decoding. The generated text differed on 41 of the 100 problems.

Source

Temperature and the seed control one step: picking the next token from the probabilities the model computed. At temperature 0 the engine takes the most probable token, and the seed stops mattering.

Source

A computer rounds after every floating-point addition, so (a + b) + c is not always equal to a + (b + c). Each logit is the end of millions of multiply-adds, and the order in which a GPU kernel adds them up decides the last bits.

Source

Batching and concurrent requests. The server computes your request together with whatever else it is doing. A matrix multiplication for one row and for eight rows can use different kernels with different reduction orders, so your row comes out slightly different depending on who else was in the batch.

Source