Why Your Local LLM Gives Different Answers at Temperature 0

Описание и применение

Установив температуру 0 и фиксированный seed, локальные модели LLM всё равно дают разные ответы из‑за неустойчивости вычислений и порядка выполнения операций.

Задача

Setting temperature to 0 and fixing the seed does not guarantee identical completions from a local LLM.

Результат

The article demonstrates that even with temperature 0 and a fixed seed, local LLMs can produce different completions due to floating‑point order effects and engine‑level nondeterminism. It provides a reproducible test setup and explains the underlying causes.

Рабочий процесс

• 1. Start a local LLM server (e.g., llama.cpp) with desired model and parameters.
• 2. Run a benchmark (e.g., lm-evaluation-harness on gsm8k) once with single requests and once with concurrent requests.
• 3. Compare outputs token‑by‑token to detect differences.
• 4. Use the provided Python script to send a prompt alone and again with concurrent prompts, then compare the two answers.
• 5. Analyze causes such as floating‑point addition order, batching, prompt cache, speculative decoding, and kernel nondeterminism.

Доказательства

One measured case: I ran lm-evaluation-harness on gsm8k (100 problems) against llama.cpp's server (build b11643, Qwen2.5-7B-Instruct Q8_0, -np 8), once with one request at a time and once with eight concurrent requests. Same model file, same settings, greedy decoding. The generated text differed on 41 of the 100 problems.

Источник

Temperature and the seed control one step: picking the next token from the probabilities the model computed. At temperature 0 the engine takes the most probable token, and the seed stops mattering.

Источник

A computer rounds after every floating-point addition, so (a + b) + c is not always equal to a + (b + c). Each logit is the end of millions of multiply-adds, and the order in which a GPU kernel adds them up decides the last bits.

Источник

Batching and concurrent requests. The server computes your request together with whatever else it is doing. A matrix multiplication for one row and for eight rows can use different kernels with different reduction orders, so your row comes out slightly different depending on who else was in the batch.

Источник