llama-server's sleep mode loses or crashes on a request that arrives just before it sleeps

Description and use

Requests arriving just before llama-server sleeps can hang or crash. Sending a short 1‑token request before the main request resets the idle timer and prevents the race, ensuring the server responds correctly.

Problem

Requests arriving just before a llama-server with sleep mode enabled can hang or crash because the server may enter sleep while tokenizing or queuing the request.

Outcome

Using a 1-token warm-up request before the main request eliminates the race condition, ensuring all requests are answered and the server does not crash.

Workflow

• 1. Start llama-server with --sleep-idle-seconds N. 2. Send a 1-token completion request (e.g., prompt "hi", n_predict 1). 3. Immediately send the real completion request. 4. The short request resets the idle timer, preventing the server from sleeping during tokenization of the real request.
• 5. Verify that the real request receives a response and the server does not crash or hang.

Evidence

Send a 1-token completion before the real request. A request whose prompt is a couple of tokens has almost no window: 0 hits in 93. When it is queued it resets the idle timer, so the real request that follows gets a full idle period of headroom instead of whatever was left.

Source

123 sends, 0.93 .. 1.01 s: 123 answered, 0 hung, 0 crashed

Source