llama-server's sleep mode loses or crashes on a request that arrives just before it sleeps

说明与用途

在 llama-server 进入睡眠模式前几毫秒到达的请求可能导致挂起或崩溃。先发送一个 1‑token 的短请求可重置空闲计时器，避免竞争，确保服务器正常响应。

问题

Requests arriving just before a llama-server with sleep mode enabled can hang or crash because the server may enter sleep while tokenizing or queuing the request.

结果

Using a 1-token warm-up request before the main request eliminates the race condition, ensuring all requests are answered and the server does not crash.

工作流

• 1. Start llama-server with --sleep-idle-seconds N. 2. Send a 1-token completion request (e.g., prompt "hi", n_predict 1). 3. Immediately send the real completion request. 4. The short request resets the idle timer, preventing the server from sleeping during tokenization of the real request.
• 5. Verify that the real request receives a response and the server does not crash or hang.

证据

Send a 1-token completion before the real request. A request whose prompt is a couple of tokens has almost no window: 0 hits in 93. When it is queued it resets the idle timer, so the real request that follows gets a full idle period of headroom instead of whatever was left.

来源

123 sends, 0.93 .. 1.01 s: 123 answered, 0 hung, 0 crashed

来源