llama-server's sleep mode loses or crashes on a request that arrives just before it sleeps

Описание и применение

Запросы, приходящие за несколько миллисекунд до перехода llama-server в режим сна, могут зависнуть или завершиться с ошибкой. Отправка короткого 1‑токенного запроса перед основным запросом устраняет гонку, позволяя серверу успеть разбудиться и обработать запрос без падения.

Задача

Requests arriving just before a llama-server with sleep mode enabled can hang or crash because the server may enter sleep while tokenizing or queuing the request.

Результат

Using a 1-token warm-up request before the main request eliminates the race condition, ensuring all requests are answered and the server does not crash.

Рабочий процесс

• 1. Start llama-server with --sleep-idle-seconds N. 2. Send a 1-token completion request (e.g., prompt "hi", n_predict 1). 3. Immediately send the real completion request. 4. The short request resets the idle timer, preventing the server from sleeping during tokenization of the real request.
• 5. Verify that the real request receives a response and the server does not crash or hang.

Доказательства

Send a 1-token completion before the real request. A request whose prompt is a couple of tokens has almost no window: 0 hits in 93. When it is queued it resets the idle timer, so the real request that follows gets a full idle period of headroom instead of whatever was left.

Источник

123 sends, 0.93 .. 1.01 s: 123 answered, 0 hung, 0 crashed

Источник