Notes from a 2 AM On-Call: When llama-swap's Model Reload Hit a 30s Timeout Wall

Описание и применение

Во время ночного обслуживания обнаружено, что переключение модели в llama‑swap занимает 29 с, превышая таймаут nginx, что приводит к ошибкам 502. Увеличив таймауты и отключив буферизацию, а также добавив скрипт предварительной загрузки модели, удалось устранить ошибки и снизить задержку до 1,2 с.

Задача

Production gateway returned 502 Bad Gateway during peak APAC hours when llama-swap performed a model swap that took 29s, exceeding nginx’s proxy_read_timeout.

Результат

The 3‑line nginx configuration and a preload script eliminated 502 errors and reduced swap latency from 29s to 1.2s, enabling reliable model switching.

Рабочий процесс

• 1. Identify 502 errors during peak traffic.
• 2. Observe that model swap takes 25‑35s on A10G.
• 3. Increase nginx proxy_read_timeout to 90s, proxy_connect_timeout to 10s, proxy_send_timeout to 90s, and disable proxy_buffering.
• 4. Add a warmup endpoint that triggers a throwaway completion to preload the desired model before peak hours.
• 5. Run preload script during deployment or via cron.
• 6. Verify post‑fix latency: 0 errors, 1.2s P99, 180ms TTFT for warm requests.

Доказательства

upstream prematurely closed connection while reading response header from upstream

Источник

proxy_read_timeout 60s is generous. It's not. The timeout clock starts at request dispatch, not at first response byte.

Источник

The 3-line nginx fix: location /v1/ { proxy_pass http://llama-swap:8000; proxy_read_timeout 90s; proxy_connect_timeout 10s; proxy_send_timeout 90s; proxy_buffering off; }

Источник

Cold swap under load: 502 errors, 29s P99 latency; Post-fix with preload: 0 errors, 1.2s P99 latency (model already loaded)

Источник