Notes from a 2 AM On-Call: When llama-swap's Model Reload Hit a 30s Timeout Wall

说明与用途

在凌晨2点的运维中，llama-swap 的模型切换耗时29秒，超过了nginx的proxy_read_timeout，导致502错误。通过提升超时、关闭缓冲并在高峰前预热模型，成功消除了错误并将延迟降至1.2秒。

问题

Production gateway returned 502 Bad Gateway during peak APAC hours when llama-swap performed a model swap that took 29s, exceeding nginx’s proxy_read_timeout.

结果

The 3‑line nginx configuration and a preload script eliminated 502 errors and reduced swap latency from 29s to 1.2s, enabling reliable model switching.

工作流

• 1. Identify 502 errors during peak traffic.
• 2. Observe that model swap takes 25‑35s on A10G.
• 3. Increase nginx proxy_read_timeout to 90s, proxy_connect_timeout to 10s, proxy_send_timeout to 90s, and disable proxy_buffering.
• 4. Add a warmup endpoint that triggers a throwaway completion to preload the desired model before peak hours.
• 5. Run preload script during deployment or via cron.
• 6. Verify post‑fix latency: 0 errors, 1.2s P99, 180ms TTFT for warm requests.

证据

upstream prematurely closed connection while reading response header from upstream

来源

proxy_read_timeout 60s is generous. It's not. The timeout clock starts at request dispatch, not at first response byte.

来源

The 3-line nginx fix: location /v1/ { proxy_pass http://llama-swap:8000; proxy_read_timeout 90s; proxy_connect_timeout 10s; proxy_send_timeout 90s; proxy_buffering off; }

来源

Cold swap under load: 502 errors, 29s P99 latency; Post-fix with preload: 0 errors, 1.2s P99 latency (model already loaded)

来源