Notes from a 2 AM On-Call: When llama-swap's Model Reload Hit a 30s Timeout Wall

Description and use

During a 2 AM on‑call, model swaps in llama‑swap exceeded nginx’s proxy_read_timeout, causing 502 errors. Raising timeouts, disabling buffering, and preloading models before peak traffic eliminated errors and reduced latency to 1.2 s.

Problem

Production gateway returned 502 Bad Gateway during peak APAC hours when llama-swap performed a model swap that took 29s, exceeding nginx’s proxy_read_timeout.

Outcome

The 3‑line nginx configuration and a preload script eliminated 502 errors and reduced swap latency from 29s to 1.2s, enabling reliable model switching.

Workflow

• 1. Identify 502 errors during peak traffic.
• 2. Observe that model swap takes 25‑35s on A10G.
• 3. Increase nginx proxy_read_timeout to 90s, proxy_connect_timeout to 10s, proxy_send_timeout to 90s, and disable proxy_buffering.
• 4. Add a warmup endpoint that triggers a throwaway completion to preload the desired model before peak hours.
• 5. Run preload script during deployment or via cron.
• 6. Verify post‑fix latency: 0 errors, 1.2s P99, 180ms TTFT for warm requests.

Evidence

upstream prematurely closed connection while reading response header from upstream

Source

proxy_read_timeout 60s is generous. It's not. The timeout clock starts at request dispatch, not at first response byte.

Source

The 3-line nginx fix: location /v1/ { proxy_pass http://llama-swap:8000; proxy_read_timeout 90s; proxy_connect_timeout 10s; proxy_send_timeout 90s; proxy_buffering off; }

Source

Cold swap under load: 502 errors, 29s P99 latency; Post-fix with preload: 0 errors, 1.2s P99 latency (model already loaded)

Source