GPUStack With vLLM in Production: Pinning KV Cache Before the OOM Finds You

Description and use

GPUStack with vLLM stabilized a 70B Llama model on mixed GPUs by pinning max‑model‑len, limiting GPU memory usage, and capping sequence count, preventing OOM crashes.

Problem

Deploying a 70B Llama model on heterogeneous GPUs caused out‑of‑memory (OOM) crashes under real traffic due to inaccurate VRAM estimates and unbounded KV cache allocation.

Outcome

Stabilized 70B inference on mixed GPUs, eliminating OOM crashes and maintaining throughput over 36 hours.

Workflow

• Deploy 70B model with vLLM via GPUStack
• Set --max-model-len 16384 to match real p99 prompt length
• Set --gpu-memory-utilization 0.88 to leave margin for graph capture
• Set --max-num-seqs 32 to prevent KV pool oversubscription
• Set --kv-cache-dtype fp8 to halve KV memory
• Disable swap-space to avoid latency cliffs
• Monitor TTFT and RSS to detect KV growth or preemption

Evidence

The estimator was treating context as a declared budget, not as a per‑sequence allocation multiplied by concurrency.

Source

Pin --max-model-len 16384 --gpu-memory-utilization 0.88 --kv-cache-dtype fp8 --max-num-seqs 32 --enable-chunked-prefill --swap-space 0 --disable-log-requests

Source