GPUStack With vLLM in Production: Pinning KV Cache Before the OOM Finds You

说明与用途

GPUStack 与 vLLM 通过固定最大模型长度、限制 GPU 内存使用和序列数，稳定了在混合 GPU 上的 70B Llama 模型，避免了 OOM 崩溃。

问题

Deploying a 70B Llama model on heterogeneous GPUs caused out‑of‑memory (OOM) crashes under real traffic due to inaccurate VRAM estimates and unbounded KV cache allocation.

结果

Stabilized 70B inference on mixed GPUs, eliminating OOM crashes and maintaining throughput over 36 hours.

工作流

• Deploy 70B model with vLLM via GPUStack
• Set --max-model-len 16384 to match real p99 prompt length
• Set --gpu-memory-utilization 0.88 to leave margin for graph capture
• Set --max-num-seqs 32 to prevent KV pool oversubscription
• Set --kv-cache-dtype fp8 to halve KV memory
• Disable swap-space to avoid latency cliffs
• Monitor TTFT and RSS to detect KV growth or preemption

证据

The estimator was treating context as a declared budget, not as a per‑sequence allocation multiplied by concurrency.

来源

Pin --max-model-len 16384 --gpu-memory-utilization 0.88 --kv-cache-dtype fp8 --max-num-seqs 32 --enable-chunked-prefill --swap-space 0 --disable-log-requests

来源