GPUStack With vLLM in Production: Pinning KV Cache Before the OOM Finds You

Описание и применение

GPUStack с vLLM стабилизировал 70‑млрд‑параметровую модель Llama на смешанных GPU, устранив OOM‑падения путём закрепления длины модели, ограничения использования памяти и количества последовательностей.

Задача

Deploying a 70B Llama model on heterogeneous GPUs caused out‑of‑memory (OOM) crashes under real traffic due to inaccurate VRAM estimates and unbounded KV cache allocation.

Результат

Stabilized 70B inference on mixed GPUs, eliminating OOM crashes and maintaining throughput over 36 hours.

Рабочий процесс

• Deploy 70B model with vLLM via GPUStack
• Set --max-model-len 16384 to match real p99 prompt length
• Set --gpu-memory-utilization 0.88 to leave margin for graph capture
• Set --max-num-seqs 32 to prevent KV pool oversubscription
• Set --kv-cache-dtype fp8 to halve KV memory
• Disable swap-space to avoid latency cliffs
• Monitor TTFT and RSS to detect KV growth or preemption

Доказательства

The estimator was treating context as a declared budget, not as a per‑sequence allocation multiplied by concurrency.

Источник

Pin --max-model-len 16384 --gpu-memory-utilization 0.88 --kv-cache-dtype fp8 --max-num-seqs 32 --enable-chunked-prefill --swap-space 0 --disable-log-requests

Источник