Your 7B Model Fits on a 4090 — Until You Open a 128K Context Window

说明与用途

7B LLM 的权重可放入 RTX 4090，但 KV 缓存随上下文长度、批量大小和架构增长，导致 OOM。考虑 KV 缓存、批量大小和量化可保持推理在 GPU 范围内。

问题

A 7B LLM inference server OOMs when enabling a long context window, despite the model weights fitting on a single RTX 4090.

结果

By accounting for KV cache, batch size, and quantization, users can keep a 7B model inference on a single RTX 4090 even with long context windows, or otherwise choose cost‑effective alternatives.

工作流

• Calculate KV cache per token using 2×layers×kv_heads×head_dim×bytes_per_value.
• Multiply by desired context length to get total KV cache size.
• Add 16 GB weights and 2 GB overhead to determine VRAM budget.
• If budget exceeded, choose to quantize KV cache to FP8, select a GQA‑heavy architecture, or rent a larger GPU.
• Use the free GPU Advisor to walk through trade‑offs and recommend the cheapest fitting setup.

证据

For each token in context, the model must store a key and a value vector per layer: KV cache per token = 2 × layers × kv_heads × head_dim × bytes_per_value

来源

Same model, same weights — the context window alone decides whether you need a $0.35/hr card or a $1.50+/hr one.

来源

Quantize the cache too. KV cache in FP8 halves these numbers with minimal quality loss on most workloads.

来源