Your 7B Model Fits on a 4090 — Until You Open a 128K Context Window

Description and use

A 7B LLM’s weights fit on an RTX 4090, yet the KV cache grows with context length, batch size, and architecture, causing OOMs. Accounting for KV cache, batch size, and quantization keeps inference within GPU limits.

Problem

A 7B LLM inference server OOMs when enabling a long context window, despite the model weights fitting on a single RTX 4090.

Outcome

By accounting for KV cache, batch size, and quantization, users can keep a 7B model inference on a single RTX 4090 even with long context windows, or otherwise choose cost‑effective alternatives.

Workflow

• Calculate KV cache per token using 2×layers×kv_heads×head_dim×bytes_per_value.
• Multiply by desired context length to get total KV cache size.
• Add 16 GB weights and 2 GB overhead to determine VRAM budget.
• If budget exceeded, choose to quantize KV cache to FP8, select a GQA‑heavy architecture, or rent a larger GPU.
• Use the free GPU Advisor to walk through trade‑offs and recommend the cheapest fitting setup.

Evidence

For each token in context, the model must store a key and a value vector per layer: KV cache per token = 2 × layers × kv_heads × head_dim × bytes_per_value

Source

Same model, same weights — the context window alone decides whether you need a $0.35/hr card or a $1.50+/hr one.

Source

Quantize the cache too. KV cache in FP8 halves these numbers with minimal quality loss on most workloads.

Source