Your 7B Model Fits on a 4090 — Until You Open a 128K Context Window

Описание и применение

Модель 7B занимает ~16 ГБ весов, но при длинном контексте KV‑кеш может превысить доступную память RTX 4090, вызывая OOM. Учёт KV‑кеша, размера пакета и квантизации позволяет удержать модель на одной карте.

Задача

A 7B LLM inference server OOMs when enabling a long context window, despite the model weights fitting on a single RTX 4090.

Результат

By accounting for KV cache, batch size, and quantization, users can keep a 7B model inference on a single RTX 4090 even with long context windows, or otherwise choose cost‑effective alternatives.

Рабочий процесс

• Calculate KV cache per token using 2×layers×kv_heads×head_dim×bytes_per_value.
• Multiply by desired context length to get total KV cache size.
• Add 16 GB weights and 2 GB overhead to determine VRAM budget.
• If budget exceeded, choose to quantize KV cache to FP8, select a GQA‑heavy architecture, or rent a larger GPU.
• Use the free GPU Advisor to walk through trade‑offs and recommend the cheapest fitting setup.

Доказательства

For each token in context, the model must store a key and a value vector per layer: KV cache per token = 2 × layers × kv_heads × head_dim × bytes_per_value

Источник

Same model, same weights — the context window alone decides whether you need a $0.35/hr card or a $1.50+/hr one.

Источник

Quantize the cache too. KV cache in FP8 halves these numbers with minimal quality loss on most workloads.

Источник