Will it fit on my GPU? I built a calculator that finally answers it

Описание и применение

Калькулятор VRAM StudioTV позволяет точно оценить, поместится ли большая языковая модель на конкретной GPU, учитывая тип внимания, точность и длину контекста, а также предоставляет рекомендации по развертыванию и сравнению стоимости аренды.

Задача

Determining whether a large language model will fit on a specific GPU given its memory constraints, precision, and context length.

Результат

The calculator accurately predicts memory usage across diverse attention mechanisms, enabling users to choose appropriate precision, context length, and hardware, and to compare on‑premise deployment with cloud rental costs.

Рабочий процесс

• 1. Select model, workload, context length, request concurrency, GPU, and engine.
• 2. View calculator output: required GPUs, fit status, memory breakdown, speed, and cost.
• 3. Explore weight‑format vs. context‑length grid to find optimal precision and token limits.
• 4. Review rental prices and obtain ready‑to‑paste launch commands for chosen engine.
• 5. If out of memory, use offload options such as MoE or layer‑level offloading.

Доказательства

The usual answer is a back‑of‑the‑envelope formula: parameters times bytes per weight, plus a KV cache computed as if every layer looked at the whole context.

Источник

The calculator reads each model's configuration layer by layer and computes the cache the way vLLM and llama.cpp actually allocate it, including FP8 cache formats and the extra window llama.cpp keeps on sliding‑window layers.

Источник

In this example, Gemma 4 31B in Q4_K_M takes 17.1 GiB of weights, 3.7 GiB of KV cache for one 32K-token request and about 3 GiB of runtime overhead: 23.7 GiB out of the 28.8 GiB usable on the RTX 5090.

Источник