Will it fit on my GPU? I built a calculator that finally answers it

说明与用途

StudioTV VRAM 计算器准确估算大型语言模型是否能在指定 GPU 上运行，考虑注意力类型、精度和上下文长度，并提供部署和成本比较建议。

问题

Determining whether a large language model will fit on a specific GPU given its memory constraints, precision, and context length.

结果

The calculator accurately predicts memory usage across diverse attention mechanisms, enabling users to choose appropriate precision, context length, and hardware, and to compare on‑premise deployment with cloud rental costs.

工作流

• 1. Select model, workload, context length, request concurrency, GPU, and engine.
• 2. View calculator output: required GPUs, fit status, memory breakdown, speed, and cost.
• 3. Explore weight‑format vs. context‑length grid to find optimal precision and token limits.
• 4. Review rental prices and obtain ready‑to‑paste launch commands for chosen engine.
• 5. If out of memory, use offload options such as MoE or layer‑level offloading.

证据

The usual answer is a back‑of‑the‑envelope formula: parameters times bytes per weight, plus a KV cache computed as if every layer looked at the whole context.

来源

The calculator reads each model's configuration layer by layer and computes the cache the way vLLM and llama.cpp actually allocate it, including FP8 cache formats and the extra window llama.cpp keeps on sliding‑window layers.

来源

In this example, Gemma 4 31B in Q4_K_M takes 17.1 GiB of weights, 3.7 GiB of KV cache for one 32K-token request and about 3 GiB of runtime overhead: 23.7 GiB out of the 28.8 GiB usable on the RTX 5090.

来源