I was tired of letting the OOM killer do my VRAM math, so I wrote a one-file CLI

Описание и применение

ggufit позволяет заранее определить, поместится ли модель в видеопамять GPU, учитывая веса, KV‑кеш и доступную память, тем самым предотвращая ошибки OOM.

Задача

Users cannot predict if a local model will fit in GPU memory before downloading, leading to OOM errors.

Результат

ggufit enables users to know in advance whether a model will fit on their GPU, avoiding OOM errors and unnecessary downloads.

Рабочий процесс

• 1. Point ggufit at a .gguf file or provide model specs via CLI.
• 2. ggufit parses the header for tensor table, quantization, layer count, KV heads.
• 3. It computes total VRAM needed, including KV cache and usable VRAM after driver reserve.
• 4. It outputs a verdict (e.g., TIGHT, WON'T FIT) and optional table or JSON.
• 5. If needed, adjust context length or offload layers to CPU and re-run.

Доказательства

Point it at a .gguf file and it reads the actual header — tensor table, quantization types, layer count, KV heads — and answers the only question that matters:

Источник

Two things most calculators get wrong, fixed here: KV cache is not a footnote. At 32k context, an 8B model's KV cache costs as much as its weights.

Источник

Your 8 GB card doesn't have 8 GB free. The database compares against usable VRAM after driver/display/OS reserve, not the number on the box.

Источник

$ python3 ggufit.py --params 8B --quant q4_k_m --ctx 32768 --gpu "RTX 4060" Verdict: WON'T FIT — short by 3.36 GB

Источник