I was tired of letting the OOM killer do my VRAM math, so I wrote a one-file CLI

Description and use

ggufit lets users pre‑check if a model fits in GPU memory by reading the .gguf header and calculating weights, KV cache, and usable VRAM, avoiding OOM errors.

Problem

Users cannot predict if a local model will fit in GPU memory before downloading, leading to OOM errors.

Outcome

ggufit enables users to know in advance whether a model will fit on their GPU, avoiding OOM errors and unnecessary downloads.

Workflow

• 1. Point ggufit at a .gguf file or provide model specs via CLI.
• 2. ggufit parses the header for tensor table, quantization, layer count, KV heads.
• 3. It computes total VRAM needed, including KV cache and usable VRAM after driver reserve.
• 4. It outputs a verdict (e.g., TIGHT, WON'T FIT) and optional table or JSON.
• 5. If needed, adjust context length or offload layers to CPU and re-run.

Evidence

Point it at a .gguf file and it reads the actual header — tensor table, quantization types, layer count, KV heads — and answers the only question that matters:

Source

Two things most calculators get wrong, fixed here: KV cache is not a footnote. At 32k context, an 8B model's KV cache costs as much as its weights.

Source

Your 8 GB card doesn't have 8 GB free. The database compares against usable VRAM after driver/display/OS reserve, not the number on the box.

Source

$ python3 ggufit.py --params 8B --quant q4_k_m --ctx 32768 --gpu "RTX 4060" Verdict: WON'T FIT — short by 3.36 GB

Source