I was tired of letting the OOM killer do my VRAM math, so I wrote a one-file CLI

说明与用途

ggufit 通过读取 .gguf 头部并计算权重、KV 缓存和可用显存，帮助用户在下载前判断模型是否能装进 GPU，避免 OOM 错误。

问题

Users cannot predict if a local model will fit in GPU memory before downloading, leading to OOM errors.

结果

ggufit enables users to know in advance whether a model will fit on their GPU, avoiding OOM errors and unnecessary downloads.

工作流

• 1. Point ggufit at a .gguf file or provide model specs via CLI.
• 2. ggufit parses the header for tensor table, quantization, layer count, KV heads.
• 3. It computes total VRAM needed, including KV cache and usable VRAM after driver reserve.
• 4. It outputs a verdict (e.g., TIGHT, WON'T FIT) and optional table or JSON.
• 5. If needed, adjust context length or offload layers to CPU and re-run.

证据

Point it at a .gguf file and it reads the actual header — tensor table, quantization types, layer count, KV heads — and answers the only question that matters:

来源

Two things most calculators get wrong, fixed here: KV cache is not a footnote. At 32k context, an 8B model's KV cache costs as much as its weights.

来源

Your 8 GB card doesn't have 8 GB free. The database compares against usable VRAM after driver/display/OS reserve, not the number on the box.

来源

$ python3 ggufit.py --params 8B --quant q4_k_m --ctx 32768 --gpu "RTX 4060" Verdict: WON'T FIT — short by 3.36 GB

来源