A 14.4 GB model on a 16 GB Mac: keep the right experts, not all of them

说明与用途

在 16 GB Mac mini 上，通过仅加载每层 80 个最热专家，可以运行 14.4 GB 的 Gemma 4 26B‑A4B 模型，保持原始文件和模型计算不变。

问题

Running a 14.4 GB mixture‑of‑experts LLM on a 16 GB Mac mini without exceeding GPU memory.

结果

The approach allows a 14.4 GB mixture‑of‑experts model to run on a 16 GB Mac mini, achieving 31–33 tokens/s write speed and 33/40 task score while keeping the original model file and math intact.

工作流

• 1. Download the official gemma‑4‑26B_q4_0‑it.gguf from Hugging Face.
• 2. Run Kraits on a 16 GB Apple‑silicon Mac: cd kraits/bin && LLAMA_SERVER=/path/to/llama-server ./kraits-server -m /path/to/gemma-4-26B_q4_0-it.gguf -c 8192 --reasoning off.
• 3. Verify output contains "Kraits loaded" to confirm successful GPU loading of the 80 hottest experts per layer.
• 4. Use the model for TypeScript code generation and testing, observing ~31 tokens/s write speed and 33/40 task score.
• 5. For larger models (e.g., Qwen3.5‑122B‑A10B) split layers and test coverage to determine feasibility on the same hardware.

证据

A 16 GB Mac mini gives its GPU about 12.7 GB. Google's Gemma 4 26B-A4B, in its official 4-bit file, is 14.4 GB. On paper it doesn't fit.

来源

The trick is that a mixture-of-experts model never uses all of itself at once.

来源

At load, Kraits reads the full model file but puts only each layer's 80 hottest experts on the GPU.

来源

Measured by my release tester on a Mac mini M4 with 16 GB, temperature 0, thinking off: Writing 31–33 tokens/s.

来源