A 14.4 GB model on a 16 GB Mac: keep the right experts, not all of them

Description and use

A 16 GB Mac mini can run the 14.4 GB Gemma 4 26B‑A4B model by loading only the 80 hottest experts per layer, preserving the original file and model math.

Problem

Running a 14.4 GB mixture‑of‑experts LLM on a 16 GB Mac mini without exceeding GPU memory.

Outcome

The approach allows a 14.4 GB mixture‑of‑experts model to run on a 16 GB Mac mini, achieving 31–33 tokens/s write speed and 33/40 task score while keeping the original model file and math intact.

Workflow

• 1. Download the official gemma‑4‑26B_q4_0‑it.gguf from Hugging Face.
• 2. Run Kraits on a 16 GB Apple‑silicon Mac: cd kraits/bin && LLAMA_SERVER=/path/to/llama-server ./kraits-server -m /path/to/gemma-4-26B_q4_0-it.gguf -c 8192 --reasoning off.
• 3. Verify output contains "Kraits loaded" to confirm successful GPU loading of the 80 hottest experts per layer.
• 4. Use the model for TypeScript code generation and testing, observing ~31 tokens/s write speed and 33/40 task score.
• 5. For larger models (e.g., Qwen3.5‑122B‑A10B) split layers and test coverage to determine feasibility on the same hardware.

Evidence

A 16 GB Mac mini gives its GPU about 12.7 GB. Google's Gemma 4 26B-A4B, in its official 4-bit file, is 14.4 GB. On paper it doesn't fit.

Source

The trick is that a mixture-of-experts model never uses all of itself at once.

Source

At load, Kraits reads the full model file but puts only each layer's 80 hottest experts on the GPU.

Source

Measured by my release tester on a Mac mini M4 with 16 GB, temperature 0, thinking off: Writing 31–33 tokens/s.

Source