A 14.4 GB model on a 16 GB Mac: keep the right experts, not all of them

Описание и применение

На Mac mini с 16 ГБ памяти GPU можно запустить модель Gemma 4 26B‑A4B (14,4 ГБ) за счёт загрузки только 80 «горячих» экспертов на каждом слое, сохраняя исходный файл и точность модели.

Задача

Running a 14.4 GB mixture‑of‑experts LLM on a 16 GB Mac mini without exceeding GPU memory.

Результат

The approach allows a 14.4 GB mixture‑of‑experts model to run on a 16 GB Mac mini, achieving 31–33 tokens/s write speed and 33/40 task score while keeping the original model file and math intact.

Рабочий процесс

• 1. Download the official gemma‑4‑26B_q4_0‑it.gguf from Hugging Face.
• 2. Run Kraits on a 16 GB Apple‑silicon Mac: cd kraits/bin && LLAMA_SERVER=/path/to/llama-server ./kraits-server -m /path/to/gemma-4-26B_q4_0-it.gguf -c 8192 --reasoning off.
• 3. Verify output contains "Kraits loaded" to confirm successful GPU loading of the 80 hottest experts per layer.
• 4. Use the model for TypeScript code generation and testing, observing ~31 tokens/s write speed and 33/40 task score.
• 5. For larger models (e.g., Qwen3.5‑122B‑A10B) split layers and test coverage to determine feasibility on the same hardware.

Доказательства

A 16 GB Mac mini gives its GPU about 12.7 GB. Google's Gemma 4 26B-A4B, in its official 4-bit file, is 14.4 GB. On paper it doesn't fit.

Источник

The trick is that a mixture-of-experts model never uses all of itself at once.

Источник

At load, Kraits reads the full model file but puts only each layer's 80 hottest experts on the GPU.

Источник

Measured by my release tester on a Mac mini M4 with 16 GB, temperature 0, thinking off: Writing 31–33 tokens/s.

Источник