Running Local LLMs on Mac Mini: What Actually Breaks

Описание и применение

На Mac Mini и Mac Studio локальное выполнение больших моделей LLM часто падает из‑за ограничений единой памяти, тепловых ограничений и роста длины контекста. Понимание этих факторов позволяет правильно выбирать оборудование и размер модели.

Задача

Local LLM inference on Apple Silicon often fails due to memory sharing, thermal limits, and context length scaling, leading to performance drops or crashes.

Результат

By accounting for unified memory limits, thermal throttling, and context‑length scaling, users can avoid performance regressions and choose appropriate Mac hardware for their LLM workloads.

Рабочий процесс

• 1. Estimate model weight memory: 0.6 GB per 1 B params for Q4 models.
• 2. Calculate KV cache: 2×num_layers×num_heads×head_dim×context_length×bytes_per_element.
• 3. Reserve 8‑10 GB for OS and apps.
• 4. Run inference server (e.g., ollama).
• 5. Monitor power/thermals with sudo powermetrics -s cpu_power,gpu_power -n 1 in a loop.
• 6. Detect GPU power drop → thermal throttling; reduce model size or context.
• 7. Profile context effects with a script using llama‑cpp‑python, measuring first‑token latency and tokens/sec.
• 8. Choose hardware: Mac Mini for 7‑13 B models, Mac Studio for 30‑70 B or long‑context workloads.
• 9. Adjust expectations: expect throughput halving between 8K and 32K contexts on 32 GB Mac Mini.

Доказательства

Unified memory means the GPU and CPU share the same pool, which sounds generous. It is, until you remember that macOS, your IDE, your browser, and the OS cache all live in that same pool.

Источник

The first sign isn't a fan curve, it's tokens-per-second dropping by 20-30% after ten minutes.

Источник

Most local stacks handle 4K or 8K contexts fine. Push to 32K or 64K and you'll see two things go wrong: first-token latency climbs and tokens per second drops.

Источник