Running Local LLMs on Mac Mini: What Actually Breaks

说明与用途

在 Mac Mini 和 Mac Studio 上本地运行大型 LLM 经常因统一内存限制、热量限制和上下文长度扩展而失败。了解这些因素可帮助正确选择硬件和模型大小。

问题

Local LLM inference on Apple Silicon often fails due to memory sharing, thermal limits, and context length scaling, leading to performance drops or crashes.

结果

By accounting for unified memory limits, thermal throttling, and context‑length scaling, users can avoid performance regressions and choose appropriate Mac hardware for their LLM workloads.

工作流

• 1. Estimate model weight memory: 0.6 GB per 1 B params for Q4 models.
• 2. Calculate KV cache: 2×num_layers×num_heads×head_dim×context_length×bytes_per_element.
• 3. Reserve 8‑10 GB for OS and apps.
• 4. Run inference server (e.g., ollama).
• 5. Monitor power/thermals with sudo powermetrics -s cpu_power,gpu_power -n 1 in a loop.
• 6. Detect GPU power drop → thermal throttling; reduce model size or context.
• 7. Profile context effects with a script using llama‑cpp‑python, measuring first‑token latency and tokens/sec.
• 8. Choose hardware: Mac Mini for 7‑13 B models, Mac Studio for 30‑70 B or long‑context workloads.
• 9. Adjust expectations: expect throughput halving between 8K and 32K contexts on 32 GB Mac Mini.

证据

Unified memory means the GPU and CPU share the same pool, which sounds generous. It is, until you remember that macOS, your IDE, your browser, and the OS cache all live in that same pool.

来源

The first sign isn't a fan curve, it's tokens-per-second dropping by 20-30% after ten minutes.

来源

Most local stacks handle 4K or 8K contexts fine. Push to 32K or 64K and you'll see two things go wrong: first-token latency climbs and tokens per second drops.

来源