Running Local LLMs on Mac Mini: What Actually Breaks

Description and use

Local LLM inference on Mac Mini/Studio often fails due to unified memory limits, thermal throttling, and context length scaling. Understanding these factors enables proper hardware selection and model sizing.

Problem

Local LLM inference on Apple Silicon often fails due to memory sharing, thermal limits, and context length scaling, leading to performance drops or crashes.

Outcome

By accounting for unified memory limits, thermal throttling, and context‑length scaling, users can avoid performance regressions and choose appropriate Mac hardware for their LLM workloads.

Workflow

• 1. Estimate model weight memory: 0.6 GB per 1 B params for Q4 models.
• 2. Calculate KV cache: 2×num_layers×num_heads×head_dim×context_length×bytes_per_element.
• 3. Reserve 8‑10 GB for OS and apps.
• 4. Run inference server (e.g., ollama).
• 5. Monitor power/thermals with sudo powermetrics -s cpu_power,gpu_power -n 1 in a loop.
• 6. Detect GPU power drop → thermal throttling; reduce model size or context.
• 7. Profile context effects with a script using llama‑cpp‑python, measuring first‑token latency and tokens/sec.
• 8. Choose hardware: Mac Mini for 7‑13 B models, Mac Studio for 30‑70 B or long‑context workloads.
• 9. Adjust expectations: expect throughput halving between 8K and 32K contexts on 32 GB Mac Mini.

Evidence

Unified memory means the GPU and CPU share the same pool, which sounds generous. It is, until you remember that macOS, your IDE, your browser, and the OS cache all live in that same pool.

Source

The first sign isn't a fan curve, it's tokens-per-second dropping by 20-30% after ten minutes.

Source

Most local stacks handle 4K or 8K contexts fine. Push to 32K or 64K and you'll see two things go wrong: first-token latency climbs and tokens per second drops.

Source