MLX vs GGUF on Apple Silicon: Benchmarking the Same Local Model Two Ways

Описание и применение

MLX и GGUF показывают сопоставимые времена сборки и работоспособность приложений, но MLX обеспечивает более низкую дисперсию и не превышает лимит в 60 минут. GGUF быстрее в вычислительных задачах, но более спайковый.

Задача

Choosing between GGUF and MLX model formats for local coding agents on Apple Silicon, balancing speed, consistency, and quality.

Результат

MLX ties GGUF on overall build time and app‑work‑ability, with lower run‑to‑run variance and no 60‑minute caps. GGUF is marginally faster on compute‑heavy tasks but shows higher peak times and occasional caps due to harness bugs.

Рабочий процесс

• 1. Prepare identical model weights in GGUF and MLX formats.
• 2. Set up a three‑node Apple Silicon fleet with LM Studio serving the model.
• 3. Run swarm‑gym in benchmark mode with the medium tier (5 runs × 3 archetypes).
• 4. Collect structured event logs, per‑task metrics, and golden‑value checks.
• 5. Analyze build times, variance, and app‑work‑ability metrics.
• 6. Identify and fix harness bugs (e.g., integrate‑verify step cap).

Доказательства

MLX had lower run‑to‑run variance (stdev 580s vs 751s), a lower p90 (3050s vs 3601s), and never hit the 60‑minute per‑run cap.

Источник

Both builds achieved ~87% app‑works (13/15) when judged by running, despite raw checks favoring MLX.

Источник