EAGLE-4 does not run on a Mac. I measured MTP instead: +92% on one model, -19% on another

Описание и применение

MTP ускоряет генерацию кода на модели Qwen3.8‑27B на 92 %, но замедляет генерацию прозы на модели Ornith‑1.5‑35B‑A3B на 19 %.

Задача

Determine if speculative decoding method EAGLE-4 improves inference speed on Apple Silicon, and compare with existing MTP technique.

Результат

MTP significantly accelerates dense Qwen3.8‑27B code generation (≈1.92×) but slightly degrades MoE Ornith‑1.5‑35B‑A3B prose generation (≈0.81×). Prefill times are unaffected or slightly slower with MTP. Concurrency benefits are mixed: dense model gains up to 1.73× with four requests, while MoE model suffers slowdown.

Рабочий процесс

• 1. Set up Mac Studio with Apple M2 Ultra, 64 GB unified memory, macOS 26.5.
• 2. Install oMLX 0.7.1rc1 and load models: Qwen3.8‑27B oQ4e‑mtp, Ornith‑1.5‑35B‑A3B oQ4e‑mtp, and Qwen3‑4B‑Instruct‑2507 4‑bit.
• 3. Run each model in four phases: MTP off, MTP on, MTP off, MTP on, with fresh server per phase.
• 4. Use temperature 0, streaming, one request at a time, no prefix cache, no memory guard.
• 5. Execute 10 runs per prompt per phase (Kotlin coding task and 350‑word prose), 512 max tokens, warm‑up runs excluded.
• 6. Record decode tokens per second, prefill time, and concurrency metrics.
• 7. Analyze results: compute mean tok/s, on/off ratios, parked runs, and compare across models and scenarios.

Доказательства

EAGLE-4 as released does not run on a Mac.

Источник

On a dense 27B model, MTP made code generation 92% faster (thinking off, one request).

Источник

On a 35B mixture-of-experts (MoE) model, MTP made prose generation 19% slower (thinking off, one request).

Источник

The control model, Qwen3-4B, ran at 159.2 to 159.9 tok/s in all four scenarios.

Источник