A 150B MoE on a Developer Laptop: An Upper Bound on What Faster Storage Buys You

Описание и применение

На ноутбуке с 150‑млрд параметров Mixture‑of‑Experts модель работает за 180 с, при этом диск отвечает за лишь 10,7 с. Ускорение NVMe даёт максимум 6 % прироста, поэтому память остаётся критически важной.

Задача

Can a developer run a 150B mixture‑of‑experts model on a typical engineering laptop, and how much does storage speed actually affect performance?

Результат

On a 2026 engineering laptop, storage contributes only ~6 % of total latency for a 150B MoE model; faster NVMe drives yield marginal gains. Memory allocation for hot experts remains critical, as reducing RAM cache significantly degrades throughput.

Рабочий процесс

• 1. Install Colibrì v1.11.0 with gcc 13.3.0 and O3 -march=native flags. 2. Download DeepSeek V4 Flash 150B REAP‑pruned checkpoint. 3. Run Colibrì’s iobench to measure NVMe bandwidth (O_DIRECT, 13 MB blocks). 4. Configure Colibrì planner: set RAM budget, enable read/compute overlap, disable speculative decoding. 5. Execute a 200‑token generation with coli run and record timing, hit/miss stats. 6. Calculate physical I/O bytes from v4_direct reads + fallbacks. 7. Compute device time: bytes ÷ measured bandwidth. 8. Repeat with halved RAM cache (20 GB) after dropping page cache to observe impact on misses and throughput.

Доказательства

Two hundred tokens in 180.1 seconds: 1.11 tokens per second, or 0.90 seconds per token.

60.6 GB delivered by a drive measured at 5.67 GB/s is 10.7 seconds of device time across a 180-second generation.

Halving the cache cost 15% of throughput, so misses are not free and the engine's own advice to give it memory is right.