Running a 133 GB MoE model on an 8 GB GPU at 11 tokens/s by streaming experts from NVMe

Description and use

Achieved 11.65 tokens/s on an 8 GB GPU for a 133 GB MoE model by streaming experts from NVMe, with correctness matching the reference implementation.

Problem

Running a 133 GB mixture‑of‑experts model on a home machine with only 8 GB VRAM while maintaining interactive token throughput.

Outcome

Achieved 11.65 tokens/s on an 8 GB GPU for a 133 GB MoE model by streaming experts from NVMe, with correctness matching the reference implementation.

Workflow

• Repack experts into contiguous 2,764,800‑byte blocks for efficient pread and pinned memory copy.
• Implement an LRU RAM cache (16 GB) with pinned memory and reader threads, plus two GPU slot banks for overlapping load and compute.
• Prefetch experts for the next layer during the current layer’s GPU work, ensuring prefetch occurs in the idle gap rather than concurrently.
• Develop custom NVFP4 SwiGLU and FP8 row‑scaled matrix‑vector kernels for expert computation.
• Batch prompt processing: run entire prompt through each layer at once, reading experts in 32‑expert batches to reduce repeated disk reads.
• Read only required PLE rows from the 53.7 GB n‑gram table, leveraging NVMe random read performance.
• Speculative copies: run next layer’s router on current layer’s input to start expert copies 65 % of the time early.
• Use CUDA graphs for the 36 one‑token DeltaNet layers to reduce CPU‑GPU sync overhead.
• Pin the decode thread to a P‑core to avoid core‑migration latency.

Evidence

I run local models on one home machine: an RTX 5060 with 8 GB, a Core Ultra 5 225F, 31 GiB of RAM and a Gen5 NVMe drive used only for model files.

Source

It runs as a normal chat model in Open WebUI at 9–12 tokens/s, and its output matches the Hugging Face reference implementation.

Source

The first version processed 512 tokens at a time, and each chunk needed nearly every expert, so the whole expert set was read again for every chunk. Running the whole prompt through each layer at once, with experts read in batches of 32, took a 2451-token prompt from 71.6 s to 23.3 s.

Source

Speculative copies. Run layer L+1's router on layer L's MoE input. It picks the right top-10 65% of the time — enough to start most copies one layer early.

Source

live chat, 1000-token answer, after: 11.65 tokens/s

Source