Can GPUStack Double GLM Inference Performance on 4 DGX Spark Nodes? From 24.3 to 51.3 tok/s

Description and use

By tuning network settings for RoCE, correcting MTP weight quantization, and aligning KV‑cache layouts, the cluster achieved a 2.11× increase in single‑stream decode throughput and a 44% increase in aggregate throughput.

Problem

Achieve higher inference throughput for GLM-5.3-Flash-NVFP4 on a cluster of NVIDIA DGX Spark nodes.

Outcome

By tuning network settings for RoCE, correcting MTP weight quantization, and aligning KV‑cache layouts, the cluster achieved a 2.11× increase in single‑stream decode throughput and a 44% increase in aggregate throughput.

Workflow

• Connect four DGX Spark nodes and register them in GPUStack.
• Configure vLLM with GLM‑5.3‑Flash‑NVFP4, B12X attention backend, FP8 KV cache, TP=4, PP=1.
• Set host MTU to 9000 and switch MTU to 9216 (effective 9000).
• Apply NIC‑side QoS: enable DSCP trust and PFC for priority 3 using mlnx_qos.
• Configure NCCL to use both logical RDMA interfaces and set appropriate TC and QPS.
• Run NCCL AllReduce test to verify bandwidth increase from 4.0 GB/s to 22.9 GB/s.
• Identify MTP weight loading issue: add quantization‑exemption rules for layer 45 on all nodes.
• Delete GPUStack model instance to reload weights.
• Resolve KV‑cache layout conflict by using B12X attention backend for both main and draft paths.
• Test speculative depth 3 vs 5; select depth 3 based on higher throughput.
• Re‑run benchmarks to confirm performance gains.
• Document configuration and results for future reference.

Evidence

single-stream decode throughput increased from 24.3 to 51.3 tok/s

Source

MTP weights were loaded with the wrong quantization format

Source

Use a Consistent MTU – set the host MTU to 9000 and persisted it through netplan

Source

NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0 NCCL_IB_TC=104 NCCL_IB_QPS_PER_CONNECTION=4 NCCL_IB_SPLIT_DATA_ON_QPS=1

Source