Running DeepSeek V4 Flash on Two DGX Sparks over 10GbE

Description and use

The two‑node setup over 10 GbE achieved stable inference with 30 tok/s per stream and 85 tok/s aggregate, matching the performance of the official recipe on a 200 Gb/s link within a factor of two.

Problem

The official FP8 DeepSeek V4 Flash 0731 checkpoint does not fit on a single 128 GB DGX Spark, and the recommended 200 Gb/s ConnectX-7 fabric was unavailable, so the author needed a way to run the model across two GPUs using only the built‑in 10 GbE ports.

Outcome

The two‑node setup over 10 GbE achieved stable inference with 30 tok/s per stream and 85 tok/s aggregate, matching the performance of the official recipe on a 200 Gb/s link within a factor of two.

Workflow

• 1. Verify 10 GbE connectivity: transfer 1 GiB TCP to confirm ~9.9 Gb/s each direction.
• 2. Configure point‑to‑point subnet on the two RJ45 ports (10.10.10.1/30 and 10.10.10.2/30) with MTU 9000 via NetworkManager.
• 3. Stage the full 167 GB checkpoint on both nodes and enable Hugging Face offline mode.
• 4. Set environment variables on the head node: WORKER_HOST, MASTER_ADDR, MASTER_PORT, NCCL_NET=Socket, NCCL_SOCKET_IFNAME=enP7s7, GLOO_SOCKET_IFNAME=enP7s7, VLLM_HOST_IP, WORKER_VLLM_HOST_IP, HF_HUB_OFFLINE=1, MAX_MODEL_LEN=1048576, MAX_NUM_SEQS=6, MTP_NUM_TOKENS=5, and disable IB with NCCL_IB_DISABLE=1, NCCL_IB_HCA=unused, NCCL_IB_GID_AUTO=0.
• 5. Launch the vLLM tensor‑parallel recipe (ghcr.io/anemll/dspark-vllm-gx10:0.1.1) with the above settings.
• 6. Monitor logs for “Using enP7s7:10.10.10.1 NET/Socket” to confirm TCP transport.
• 7. Run inference benchmarks (256‑token to 131 072‑token prompts, 1–6 concurrent streams) and record TTFT, per‑stream decode, and aggregate throughput.
• 8. Verify no NCCL, CUDA, or OOM errors during the sweep.

Evidence

Two 128GB GB10 computers can run the official FP8 DeepSeek V4 Flash 0731 checkpoint as one tensor‑parallel server over their built‑in 10Gb Ethernet ports.

Source

The recipe expected the ConnectX/RoCE path. Its startup logic tried to discover an InfiniBand or RoCE HCA and a GID before NCCL had a chance to fall back to an IP socket.

Source

Both nodes also needed a complete local copy of the model. The 167GB checkpoint does not cross the cable at startup on every run, and it definitely does not cross it for every prompt.

Source