I bought Thunderbolt 5 and got USB4 40G: re-benchmarking a two-machine llama.cpp RPC setup across three links and six models

Description and use

The author benchmarked a two‑machine llama.cpp RPC setup over 5GbE, fake TB5, and USB10G, finding USB10G offers higher unidirectional bandwidth but both lag behind 5GbE in bidirectional throughput. Prefill gains are modest, and concurrency is limited to short prompts.

Problem

Assess how different network links (5GbE, fake TB5, USB 10GbE) affect performance of a two‑machine llama.cpp RPC setup across six models.

Outcome

USB10G achieved higher unidirectional bandwidth (7.28 Gbps) than fake TB5 (4.38 Gbps) but both were lower than 5GbE (4.71 Gbps). Prefill latency improved modestly (+5–17%) with higher bandwidth, while decode latency decreased by up to 1 ms/token on USB10G. Concurrency limits were similar across links, with genuine concurrency only achievable for short prompts.

Workflow

• 1. Acquire and connect cables for 5GbE, fake TB5, and USB10G.
• 2. Install Realtek RTL8159 driver for USB10G.
• 3. Run single‑stream prefill and decode tests for each model on each link.
• 4. Record bandwidth, latency, and concurrency limits.
• 5. Analyze results and compare across links.

Evidence

TB5 failed to establish end-to-end connectivity on the ASMedia4242 chip.

Source

USB10G only shown here; see the table above for the relative relationship across all three links.

Source

The whole thing took a week of tuning and 72 hours of actual test runtime, with ambient temperature around 35°C.

Source