I bought Thunderbolt 5 and got USB4 40G: re-benchmarking a two-machine llama.cpp RPC setup across three links and six models

Описание и применение

Автор сравнил производительность двухмашинной установки llama.cpp по трем сетевым каналам (5GbE, фейковый TB5 и USB10G), выявив, что USB10G обеспечивает лучшую пропускную способность, но обе сети уступают 5GbE в двунаправленной скорости. Предварительная обработка получает небольшие улучшения, а одновременная обработка потоков ограничена короткими запросами.

Задача

Assess how different network links (5GbE, fake TB5, USB 10GbE) affect performance of a two‑machine llama.cpp RPC setup across six models.

Результат

USB10G achieved higher unidirectional bandwidth (7.28 Gbps) than fake TB5 (4.38 Gbps) but both were lower than 5GbE (4.71 Gbps). Prefill latency improved modestly (+5–17%) with higher bandwidth, while decode latency decreased by up to 1 ms/token on USB10G. Concurrency limits were similar across links, with genuine concurrency only achievable for short prompts.

Рабочий процесс

• 1. Acquire and connect cables for 5GbE, fake TB5, and USB10G.
• 2. Install Realtek RTL8159 driver for USB10G.
• 3. Run single‑stream prefill and decode tests for each model on each link.
• 4. Record bandwidth, latency, and concurrency limits.
• 5. Analyze results and compare across links.

Доказательства

TB5 failed to establish end-to-end connectivity on the ASMedia4242 chip.

Источник

USB10G only shown here; see the table above for the relative relationship across all three links.

Источник

The whole thing took a week of tuning and 72 hours of actual test runtime, with ambient temperature around 35°C.

Источник