DeepInfer vs. vLLM on 8 B200: Which Is Faster?

Описание и применение

DeepInfer-server интегрирован в GPUStack, создавая полноценный сервис генерации текста на 8 GPU NVIDIA B200 и демонстрирует более высокую пропускную способность и меньшую задержку между токенами по сравнению с vLLM, хотя последний быстрее выдаёт первый токен и имеет лучшую P95 задержку.

Задача

Deploying a high‑performance inference engine (DeepInfer) as a production‑ready service on GPUStack and comparing its performance to vLLM on the same hardware.

Результат

DeepInfer-server achieved higher output throughput and lower inter‑token latency, while vLLM had lower time‑to‑first‑token and better P95 request latency on the same eight‑GPU setup.

Рабочий процесс

• 1. Prepare runtime: install GPU drivers, container runtime, download model weights, build DeepInfer‑server image.
• 2. Import custom backend YAML into GPUStack.
• 3. Create model deployment selecting DeepInfer‑server backend and setting parallelism flags (--expert-parallel-size=8, --data-parallel-size=8, --context-parallel-size=1).
• 4. Deploy and start the service via GPUStack.
• 5. Benchmark throughput and latency against vLLM on the same hardware.

Доказательства

We integrated DeepInfer with GPUStack through a lightweight serving project, deployed DeepSeek‑V4.1‑Flash on a single server with eight NVIDIA B200 GPUs, and compared the resulting service with vLLM under a matched DP8+EP8 setup.

Источник

DeepInfer-server provides an HTTP API, streaming responses, and service lifecycle management.

Источник

To keep the deployment aligned with the benchmark described later, we explicitly set --expert-parallel-size=8, --data-parallel-size=8, and --context-parallel-size=1 in the model's backend parameters.

Источник