DeepInfer vs. vLLM on 8 B200: Which Is Faster?

Description and use

DeepInfer-server is integrated into GPUStack to provide a full text‑generation service on eight NVIDIA B200 GPUs, showing higher throughput and lower inter‑token latency than vLLM, though vLLM achieves lower time‑to‑first‑token and better P95 latency.

Problem

Deploying a high‑performance inference engine (DeepInfer) as a production‑ready service on GPUStack and comparing its performance to vLLM on the same hardware.

Outcome

DeepInfer-server achieved higher output throughput and lower inter‑token latency, while vLLM had lower time‑to‑first‑token and better P95 request latency on the same eight‑GPU setup.

Workflow

• 1. Prepare runtime: install GPU drivers, container runtime, download model weights, build DeepInfer‑server image.
• 2. Import custom backend YAML into GPUStack.
• 3. Create model deployment selecting DeepInfer‑server backend and setting parallelism flags (--expert-parallel-size=8, --data-parallel-size=8, --context-parallel-size=1).
• 4. Deploy and start the service via GPUStack.
• 5. Benchmark throughput and latency against vLLM on the same hardware.

Evidence

We integrated DeepInfer with GPUStack through a lightweight serving project, deployed DeepSeek‑V4.1‑Flash on a single server with eight NVIDIA B200 GPUs, and compared the resulting service with vLLM under a matched DP8+EP8 setup.

Source

DeepInfer-server provides an HTTP API, streaming responses, and service lifecycle management.

Source

To keep the deployment aligned with the benchmark described later, we explicitly set --expert-parallel-size=8, --data-parallel-size=8, and --context-parallel-size=1 in the model's backend parameters.

Source