DeepInfer vs. vLLM on 8 B200: Which Is Faster?

说明与用途

DeepInfer‑server 集成到 GPUStack，提供在八块 NVIDIA B200 GPU 上的完整文本生成服务，显示出比 vLLM 更高的吞吐量和更低的 token 间延迟，尽管 vLLM 在首个 token 的时间和 P95 延迟方面更优。

问题

Deploying a high‑performance inference engine (DeepInfer) as a production‑ready service on GPUStack and comparing its performance to vLLM on the same hardware.

结果

DeepInfer-server achieved higher output throughput and lower inter‑token latency, while vLLM had lower time‑to‑first‑token and better P95 request latency on the same eight‑GPU setup.

工作流

• 1. Prepare runtime: install GPU drivers, container runtime, download model weights, build DeepInfer‑server image.
• 2. Import custom backend YAML into GPUStack.
• 3. Create model deployment selecting DeepInfer‑server backend and setting parallelism flags (--expert-parallel-size=8, --data-parallel-size=8, --context-parallel-size=1).
• 4. Deploy and start the service via GPUStack.
• 5. Benchmark throughput and latency against vLLM on the same hardware.

证据

We integrated DeepInfer with GPUStack through a lightweight serving project, deployed DeepSeek‑V4.1‑Flash on a single server with eight NVIDIA B200 GPUs, and compared the resulting service with vLLM under a matched DP8+EP8 setup.

来源

DeepInfer-server provides an HTTP API, streaming responses, and service lifecycle management.

来源

To keep the deployment aligned with the benchmark described later, we explicitly set --expert-parallel-size=8, --data-parallel-size=8, and --context-parallel-size=1 in the model's backend parameters.

来源