Serving a 6 GB ternary 27B model: the settings that decide whether it works

说明与用途

在 RTX 4090 上部署 6 GB 的 27B Qwen3.8‑27B 模型，需要使用 Prism ML 的 llama.cpp 分支，设置 max_tokens=16384、medium 认知级别和 max_in_flight=8，以实现每用户 104 tokens/s 和四个并发请求 9.6 s 的性能。

问题

Deploying a 27B reasoning model on a single RTX 4090 requires specific runtime, token, and concurrency settings to avoid failures and achieve acceptable performance.

结果

The model achieved 104 tokens/s per user, 12 GB VRAM peak, and 9.6 s for four concurrent requests, demonstrating that proper runtime, token limits, reasoning level, and concurrency settings are critical for successful deployment.

工作流

• Download Prism ML’s forked llama.cpp build for Linux with CUDA 12.8
• Install missing NVIDIA libraries from Python packages
• Configure API with max_tokens=16384 and reasoning_effort="medium"
• Map chat template high to xhigh if needed
• Set max_in_flight to 8 to allow concurrent requests
• Deploy on InferenceKey with kv_unified context of 65,536 tokens
• Measure per‑request latency and memory usage

证据

Stock llama.cpp will not run these files.

来源

Set a small cap and the reply can come back empty.

来源

After raising one setting (max_in_flight to 8), the same four finished in 9.6 s at about 159 tokens a second in total.

来源