Serving a 6 GB ternary 27B model: the settings that decide whether it works

Description and use

Deploying a 27B Qwen3.8‑27B model compressed to 6 GB on an RTX 4090 requires Prism ML’s llama.cpp fork, max_tokens=16384, medium reasoning effort, and max_in_flight=8 to achieve 104 tokens/s per user and 9.6 s for four concurrent requests.

Problem

Deploying a 27B reasoning model on a single RTX 4090 requires specific runtime, token, and concurrency settings to avoid failures and achieve acceptable performance.

Outcome

The model achieved 104 tokens/s per user, 12 GB VRAM peak, and 9.6 s for four concurrent requests, demonstrating that proper runtime, token limits, reasoning level, and concurrency settings are critical for successful deployment.

Workflow

• Download Prism ML’s forked llama.cpp build for Linux with CUDA 12.8
• Install missing NVIDIA libraries from Python packages
• Configure API with max_tokens=16384 and reasoning_effort="medium"
• Map chat template high to xhigh if needed
• Set max_in_flight to 8 to allow concurrent requests
• Deploy on InferenceKey with kv_unified context of 65,536 tokens
• Measure per‑request latency and memory usage

Evidence

Stock llama.cpp will not run these files.

Source

Set a small cap and the reply can come back empty.

Source

After raising one setting (max_in_flight to 8), the same four finished in 9.6 s at about 159 tokens a second in total.

Source