Serving a 6 GB ternary 27B model: the settings that decide whether it works

Описание и применение

Развертывание 27‑млрдовой модели Qwen3.8‑27B в 6 ГБ на RTX 4090 требует использования форка llama.cpp от Prism ML, установки max_tokens=16384, уровня разума medium и max_in_flight=8 для достижения 104 токенов/с и 9,6 с на четыре одновременных запроса.

Задача

Deploying a 27B reasoning model on a single RTX 4090 requires specific runtime, token, and concurrency settings to avoid failures and achieve acceptable performance.

Результат

The model achieved 104 tokens/s per user, 12 GB VRAM peak, and 9.6 s for four concurrent requests, demonstrating that proper runtime, token limits, reasoning level, and concurrency settings are critical for successful deployment.

Рабочий процесс

• Download Prism ML’s forked llama.cpp build for Linux with CUDA 12.8
• Install missing NVIDIA libraries from Python packages
• Configure API with max_tokens=16384 and reasoning_effort="medium"
• Map chat template high to xhigh if needed
• Set max_in_flight to 8 to allow concurrent requests
• Deploy on InferenceKey with kv_unified context of 65,536 tokens
• Measure per‑request latency and memory usage

Доказательства

Stock llama.cpp will not run these files.

Источник

Set a small cap and the reply can come back empty.

Источник

After raising one setting (max_in_flight to 8), the same four finished in 9.6 s at about 159 tokens a second in total.

Источник