Run Qwen3-Coder-Next Locally on a Cost-Effective AI Home PC with llama.cpp

Описание и применение

Пошаговый план запуска модели Qwen3‑Coder‑Next на домашнем ПК с 8–16 ГБ видеопамяти, используя llama.cpp, GPU‑offloading и хранение экспертных весов в системной памяти.

Задача

Running a large MoE language model locally requires balancing GPU and CPU memory on a budget home PC.

Результат

The guide demonstrates that a modest home PC can run a 80 B‑parameter MoE model locally, providing a fast, private coding assistant via a local API.

Рабочий процесс

• Install llama.cpp via WinGet or download binaries
• Download the Qwen3‑Coder‑Next GGUF model from Hugging Face using llama cli -hf or manual download
• Start the model for quick test: llama cli -hf unsloth/Qwen3-Coder-Next-GGUF:UD-IQ2_M -c 4096
• Run a local server: llama serve -hf unsloth/Qwen3-Coder-Next-GGUF:UD-IQ2_M -c 4096 --host 127.0.0.1 --port 8080
• Configure GPU layer offloading with -ngl 999 and enable CPU‑resident MoE with --cpu-moe
• Adjust memory settings if out of memory: reduce context size, use lower quantization, lower -ngl, or close GPU‑heavy apps

Доказательства

The model page lists the UD-Q4_K_M file at roughly 49 GB. An 8 GB GPU and 32 GB of RAM will not be a comfortable Q4 setup.

Источник

For a local service that other applications can use, start llama serve...

Источник