Run Qwen3-Coder-Next Locally on a Cost-Effective AI Home PC with llama.cpp

Description and use

Step‑by‑step guide to run Qwen3‑Coder‑Next locally on a budget home PC using llama.cpp, GPU offloading, and CPU‑resident MoE weights.

Problem

Running a large MoE language model locally requires balancing GPU and CPU memory on a budget home PC.

Outcome

The guide demonstrates that a modest home PC can run a 80 B‑parameter MoE model locally, providing a fast, private coding assistant via a local API.

Workflow

• Install llama.cpp via WinGet or download binaries
• Download the Qwen3‑Coder‑Next GGUF model from Hugging Face using llama cli -hf or manual download
• Start the model for quick test: llama cli -hf unsloth/Qwen3-Coder-Next-GGUF:UD-IQ2_M -c 4096
• Run a local server: llama serve -hf unsloth/Qwen3-Coder-Next-GGUF:UD-IQ2_M -c 4096 --host 127.0.0.1 --port 8080
• Configure GPU layer offloading with -ngl 999 and enable CPU‑resident MoE with --cpu-moe
• Adjust memory settings if out of memory: reduce context size, use lower quantization, lower -ngl, or close GPU‑heavy apps

Evidence

The model page lists the UD-Q4_K_M file at roughly 49 GB. An 8 GB GPU and 32 GB of RAM will not be a comfortable Q4 setup.

Source

For a local service that other applications can use, start llama serve...

Source