How we fit a 27B model on a 12 GB laptop GPU

说明与用途

我们将27B Qwen3.8模型压缩至8.31GB，使用2位Pingala‑8量化，并在12GB笔记本GPU上完整运行，速度为37.7 tokens/s。

问题

A 27‑B Qwen3.8‑27B model weighs 53.8 GB in 16‑bit format, which cannot fit into a 12 GB laptop GPU; the goal was to run the entire model on the GPU without off‑loading layers to system RAM.

结果

The 27‑B model was successfully run entirely on a 12 GB laptop GPU, with a 6.5× reduction in file size and acceptable perplexity (6.89) compared to the original 7.34.

工作流

• 1. Quantize the 16‑bit weights to a 2‑bit format using Pingala‑8 trellis coding.
• 2. Allocate bits per tensor role and depth following DASH‑Q’s plan.
• 3. Pack the quantized tensors into a 9‑part GGUF file (8.31 GB).
• 4. Load the file with patched llama.cpp on the 12 GB GPU using the –ngl 99 flag.
• 5. Run inference and measure speed and WikiText‑2 perplexity.

证据

Qwen3.8-27B goes from 53.8 GB of 16-bit weights to an 8.31 GB Pingala-8 file, 84.5% smaller (6.5x).

来源

It runs with every layer on one 12 GB RTX 5070 Ti Laptop GPU at 37.7 tokens/s (llama-bench, batch 1).

来源

The file needs our patched llama.cpp runtime on an NVIDIA GPU. Stock llama.cpp, Ollama and LM Studio cannot load it.

来源