How we fit a 27B model on a 12 GB laptop GPU

Description and use

We reduced the 27‑B Qwen3.8 model to 8.31 GB using 2‑bit Pingala‑8 quantization and ran it entirely on a 12 GB laptop GPU, achieving 37.7 tokens/s.

Problem

A 27‑B Qwen3.8‑27B model weighs 53.8 GB in 16‑bit format, which cannot fit into a 12 GB laptop GPU; the goal was to run the entire model on the GPU without off‑loading layers to system RAM.

Outcome

The 27‑B model was successfully run entirely on a 12 GB laptop GPU, with a 6.5× reduction in file size and acceptable perplexity (6.89) compared to the original 7.34.

Workflow

• 1. Quantize the 16‑bit weights to a 2‑bit format using Pingala‑8 trellis coding.
• 2. Allocate bits per tensor role and depth following DASH‑Q’s plan.
• 3. Pack the quantized tensors into a 9‑part GGUF file (8.31 GB).
• 4. Load the file with patched llama.cpp on the 12 GB GPU using the –ngl 99 flag.
• 5. Run inference and measure speed and WikiText‑2 perplexity.

Evidence

Qwen3.8-27B goes from 53.8 GB of 16-bit weights to an 8.31 GB Pingala-8 file, 84.5% smaller (6.5x).

Source

It runs with every layer on one 12 GB RTX 5070 Ti Laptop GPU at 37.7 tokens/s (llama-bench, batch 1).

Source

The file needs our patched llama.cpp runtime on an NVIDIA GPU. Stock llama.cpp, Ollama and LM Studio cannot load it.

Source