How do you fit a 4B LLM in 1.24 GB? Trellis-coded 2-bit quantization, explained with real benchmarks

Description and use

Pingala‑8 compresses Qwen3‑4B to 1.24 GB using 2‑bit quantization with rotation, error‑feedback rounding, and trellis coding, achieving modest quality loss and faster generation.

Problem

Compress a 4B LLM to fit in 1.24 GB while preserving quality without retraining.

Outcome

Pingala‑8 compresses Qwen3‑4B to 1.24 GB (85 % smaller) with only modest quality loss (perplexity 16.64 vs 12.46) and comparable or better quiz performance, while achieving faster generation than a comparable 2‑bit codebook format.

Workflow

• Rotate weights with a Hadamard matrix to spread outliers.
• Apply LDLQ error‑feedback rounding to reduce cumulative error.
• Encode groups with trellis coding (Ungerboeck 8‑state) using Viterbi search for optimal path.
• Pack quantized weights into a gguf file compatible with llama.cpp.
• Benchmark perplexity, quiz accuracy, and generation speed on a single GPU.
• Perform ablation by removing trellis memory to assess its impact.
• Reproduce results by downloading the one‑click build and running provided scripts.

Evidence

Qwen3‑4B ships as an 8.05 GB file of 16‑bit weights.

Source

Pingala‑8 1.24 GB 16.64 perplexity.

Source

Generating text 1.10x vs IQ2_XXS (same size).

Source