How do you fit a 4B LLM in 1.24 GB? Trellis-coded 2-bit quantization, explained with real benchmarks

说明与用途

Pingala‑8 通过旋转、误差反馈量化和三角码实现 2‑bit 量化，将 Qwen3‑4B 压缩至 1.24 GB，保持质量并提升生成速度。

问题

Compress a 4B LLM to fit in 1.24 GB while preserving quality without retraining.

结果

Pingala‑8 compresses Qwen3‑4B to 1.24 GB (85 % smaller) with only modest quality loss (perplexity 16.64 vs 12.46) and comparable or better quiz performance, while achieving faster generation than a comparable 2‑bit codebook format.

工作流

• Rotate weights with a Hadamard matrix to spread outliers.
• Apply LDLQ error‑feedback rounding to reduce cumulative error.
• Encode groups with trellis coding (Ungerboeck 8‑state) using Viterbi search for optimal path.
• Pack quantized weights into a gguf file compatible with llama.cpp.
• Benchmark perplexity, quiz accuracy, and generation speed on a single GPU.
• Perform ablation by removing trellis memory to assess its impact.
• Reproduce results by downloading the one‑click build and running provided scripts.

证据

Qwen3‑4B ships as an 8.05 GB file of 16‑bit weights.

来源

Pingala‑8 1.24 GB 16.64 perplexity.

来源

Generating text 1.10x vs IQ2_XXS (same size).

来源