How do you fit a 4B LLM in 1.24 GB? Trellis-coded 2-bit quantization, explained with real benchmarks

Описание и применение

Pingala‑8 использует 2‑битную квантизацию с вращением весов, обратной связью ошибок и треллис‑кодированием, чтобы уменьшить Qwen3‑4B до 1,24 ГБ, сохраняя при этом качество и ускоряя генерацию.

Задача

Compress a 4B LLM to fit in 1.24 GB while preserving quality without retraining.

Результат

Pingala‑8 compresses Qwen3‑4B to 1.24 GB (85 % smaller) with only modest quality loss (perplexity 16.64 vs 12.46) and comparable or better quiz performance, while achieving faster generation than a comparable 2‑bit codebook format.

Рабочий процесс

• Rotate weights with a Hadamard matrix to spread outliers.
• Apply LDLQ error‑feedback rounding to reduce cumulative error.
• Encode groups with trellis coding (Ungerboeck 8‑state) using Viterbi search for optimal path.
• Pack quantized weights into a gguf file compatible with llama.cpp.
• Benchmark perplexity, quiz accuracy, and generation speed on a single GPU.
• Perform ablation by removing trellis memory to assess its impact.
• Reproduce results by downloading the one‑click build and running provided scripts.

Доказательства

Qwen3‑4B ships as an 8.05 GB file of 16‑bit weights.

Источник

Pingala‑8 1.24 GB 16.64 perplexity.

Источник

Generating text 1.10x vs IQ2_XXS (same size).

Источник