How 8-bit quantization shrinks an LLM to a quarter of its size — and why a single outlier weight can quietly ruin it

Описание и применение

Квантование весов больших языковых моделей до 8‑бит уменьшает их размер до четверти, сохраняя при этом приемлемую точность. Процесс включает масштабирование, округление и хранение INT8‑значений с одним коэффициентом масштабирования.

Задача

Reduce the memory footprint of large language models while maintaining acceptable accuracy.

Результат

Quantizing to INT8 reduces weight storage from 32 GB to 8 GB for an 8‑billion‑parameter model, achieving a 75% memory savings and potential inference speedup. However, quantization introduces a maximum error of half a step (≈0.01 for the example), and weights that differ by less than a step collapse into the same value, slightly degrading accuracy.

Рабочий процесс

• 1. Load the full‑precision weight tensor.
• 2. Compute the scale as max(|w|)/127 for symmetric quantization or (max(w)-min(w))/255 for asymmetric.
• 3. Clip and round each weight: q = clip(round(w / scale), -127, 127).astype(int8).
• 4. Store the INT8 tensor and the scale value.
• 5. During inference, dequantize on‑the‑fly: w_q = q.astype(float32) * scale.
• 6. Use the dequantized weights in the model’s forward pass.
• 7. Evaluate model accuracy and adjust quantization strategy if needed.

Доказательства

The size of the overall weights is simple arithmetic: number of weights × bytes per weight.

Источник

Quantization, as a concept, is simply trying to store those weights in a smaller type, like INT8.

Источник

The largest weight is 2.54, so: scale = 2.54 / 127 = 0.02

Источник

You store the INT8 values plus one scale. That's how 4 bytes per weight becomes 1.

Источник