How 8-bit quantization shrinks an LLM to a quarter of its size — and why a single outlier weight can quietly ruin it

说明与用途

将大型语言模型权重量化为8位可将模型大小缩小到四分之一，同时保持可接受的准确性。工作流程包括缩放、四舍五入、存储INT8值以及单个缩放因子。

问题

Reduce the memory footprint of large language models while maintaining acceptable accuracy.

结果

Quantizing to INT8 reduces weight storage from 32 GB to 8 GB for an 8‑billion‑parameter model, achieving a 75% memory savings and potential inference speedup. However, quantization introduces a maximum error of half a step (≈0.01 for the example), and weights that differ by less than a step collapse into the same value, slightly degrading accuracy.

工作流

• 1. Load the full‑precision weight tensor.
• 2. Compute the scale as max(|w|)/127 for symmetric quantization or (max(w)-min(w))/255 for asymmetric.
• 3. Clip and round each weight: q = clip(round(w / scale), -127, 127).astype(int8).
• 4. Store the INT8 tensor and the scale value.
• 5. During inference, dequantize on‑the‑fly: w_q = q.astype(float32) * scale.
• 6. Use the dequantized weights in the model’s forward pass.
• 7. Evaluate model accuracy and adjust quantization strategy if needed.

证据

The size of the overall weights is simple arithmetic: number of weights × bytes per weight.

来源

Quantization, as a concept, is simply trying to store those weights in a smaller type, like INT8.

来源

The largest weight is 2.54, so: scale = 2.54 / 127 = 0.02

来源

You store the INT8 values plus one scale. That's how 4 bytes per weight becomes 1.

来源