How 8-bit quantization shrinks an LLM to a quarter of its size — and why a single outlier weight can quietly ruin it

Description and use

Quantizing LLM weights to 8‑bit reduces model size to a quarter while preserving acceptable accuracy. The workflow involves scaling, rounding, storing INT8 values with a single scale factor.

Problem

Reduce the memory footprint of large language models while maintaining acceptable accuracy.

Outcome

Quantizing to INT8 reduces weight storage from 32 GB to 8 GB for an 8‑billion‑parameter model, achieving a 75% memory savings and potential inference speedup. However, quantization introduces a maximum error of half a step (≈0.01 for the example), and weights that differ by less than a step collapse into the same value, slightly degrading accuracy.

Workflow

• 1. Load the full‑precision weight tensor.
• 2. Compute the scale as max(|w|)/127 for symmetric quantization or (max(w)-min(w))/255 for asymmetric.
• 3. Clip and round each weight: q = clip(round(w / scale), -127, 127).astype(int8).
• 4. Store the INT8 tensor and the scale value.
• 5. During inference, dequantize on‑the‑fly: w_q = q.astype(float32) * scale.
• 6. Use the dequantized weights in the model’s forward pass.
• 7. Evaluate model accuracy and adjust quantization strategy if needed.

Evidence

The size of the overall weights is simple arithmetic: number of weights × bytes per weight.

Source

Quantization, as a concept, is simply trying to store those weights in a smaller type, like INT8.

Source

The largest weight is 2.54, so: scale = 2.54 / 127 = 0.02

Source

You store the INT8 values plus one scale. That's how 4 bytes per weight becomes 1.

Source