Serving a 70-billion-parameter foundation model in FP16 precision demands 140 GB of VRAM solely to store weights, requiring multiple high-end enterprise GPUs. Model Quantization compresses continuous 16-bit floating-point weights into discrete 8-bit, 4-bit, or even 2-bit representations. This reduction is not merely a matter of truncation; it is a rigorous optimization problem of projecting multi-dimensional weight tensors onto discrete lattices while minimizing output perturbation on downstream representations.
1. Fundamental Quantization Mechanics: Affine vs Symmetric
Uniform quantization maps a continuous floating-point range $[x_{min}, x_{max}]$ onto a signed integer grid $[-2^{b-1}, 2^{b-1}-1]$ of bit-width $b$:
$q = text{clip}left( leftlfloor frac{x}{s} rightrceil + z, -2^{b-1}, 2^{b-1}-1 right)$
Where $s$ is the scale factor and $z$ is the integer zero-point offset:
$s = frac{x_{max} – x_{min}}{2^b – 1}, quad z = leftlfloor -frac{x_{min}}{s} rightrceil – 2^{b-1}$
In Symmetric Quantization, zero-point $z = 0$, simplifying hardware operations by mapping symmetric ranges $[-x_{max}, x_{max}]$ directly via $s = x_{max} / (2^{b-1}-1)$. Dequantization reconstructs the floating-point approximation: $hat{x} = s(q – z)$.
2. The Emergence of Outlier Dimensions at Scale
When language models surpass ~6.7 billion parameters, an abrupt phase transition occurs in internal representations: Emergent Outlier Dimensions (Dettmers et al., 2022). Across specific hidden dimension channels, activation magnitudes spike up to 100x higher than typical values ($|X_{i,j}| > 100$).
If standard per-tensor quantization is applied, these massive outliers force scale factor $s$ to expand dramatically, collapsing normal activations into identical integer bins and devastating model perplexity. Methods like LLM.int8() resolve this via vector-wise mixed-precision decomposition: outlier channels are isolated and multiplied in FP16, while the remaining 99.9% of channels are computed in INT8.
3. Second-Order Optimization: GPTQ
Rather than quantizing weights independently (Round-to-Nearest / RTN), GPTQ (Frantar et al., 2022) formulates quantization as a layer-wise squared error minimization problem:
$argmin_{widehat{mathbf{W}}} |mathbf{W}mathbf{X} – widehat{mathbf{W}}mathbf{X}|_2^2$
By taking a second-order Taylor expansion, the optimal compensation for quantizing weight $w_q$ into $hat{w}_q$ is dictated by the inverse Hessian matrix $mathbf{H}^{-1} = (2 mathbf{X}mathbf{X}^T + lambda mathbf{I})^{-1}$:
$Delta mathbf{w} = – frac{w_q – hat{w}_q}{[mathbf{H}^{-1}]_{qq}} mathbf{H}^{-1}_{:, q}$
As each weight column is quantized to 4-bit, the remaining unquantized weights are updated immediately to absorb the quantization error, preserving mathematical coherence across 70B+ models.
4. Activation-Aware Weight Quantization (AWQ)
Lin et al. (2023) discovered that weights are not equally important: protecting only the top 1% of weights corresponding to high-magnitude activations prevents almost all quantization loss. AWQ applies per-channel scaling transformations $mathbf{S}$ prior to quantization:
$mathbf{W}’ = mathbf{W} mathbf{S}^{-1}, quad mathbf{X}’ = mathbf{S} mathbf{X}$
Because $mathbf{W}’ mathbf{X}’ = mathbf{W} mathbf{X}$, the linear layer output is mathematically identical, but the weight dynamic range is equalized, enabling 4-bit weight execution directly on consumer hardware at near-lossless perplexity.
