In the original Transformer, Vaswani et al. encoded sequence order by additively injecting absolute sinusoidal positional vectors directly into token embeddings: $x_i + p_i$. While simple, additive positional encoding severely impairs the model’s ability to generalize beyond its training context length. In 2021, Jianlin Su et al. introduced Rotary Position Embeddings (RoPE), establishing a mathematically pristine framework that encodes relative token distance through multiplicative orthogonal rotations in the complex plane.
1. The Theoretical Objective of Relative Positional Encoding
Let $q_m = f_q(x_m, m)$ and $k_n = f_k(x_n, n)$ be the query and key vectors representing tokens $x_m$ and $x_n$ at positions $m, n in mathbb{N}$. The central mathematical desideratum is that their inner product must depend solely on their semantic content and their relative displacement $m – n$:
$$langle f_q(x_m, m), f_k(x_n, n) rangle = g(x_m, x_n, m – n)$$
Additive positional encodings violate this condition because the cross-terms $langle x_m, p_n rangle$ and $langle p_m, p_n rangle$ corrupt relative geometric distances.
2. 2D Complex Rotation Derivation
Su considered the 2D case where vectors are identified as complex numbers $q_m, k_n in mathbb{C}$. The operational mapping satisfies:
$$f_q(x_m, m) = R_m q = q e^{i m theta}, quad f_k(x_n, n) = R_n k = k e^{i n theta}$$
Evaluating their Hermitian inner product:
$$langle f_q(x_m, m), f_k(x_n, n) rangle = text{Re}left[ (q e^{i m theta}) (k e^{i n theta})^* right] = text{Re}left[ q k^* e^{i (m – n)theta} right]$$
Astonishingly, the absolute positions $m$ and $n$ cancel out completely! The geometric interaction between query and key is governed strictly by the relative rotation angle $(m – n)theta$.
3. Generalization to $d$-Dimensional Space: Block-Diagonal Rotation
To extend to arbitrary even dimension $d$, RoPE partitions the vector space into $d/2$ independent 2-dimensional orthogonal subspaces. For each subspace $j in {1, dots, d/2}$, a distinct angular frequency is assigned:
$$theta_j = 10000^{-2(j-1)/d}$$
The global rotary operator is an orthogonal block-diagonal matrix $R_{Theta, m}^d$:
$$R_{Theta, m}^d = begin{bmatrix} cos mtheta_1 & -sin mtheta_1 & 0 & 0 & dots \ sin mtheta_1 & cos mtheta_1 & 0 & 0 & dots \ 0 & 0 & cos mtheta_2 & -sin mtheta_2 & dots \ 0 & 0 & sin mtheta_2 & cos mtheta_2 & dots \ vdots & vdots & vdots & vdots & ddots end{bmatrix}$$
In practice, matrix multiplication is avoided: RoPE is evaluated in $O(d)$ time via element-wise vector operations: $R_{Theta, m}^d x = x odot cos(mTheta) + tilde{x} odot sin(mTheta)$ where $tilde{x} = [-x_2, x_1, -x_4, x_3, dots]^T$.
4. Long-Context Extension: NTK-Aware Scaling and YaRN
When extending context windows from $4text{k}$ to $128text{k}$ tokens, rotating tokens by unobserved angles $m > L_{text{train}}$ causes high-frequency components to oscillate chaotically.
- Linear Position Interpolation (PI): Compresses positions by ratio $kappa = L_{text{new}} / L_{text{train}}$: $m’ = m / kappa$. While low frequencies adapt, high frequencies suffer resolution loss.
- NTK-Aware Scaling (bloc97, 2023): Draws on Neural Tangent Kernel theory, recognizing that high-frequency dimensions encode local syntax (which should not be compressed) while low-frequency dimensions encode long-range order. By scaling base frequency $b’ = b cdot kappa^{d/(d-2)}$, RoPE scales context seamlessly.
- YaRN (Yet another RoPE extensioN, Peng et al., 2023): Applies piecewise ramp interpolation across frequency bands alongside temperature scaling, enabling stable extrapolation to $128text{k}-1text{M}$ context windows without fine-tuning catastrophic forgetting.
