The final forward pass of a Large Language Model does not output words; it produces a vector of unnormalized logits $mathbf{z} in mathbb{R}^{|mathcal{V}|}$ across its vocabulary. The algorithms that convert these continuous logits into discrete, flowing sentences—Decoding Strategies—dictate whether the model produces repetitive, rigid text, creative prose, or incoherent gibberish. Furthermore, modern innovations like Speculative Decoding fundamentally accelerate the serial generation bottleneck without altering the target distribution.
1. Thermodynamic Logit Temperature Scaling
Logits are converted into probability distributions via Softmax. Introducing temperature parameter $T > 0$ alters the entropy of the Boltzmann distribution:
P(w_i) = frac{exp(z_i / T)}{sum_{j=1}^{|mathcal{V}|} exp(z_j / T)}
- $T to 0$ (Greedy Search): The distribution collapses into a Dirac delta on the maximum logit ($argmax z_i$). This minimizes hallucinations in mathematics and coding, but often causes severe cyclic repetitions in creative dialogue.
- $T = 1.0$: Exact sampling from the true model distribution.
- $T > 1.0$: Flattens the distribution toward a uniform distribution, increasing lexical diversity and creativity at the risk of syntactic breakdown.
2. Truncation Strategies: Top-$k$ vs Top-$p$ (Nucleus) Sampling
Pure temperature sampling suffers from the heavy tail of the vocabulary: rare words with tiny probabilities can occasionally be sampled, destroying coherence.
- Top-$k$ Sampling (Fan et al., 2018): Hard-truncates the distribution to the $k$ most probable tokens, setting all other logits to $-infty$. The limitation: a fixed $k=50$ is too restrictive when the distribution is flat, yet too permissive when a single token has 99% probability.
- Top-$p$ / Nucleus Sampling (Holtzman et al., 2019): Dynamically truncates the vocabulary to the smallest subset $V^{(p)} subset mathcal{V}$ whose cumulative probability exceeds threshold $p in (0, 1]$:
sum_{w in V^{(p)}} P(w) ge p
The cutoff expands dynamically when uncertainty is high and contracts tightly when the next token is obvious.
3. Speculative Decoding: Breaking the Sequential Barrier
Autoregressive decoding is inherently serial: generating $K$ tokens requires $K$ sequential forward passes through a massive model (e.g., 70B parameters), which is severely memory-bandwidth bound. Speculative Decoding (Leviathan et al., 2023; Chen et al., 2023) breaks this sequential dependency.
A lightweight draft model (e.g., 1B parameters) speculatively generates $K$ candidate tokens rapidly: $(hat{x}_1, dots, hat{x}_K)$. The large target model then evaluates all $K$ tokens simultaneously in a single parallel forward pass. Each candidate token is accepted or rejected based on a rejection sampling criterion that mathematically preserves the exact distribution of the target model:
P(text{accept } hat{x}_i) = minleft(1, frac{P_{text{target}}(hat{x}_i mid dots)}{P_{text{draft}}(hat{x}_i mid dots)}right)
If token $j$ is rejected, the target model resamples the token and discards the remaining speculative tokens. Because large models are compute-underutilized during single-token decoding, verifying $K$ tokens takes virtually the same time as generating one, accelerating overall inference latency by 2x to 3x with zero loss in mathematical output fidelity.
