Executing continuous diffusion directly in high-resolution pixel space ($1024 times 1024 times 3 approx 3 times 10^6$ dimensions) is computationally prohibitive: the network wastes over 90% of its capacity modeling imperceptible high-frequency pixel noise. Latent Diffusion Models (LDM / Stable Diffusion) and Flow Matching solve this by partitioning generation into perceptual compression and semantic synthesis, propelling generative AI into consumer adoption and millisecond-level sampling.
1. Two-Stage Perceptual Space Decomposition
Rombach et al. (2022) established that visual generative modeling involves two distinct regimes:
- Perceptual Compression: An autoencoder (VAE) trained with adversarial and perceptual losses compresses an RGB image $mathbf{x} in mathbb{R}^{H times W times 3}$ into a low-dimensional continuous latent space $mathbf{z} = mathcal{E}(mathbf{x}) in mathbb{R}^{h times w times c}$, typically with downsampling factor $f = H/h = 8$. Spatial resolution is reduced 64-fold, filtering out imperceptible noise while preserving semantic geometry.
- Semantic Synthesis: The diffusion backbone (U-Net or Diffusion Transformer / DiT) operates exclusively within the compact latent space $mathbf{z}$, slashing FLOPs by orders of magnitude. The final latent is decoded into full resolution via $hat{mathbf{x}} = mathcal{D}(mathbf{z})$.
2. Conditioning & Classifier-Free Guidance (CFG)
To steer image generation with textual prompts $y$, text is embedded via a pre-trained frozen encoder (CLIP ViT-L or T5-XXL) into embeddings $mathbf{c} = tau_theta(y)$ and injected into diffusion layers via cross-attention:
text{Attention}(mathbf{Q}, mathbf{K}, mathbf{V}) = text{Softmax}left(frac{mathbf{Q} mathbf{K}^T}{sqrt{d}}right) mathbf{V}, quad mathbf{Q} = mathbf{W}_Q mathbf{z}_t, quad mathbf{K} = mathbf{W}_K mathbf{c}, quad mathbf{V} = mathbf{W}_V mathbf{c}
To guarantee strict adherence to the prompt, Jonathan Ho and Tim Salimans (2021) formulated Classifier-Free Guidance (CFG). During inference, the predicted noise is computed as an extrapolation between the conditioned prediction and an unconditioned prediction (passing empty prompt $emptyset$):
tilde{boldsymbol{epsilon}}_theta(mathbf{z}_t, mathbf{c}) = boldsymbol{epsilon}_theta(mathbf{z}_t, emptyset) + s cdot left( boldsymbol{epsilon}_theta(mathbf{z}_t, mathbf{c}) – boldsymbol{epsilon}_theta(mathbf{z}_t, emptyset) right)
Guidance scale $s > 1$ (typically $s approx 7.5$) amplifies text alignment and color saturation while pruning improbable visual artifacts.
3. The Flow Matching Paradigm: Rectified Straight Trajectories
Standard diffusion trajectories are curved Brownian paths, requiring 30 to 50 solver steps to integrate. Flow Matching (Lipman et al., 2022; Albergo & Vanden-Eijnden, 2022) reformulates generative modeling as learning a vector field $v_t(mathbf{x})$ that generates a Continuous Normalizing Flow:
frac{mathrm{d}mathbf{x}_t}{mathrm{d}t} = v_t(mathbf{x}_t)
Under Rectified Flow, the probability paths between pure noise $mathbf{x}_0$ and real data $mathbf{x}_1$ are defined as straight lines:
mathbf{x}_t = (1 – t) mathbf{x}_0 + t mathbf{x}_1, quad frac{mathrm{d}mathbf{x}_t}{mathrm{d}t} = mathbf{x}_1 – mathbf{x}_0
The neural network simply minimizes the regression objective: $mathcal{L}_{text{FM}}(theta) = mathbb{E} [ | v_theta(mathbf{x}_t, t) – (mathbf{x}_1 – mathbf{x}_0) |^2 ]$. Because the trajectories are straight lines, the ODE can be integrated in as few as 1 to 4 steps, powering modern frontier engines like Stable Diffusion 3 and Flux.
