While autoregressive Transformers dominate discrete symbolic sequences, continuous high-dimensional generative synthesis—spanning photorealistic visual imagery, video generation, and audio waveform synthesis—is ruled by Diffusion Models. Rooted in non-equilibrium statistical thermodynamics, diffusion conceptualizes generative synthesis not as classification, but as an inversion of physical entropy: systematically adding Gaussian noise to an image until information dissolves into pure chaos, and training a neural network to reverse the arrow of time to reconstruct complex structured manifolds.
1. The Forward Noising Process: Ornstein-Uhlenbeck Mechanics
In Denoising Diffusion Probabilistic Models (DDPM, Ho et al., 2020), given a clean data sample $mathbf{x}_0 sim q(mathbf{x}_0)$, a forward Markov chain progressively injects Gaussian noise across $T$ discrete timesteps according to a predefined variance schedule $beta_1, dots, beta_T$:
q(mathbf{x}_t mid mathbf{x}_{t-1}) = mathcal{N}left(mathbf{x}_t; sqrt{1 – beta_t} mathbf{x}_{t-1}, beta_t mathbf{I}right)
Crucially, using the reparameterization trick with $alpha_t = 1 – beta_t$ and cumulative product $bar{alpha}_t = prod_{s=1}^t alpha_s$, any arbitrary noisy state $mathbf{x}_t$ can be sampled in closed form directly from $mathbf{x}_0$ without computing intermediate steps:
mathbf{x}_t = sqrt{bar{alpha}_t} mathbf{x}_0 + sqrt{1 – bar{alpha}_t} boldsymbol{epsilon}, quad boldsymbol{epsilon} sim mathcal{N}(mathbf{0}, mathbf{I})
As $t to T$, $bar{alpha}_t to 0$, causing $q(mathbf{x}_T) approx mathcal{N}(mathbf{0}, mathbf{I})$, transforming any structured data distribution into isotropic Gaussian noise.
2. The Reverse Generative Trajectory & Score Matching
Generative synthesis requires inverting this process: sampling $mathbf{x}_T sim mathcal{N}(mathbf{0}, mathbf{I})$ and traversing backward along the reverse conditional distribution $p_theta(mathbf{x}_{t-1} mid mathbf{x}_t)$. While the exact true reverse distribution $q(mathbf{x}_{t-1} mid mathbf{x}_t)$ is mathematically intractable, for infinitesimal steps it is provably Gaussian:
p_theta(mathbf{x}_{t-1} mid mathbf{x}_t) = mathcal{N}left(mathbf{x}_{t-1}; boldsymbol{mu}_theta(mathbf{x}_t, t), boldsymbol{Sigma}_theta(mathbf{x}_t, t)right)
Yang Song and Stefano Ermon (2019) demonstrated that reversing this process is equivalent to learning the Stein Score Function: the gradient of the log data density with respect to the input space, $nabla_{mathbf{x}} log q_t(mathbf{x})$.
3. Continuous-Time Formulation via Stochastic Differential Equations (SDEs)
Song et al. (2020) generalized discrete diffusion into a continuous time continuum $t in [0, 1]$ governed by an Itô Stochastic Differential Equation:
mathrm{d}mathbf{x} = mathbf{f}(mathbf{x}, t) mathrm{d}t + g(t) mathrm{d}mathbf{w}
Where $mathbf{f}(mathbf{x}, t)$ is the drift coefficient, $g(t)$ is the diffusion coefficient, and $mathbf{w}$ is standard Brownian motion. By Anderson’s reverse-time theorem (1982), this forward SDE has a unique, deterministic reverse-time SDE:
mathrm{d}mathbf{x} = left[ mathbf{f}(mathbf{x}, t) – g(t)^2 nabla_{mathbf{x}} log p_t(mathbf{x}) right] mathrm{d}t + g(t) mathrm{d}bar{mathbf{w}}
A neural network $boldsymbol{epsilon}_theta(mathbf{x}_t, t)$ is trained via Denoising Score Matching to predict the added noise vector:
mathcal{L}_{text{simple}}(theta) = mathbb{E}_{t, mathbf{x}_0, boldsymbol{epsilon}} left[ | boldsymbol{epsilon} – boldsymbol{epsilon}_theta(mathbf{x}_t, t) |_2^2 right]
Once trained, the reverse SDE can be integrated using standard numerical ODE solvers (Euler-Maruyama, Runge-Kutta, DDIM), generating breathtaking continuous manifolds from pure stochastic noise.
