Most deep generative models—such as VAEs and GANs—suffer from fundamental theoretical compromises: VAEs optimize only a surrogate lower bound (the ELBO) rather than exact data likelihood, while GANs provide no density evaluation whatsoever and battle severe minimax instability. Normalizing Flows resolve both limitations by constructing invertible, bijective neural networks. By leveraging the multivariable Change-of-Variables Theorem, normalizing flows transform simple isotropic Gaussian base distributions into arbitrarily intricate multimodal densities while computing exact marginal likelihoods in closed form.
1. The Multivariable Change-of-Variables Theorem
Let $Z in mathbb{R}^d$ be a latent continuous random vector governed by a known, tractable prior probability density $p_Z(z)$ (such as a standard isotropic Gaussian $mathcal{N}(0, I_d)$). Let $f: mathbb{R}^d to mathbb{R}^d$ be a smooth, bijective (invertible) diffeomorphism with inverse $g = f^{-1}$ such that $X = f(Z)$ and $Z = g(X)$.
By the conservation of probability mass across differential volumes $p_X(x) |dx| = p_Z(z) |dz|$, the probability density function of target variable $X$ is given by:
$$p_X(x) = p_Z(g(x)) cdot left| det left( frac{partial g(x)}{partial x} right) right| = p_Z(f^{-1}(x)) cdot left| det left( frac{partial f(z)}{partial z} right) right|^{-1}$$
where $J_f(z) = frac{partial f(z)}{partial z} in mathbb{R}^{d times d}$ is the Jacobian matrix of partial derivatives $J_{ij} = frac{partial f_i}{partial z_j}$. Taking logarithms yields the exact log-likelihood:
$$log p_X(x) = log p_Z(f^{-1}(x)) – log left| det J_f(f^{-1}(x)) right|$$
2. Compositional Depth and Jacobian Algebra
A single simple transformation cannot capture the geometry of complex empirical data. However, the composition of $K$ invertible diffeomorphic functions $f = f_K circ f_{K-1} circ dots circ f_1$ remains strictly invertible ($f^{-1} = f_1^{-1} circ dots circ f_K^{-1}$).
By the multivariable chain rule, the Jacobian of a composite mapping is the matrix product of intermediate Jacobians:
$$J_f(z_0) = J_{f_K}(z_{K-1}) cdot J_{f_{K-1}}(z_{K-2}) cdots J_{f_1}(z_0)$$
By the multiplicative property of determinants ($det(AB) = det(A)det(B)$), the log-determinant decomposes into a simple sum of scalar terms:
$$log left| det J_f(z_0) right| = sum_{k=1}^K log left| det J_{f_k}(z_{k-1}) right|$$
3. The Computational Bottleneck: Triangular Coupling Layers
Evaluating the determinant of an arbitrary dense $d times d$ matrix scales cubically: $O(d^3)$. In high-dimensional vision where $d = 10^6$, cubic scaling is impossible. The core architectural challenge of normalizing flows is designing layers whose Jacobian is triangular, collapsing determinant evaluation to the product of diagonal elements in linear time $O(d)$:
$$det(J) = prod_{i=1}^d J_{ii} implies log |det(J)| = sum_{i=1}^d log |J_{ii}|$$
In 2016, Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio introduced RealNVP (Real-valued Non-Volume Preserving) flows via Affine Coupling Layers:
- Partition the input coordinates into two disjoint halves: $x = [x_{1:d/2}, x_{d/2+1:d}]$.
- Leave the first half unaltered: $y_{1:d/2} = x_{1:d/2}$.
- Transform the second half using scale $s(cdot)$ and translation $t(cdot)$ networks conditioned on the first half:
$$y_{d/2+1:d} = x_{d/2+1:d} odot exp(s(x_{1:d/2})) + t(x_{1:d/2})$$
The Jacobian of this coupling layer is lower triangular with diagonal entries $[1, dots, 1, exp(s_1), dots, exp(s_{d/2})]$. Its log-determinant is simply $sum_j s(x_{1:d/2})_j$—requiring zero matrix inversions! Remarkably, the scale and translation networks $s$ and $t$ can be arbitrarily complex, non-invertible deep neural networks (e.g., ResNets).
4. Continuous Normalizing Flows (CNFs) & Neural ODEs
Taking the continuous-time limit where depth approaches infinity ($K to infty$) transitions normalizing flows into Continuous Normalizing Flows (CNFs) governed by ordinary differential equations $frac{dz(t)}{dt} = f(z(t), t)$. By Liouville’s continuity theorem, the instantaneous change in log density is governed by the matrix trace rather than determinant:
$$frac{partial log p(z(t))}{partial t} = -text{Tr}left( frac{partial f(z(t), t)}{partial z(t)} right)$$
Applying the Hutchinson stochastic trace estimator enables unbiased $O(d)$ likelihood estimation across continuous neural vector fields.
