Standard deterministic autoencoders suffer from a fatal generative flaw: their latent space is plagued by discontinuous, unregularized gaps. Sampling an arbitrary latent point $z sim mathcal{N}(0, I)$ and passing it through the decoder yields meaningless artifacts. In 2013, Diederik Kingma and Max Welling, alongside Danilo Rezende et al., introduced the Variational Autoencoder (VAE), grounding deep latent representation in principled Bayesian variational inference and solving the stochastic backpropagation barrier via the Reparameterization Trick.
1. The Intractable Posterior Problem in Latent Variable Models
Consider a directed probabilistic graphical model where unobserved latent variables $z sim p(z) = mathcal{N}(0, I)$ generate observed data $x$ according to conditional distribution $p_theta(x mid z)$. We seek to maximize the marginal log-likelihood (the “evidence”):
$$log p_theta(x) = log int p_theta(x mid z) p(z) dz$$
This integral is analytically intractable for non-linear neural networks: evaluating it requires integrating across the entire infinite-dimensional latent manifold. Furthermore, the true posterior $p_theta(z mid x) = frac{p_theta(x mid z) p(z)}{p_theta(x)}$ cannot be computed.
2. Derivation of the Evidence Lower Bound (ELBO)
Variational inference introduces a tractable parametric recognition model $q_phi(z mid x) approx p_theta(z mid x)$, parameterized by a neural network encoder. Evaluating the Kullback-Leibler divergence between the approximate and true posterior:
$$D_{text{KL}}(q_phi(z mid x) parallel p_theta(z mid x)) = int q_phi(z mid x) log left( frac{q_phi(z mid x)}{p_theta(z mid x)} right) dz$$
Applying Bayes’ rule $log p_theta(z mid x) = log p_theta(x, z) – log p_theta(x)$ and rearranging terms yields:
$$log p_theta(x) = D_{text{KL}}(q_phi(z mid x) parallel p_theta(z mid x)) + mathcal{L}_{text{ELBO}}(theta, phi; x)$$
where the Evidence Lower Bound (ELBO) is defined as:
$$mathcal{L}_{text{ELBO}}(theta, phi; x) = mathbb{E}_{q_phi(z mid x)}[log p_theta(x mid z)] – D_{text{KL}}(q_phi(z mid x) parallel p(z))$$
Since the KL divergence is non-negative ($D_{text{KL}} ge 0$), $mathcal{L}_{text{ELBO}} le log p_theta(x)$. Maximizing the ELBO simultaneously pushes the marginal data likelihood up while driving the variational posterior $q_phi(z mid x)$ toward the true posterior.
3. Anatomy of the ELBO Objectives
The ELBO balances two competing thermodynamic forces:
- Reconstruction Accuracy ($mathbb{E}_{q_phi}[log p_theta(x mid z)]$): Encourages the decoder to reconstruct the specific input $x$ faithfully from sampled latent coordinates.
- Latent Regularization ($-D_{text{KL}}(q_phi(z mid x) parallel p(z))$): Penalizes deviation of the variational posterior from the standard Gaussian prior $mathcal{N}(0, I)$, enforcing continuous, smooth, and complete coverage without isolated islands.
Assuming a diagonal Gaussian recognition model $q_phi(z mid x) = mathcal{N}(mu(x), text{diag}(sigma^2(x)))$, the KL term evaluates analytically in closed form:
$$D_{text{KL}}(q_phi(z mid x) parallel mathcal{N}(0, I)) = -frac{1}{2} sum_{j=1}^d left( 1 + log(sigma_j^2) – mu_j^2 – sigma_j^2 right)$$
4. The Reparameterization Trick: Differentiable Stochasticity
To optimize encoder parameters $phi$ via gradient descent, we must compute $nabla_phi mathbb{E}_{q_phi(z mid x)}[f(z)]$. However, backpropagation cannot pass through stochastic random sampling nodes because the expectation distribution itself depends on $phi$.
Kingma and Welling solved this through the Reparameterization Trick: isolating the stochastic randomness by expressing random variable $z$ as a deterministic, differentiable transformation of an auxiliary noise source $epsilon sim mathcal{N}(0, I)$:
$$z = g_phi(epsilon, x) = mu(x) + sigma(x) odot epsilon, quad epsilon sim mathcal{N}(0, I)$$
Now, the expectation is taken over an independent distribution $p(epsilon)$ that does not depend on $phi$:
$$nabla_phi mathbb{E}_{q_phi(z mid x)}[f(z)] = mathbb{E}_{epsilon sim mathcal{N}(0, I)}left[ nabla_z f(z) cdot nabla_phi g_phi(epsilon, x) right]$$
Monte Carlo gradients flow seamlessly into $mu(x)$ and $sigma(x)$, establishing the foundational cornerstone of deep probabilistic generative modeling.
