In Bayesian statistics, observing evidence $X$ updates prior beliefs $p(Z)$ into a posterior distribution $p(Z mid X) = frac{p(X, Z)}{p(X)}$. However, calculating the marginal likelihood evidence $p(X) = int p(X, Z) dZ$ requires high-dimensional integration across thousands of correlated continuous variables—an analytically intractable barrier. While Markov Chain Monte Carlo (MCMC) samples the posterior asymptotically, it is computationally slow. Variational Inference (VI) revolutionized Bayesian computing by casting posterior integration as a deterministic optimization problem.
1. The Variational Inference Objective
Let $p(Z mid X)$ be an intractable target posterior. Instead of simulating random Markov walks, Variational Inference specifies a family of tractable parametric distributions $mathcal{Q} = {q(Z)}$ and searches for the specific member $q^*(Z)$ that minimizes the Kullback-Leibler (KL) Divergence to the true posterior:
$$q^*(Z) = argmin_{q in mathcal{Q}} D_{text{KL}}left( q(Z) parallel p(Z mid X) right)$$
Evaluating the definition of forward KL divergence:
$$D_{text{KL}}(q parallel p) = int q(Z) ln left( frac{q(Z)}{p(Z mid X)} right) dZ = int q(Z) ln q(Z) dZ – int q(Z) ln p(Z mid X) dZ$$
Substituting Bayes’ theorem $p(Z mid X) = frac{p(X, Z)}{p(X)}$:
$$D_{text{KL}}(q parallel p) = mathbb{E}_q[ln q(Z)] – mathbb{E}_q[ln p(X, Z)] + ln p(X) int q(Z) dZ$$
Because $int q(Z) dZ = 1$, rearranging terms yields the foundational identity of variational Bayesian analysis:
$$ln p(X) = underbrace{mathbb{E}_q[ln p(X, Z)] – mathbb{E}_q[ln q(Z)]}_{text{ELBO}(q)} + D_{text{KL}}(q(Z) parallel p(Z mid X))$$
2. Minimizing KL by Maximizing the ELBO
Notice the monumental beauty of this decomposition:
- The marginal log-likelihood $ln p(X)$ is a constant with respect to variational distribution $q$.
- Because the KL divergence is strictly non-negative ($D_{text{KL}} ge 0$), the Evidence Lower Bound is always a lower bound: $text{ELBO}(q) le ln p(X)$.
- Therefore, maximizing $text{ELBO}(q)$ directly minimizes the KL divergence $D_{text{KL}}(q parallel p)$!
Crucially, computing $text{ELBO}(q) = mathbb{E}_q[ln p(X, Z)] + mathcal{H}(q)$ requires only the joint probability $p(X, Z)$, completely bypassing the intractable evidence denominator $p(X)$.
3. The Mean-Field Approximation
To make the optimization space $mathcal{Q}$ tractable, we introduce the Mean-Field Variational Family, assuming the latent variables completely factorize into independent marginal components:
$$q(Z) = prod_{j=1}^M q_j(Z_j)$$
While this ignores true correlations between latent variables, it empowers closed-form analytical coordinate ascent.
4. Coordinate Ascent Variational Inference (CAVI)
To maximize the ELBO over factor $q_j(Z_j)$ holding all other factors $q_{-j}$ fixed, we isolate $q_j$ inside the expectation:
$$text{ELBO}(q_j) = int q_j(Z_j) mathbb{E}_{-j}left[ ln p(X, Z) right] dZ_j – int q_j(Z_j) ln q_j(Z_j) dZ_j + text{const}$$
This expression is the negative KL divergence between $q_j(Z_j)$ and $expleft( mathbb{E}_{-j}[ln p(X, Z)] right)$. Thus, the optimal variational update for factor $j$ possesses the closed-form:
$$q_j^*(Z_j) propto expleft( mathbb{E}_{q_{-j}}left[ ln p(X, Z_1, dots, Z_M) right] right)$$
Coordinate Ascent Variational Inference (CAVI) iteratively cycles through updating each factor $q_j^*$ until the ELBO converges. When paired with conditionally conjugate exponential families, CAVI updates mirror classical Gibbs sampling updates, but operate deterministically through expectations.
5. The Variational Legacy
By replacing stochastic random-walk sampling with continuous gradient-based optimization, Variational Inference scaled Bayesian reasoning to massive web datasets. Decades later, Kingma & Welling (2013) generalized variational inference with neural networks to invent the Variational Autoencoder (VAE), proving that the mean-field ELBO remains the mathematical bridge connecting Bayesian probability to generative deep learning.
