In 2014, Ian Goodfellow et al. introduced an entirely new generative philosophy that dispensed with explicit likelihood estimation: Generative Adversarial Networks (GANs). Framed as a zero-sum game between two dueling neural networks—a Generator synthesizing counterfeit artifacts and a Discriminator acting as an art critic—GANs unlocked sharp, photorealistic sample generation. Later, the theoretical integration of optimal transport geometry and Wasserstein distance transformed adversarial training from erratic instability into provably monotonic convergence.
1. The Two-Player Zero-Sum Minimax Game
Let $p_{text{data}}$ be the true empirical data distribution over $mathbb{R}^d$, and let $p_z$ be a simple noise prior (e.g., uniform or standard Gaussian) over latent space $mathbb{R}^k$.
- Generator ($G_theta: mathcal{Z} to mathcal{X}$): Maps noise vectors $z sim p_z$ to generated samples $G(z)$, inducing synthetic distribution $p_g$.
- Discriminator ($D_phi: mathcal{X} to [0, 1]$): Predicts the probability that an input $x$ originates from real data rather than the generator.
The objective function is the minimax value function $V(D, G)$:
$$min_G max_D V(D, G) = mathbb{E}_{x sim p_{text{data}}}[log D(x)] + mathbb{E}_{z sim p_z}[log(1 – D(G(z)))]$$
2. Optimal Discriminator and the Jensen-Shannon Divergence
For any fixed generator $G$, the optimal discriminator $D_G^*(x)$ can be derived analytically by differentiating the objective with respect to $D(x)$:
$$D_G^*(x) = frac{p_{text{data}}(x)}{p_{text{data}}(x) + p_g(x)}$$
Substituting $D_G^*$ back into the minimax objective yields the effective loss minimized by the generator:
$$V(D_G^*, G) = -log(4) + 2 cdot D_{text{JS}}(p_{text{data}} parallel p_g)$$
where $D_{text{JS}}$ is the symmetric Jensen-Shannon Divergence. The global minimum occurs if and only if $p_g = p_{text{data}}$, where $D_G^*(x) = 1/2$ and $V(D_G^*, G) = -log 4$.
3. The Pathology of Disjoint Supports: Mode Collapse & Vanishing Gradients
In high dimensions, real data concentrates on lower-dimensional sub-manifolds $mathcal{M}_{text{data}}$ and $mathcal{M}_g$ of dimension $d^* ll d$. By differential geometry, two low-dimensional manifolds in high-dimensional space intersect on a set of measure zero.
When supports are disjoint ($text{supp}(p_{text{data}}) cap text{supp}(p_g) = emptyset$):
$$D_{text{JS}}(p_{text{data}} parallel p_g) = log 2 = text{constant}$$
Consequently, the gradient of the Jensen-Shannon divergence vanishes completely: $nabla_theta D_{text{JS}} = 0$. The discriminator achieves $100%$ accuracy effortlessly, providing zero informative gradient signal to the generator. Conversely, if the generator exploits a single localized mode to deceive the discriminator, Mode Collapse occurs, collapsing sample diversity.
4. Wasserstein GAN (WGAN): Optimal Transport & Kantorovich Duality
In 2017, Martin Arjovsky, Soumith Chintala, and Léon Bottou cured adversarial instability by replacing JS divergence with the Earth-Mover (Wasserstein-1) Distance from optimal transport theory:
$$W_1(p_{text{data}}, p_g) = inf_{gamma in Pi(p_{text{data}}, p_g)} mathbb{E}_{(x, y) sim gamma}[|x – y|]$$
Unlike JS divergence, $W_1$ provides a smooth, continuous, and non-zero distance metric even when supports are completely disjoint.
By the Kantorovich-Rubinstein Duality Theorem, the intractable infimum over joint measures transforms into a supremum over 1-Lipschitz functions $f in mathcal{F}_L$:
$$W_1(p_{text{data}}, p_g) = sup_{|f|_L le 1} mathbb{E}_{x sim p_{text{data}}}[f(x)] – mathbb{E}_{x sim p_g}[f(x)]$$
Here, the discriminator becomes a continuous Critic $f_w(x)$. Gulrajani et al. (2017) enforced the 1-Lipschitz constraint via the Gradient Penalty (WGAN-GP):
$$mathcal{L}_{text{GP}} = mathbb{E}_{hat{x}}left[ (|nabla_{hat{x}} f_w(hat{x})|_2 – 1)^2 right]$$
penalizing critics whose directional gradients deviate from unit slope along interpolations between real and generated points, ensuring stable monotonic generative convergence.
