In unsupervised representation learning, the fundamental objective is to discover compact, generative coordinates that capture the underlying causal factors of high-dimensional data without external labels. Autoencoders realize this by forcing input vectors $x in mathbb{R}^d$ through a low-dimensional bottleneck $z in mathbb{R}^k$ ($k ll d$) before reconstructing $hat{x}$. By restricting capacity, the network is compelled to discover the intrinsic low-dimensional non-linear sub-manifold on which real-world data concentrates.
1. The Undercomplete Bottleneck Objective
An autoencoder consists of two parameterized mappings:
- Encoder ($f_phi: mathcal{X} to mathcal{Z}$): Compresses input $x$ into latent code $z = f_phi(x) = sigma(W_e x + b_e)$.
- Decoder ($g_psi: mathcal{Z} to mathcal{X}$): Reconstructs the original input $hat{x} = g_psi(z) = sigma(W_d z + b_d)$.
Parameters are optimized by minimizing empirical reconstruction distortion under a Bregman divergence or mean squared error:
$$min_{phi, psi} mathcal{L}_{text{AE}}(phi, psi) = frac{1}{n} sum_{i=1}^n | x_i – g_psi(f_phi(x_i)) |^2$$
If activation functions $sigma$ are purely linear, Bourlard and Kamp (1988) proved that the $k$-dimensional latent space spans exactly the same subspace as Principal Component Analysis (PCA). Non-linear activations empower autoencoders to learn non-linear coordinate charts for curved Riemannian manifolds.
2. Denoising Autoencoders (DAE): Score Matching on Manifolds
An unconstrained autoencoder with high latent capacity ($k ge d$) risks learning a trivial identity mapping $g(f(x)) = x$. In 2008, Pascal Vincent et al. introduced the Denoising Autoencoder (DAE): corrupting input $x$ with stochastic noise $tilde{x} sim q(tilde{x} mid x)$ (e.g., Gaussian noise or masking) and training the network to reconstruct the original clean $x$:
$$mathcal{L}_{text{DAE}} = mathbb{E}_{x sim p_{text{data}}, tilde{x} sim q(tilde{x}|x)}left[ | x – g_psi(f_phi(tilde{x})) |^2 right]$$
In 2011, Guillaume Alain and Yoshua Bengio established a profound theoretical equivalence: for small Gaussian noise $tilde{x} sim mathcal{N}(x, sigma^2 I)$, the vector pointing from corrupted input to reconstructed output estimates the score function (the gradient of the data log-density):
$$g(f(tilde{x})) – tilde{x} approx sigma^2 nabla_{tilde{x}} log p(tilde{x})$$
A DAE learns a vector field that points toward the nearest point on the data manifold, directly prefiguring modern continuous diffusion models.
3. Contractive Autoencoders (CAE): Robust Representation Flatness
Rifai et al. (2011) formulated the Contractive Autoencoder (CAE), adding an explicit penalty on the Frobenius norm of the encoder Jacobian:
$$mathcal{L}_{text{CAE}} = | x – g(f(x)) |^2 + lambda | J_f(x) |_F^2, quad | J_f(x) |_F^2 = sum_{i,j} left( frac{partial h_i}{partial x_j} right)^2$$
The reconstruction loss pulls the model to preserve information along the tangent bundle of the manifold, while the contractive Frobenius penalty crushes sensitivity in all orthogonal directions. Consequently, the representation remains constant when perturbed off the data manifold.
4. Representation Disentanglement
A representation $z = (z_1, dots, z_k)$ is said to be disentangled if independent latent dimensions correspond to isolated generative factors of variation in the world (e.g., azimuth, lighting, color, scale). While deterministic autoencoders achieve compression, they do not enforce coordinate independence, motivating the probabilistic framework of Variational Autoencoders.
