Classical parametric Bayesian models enforce rigid assumptions: parameters $theta in mathbb{R}^d$ possess finite dimensionality, bounding model capacity regardless of sample size $n$. Bayesian Non-Parametrics dissolves this constraint by allowing model capacity to grow organically with observations. Foremost among these non-parametric models is the Gaussian Process (GP)—a principled mathematical framework that places prior probability distributions directly over infinite-dimensional function spaces $mathcal{F} = {f: mathcal{X} to mathbb{R}}$.
1. Formal Definition and Consistency
Formally, a Gaussian Process is a collection of infinitely many random variables, any finite subset of which possesses a joint multivariate Gaussian distribution. A stochastic process $f(x)$ is specified completely by its mean function $m(x)$ and positive semi-definite covariance kernel function $k(x, x’)$:
$$f(x) sim mathcal{GP}left( m(x), k(x, x’) right)$$
where:
$$m(x) = mathbb{E}[f(x)], quad k(x, x’) = text{Cov}(f(x), f(x’)) = mathbb{E}left[ (f(x) – m(x))(f(x’) – m(x’)) right]$$
By the Kolmogorov Extension Theorem, the consistency of finite marginal distributions under linear marginalization guarantees that the infinite-dimensional stochastic process is mathematically well-defined.
2. Covariance Kernels and Sample Path Regularity
The geometric smoothness, differentiability, and periodicity of sample functions drawn from a Gaussian process are dictated by the covariance kernel $k(x, x’)$:
- Squared Exponential (RBF / Gaussian):
$$k_{text{SE}}(r) = sigma_f^2 expleft( -frac{r^2}{2ell^2} right), quad r = |x – x’|$$
Functions sampled under $k_{text{SE}}$ are infinitely mean-square differentiable ($C^infty$), producing exceptionally smooth curves.
- Matérn Class Kernels:
$$k_{text{Matérn}}(r) = sigma_f^2 frac{2^{1-nu}}{Gamma(nu)} left( sqrt{2nu}frac{r}{ell} right)^nu K_nuleft( sqrt{2nu}frac{r}{ell} right)$$
When $nu = 3/2$ or $5/2$, sample functions are only once or twice differentiable, providing realistic modeling of physical phenomena without artificial hyper-smoothness.
- Periodic Kernel:
$$k_{text{Per}}(r) = sigma_f^2 expleft( -frac{2sin^2(pi r / p)}{ell^2} right)$$
Models cyclical seasonal oscillations with exact period $p$.
3. Analytical Posterior Inference for GP Regression
Consider noisy observations $y = f(x) + epsilon$ where $epsilon sim mathcal{N}(0, sigma_n^2)$. Given training inputs $X = [x_1, dots, x_n]^T$ and test queries $X_*$, the joint distribution of observed targets $y$ and latent function values $f_* = f(X_*)$ is a joint Gaussian:
$$begin{bmatrix} y \ f_* end{bmatrix} sim mathcal{N}left( mathbf{0}, begin{bmatrix} K(X, X) + sigma_n^2 I & K(X, X_*) \ K(X_*, X) & K(X_*, X_*) end{bmatrix} right)$$
Applying the standard Gaussian conditioning theorem yields the exact analytical posterior distribution $f_* mid X, y, X_* sim mathcal{N}(mu_*, Sigma_*)$:
$$mu_* = K(X_*, X) left[ K(X, X) + sigma_n^2 I right]^{-1} y$$
$$Sigma_* = K(X_*, X_*) – K(X_*, X) left[ K(X, X) + sigma_n^2 I right]^{-1} K(X, X_*)$$
The posterior variance $Sigma_*$ quantifies epistemic uncertainty: it shrinks to $sigma_n^2$ in dense observation clusters and expands back to the prior variance in unexplored feature domains.
4. Marginal Likelihood and Bayesian Optimization
Hyperparameters $theta = {ell, sigma_f^2, sigma_n^2}$ are optimized through Empirical Bayes by maximizing the log marginal likelihood:
$$log p(y mid X, theta) = -frac{1}{2} y^T (K_y)^{-1} y – frac{1}{2} log |K_y| – frac{n}{2}log(2pi)$$
The first term measures data fit, while the second term $frac{1}{2}log |K_y|$ acts as an automatic Occam’s razor penalizing model complexity. Furthermore, Gaussian processes power Bayesian Optimization of expensive black-box objective functions via acquisition functions such as Expected Improvement (EI) and Upper Confidence Bound (GP-UCB):
$$alpha_{text{UCB}}(x) = mu(x) + beta_t sigma(x)$$
balancing exploration of high variance regions against exploitation of known functional maxima.
