Linear models represent the foundational benchmark of statistical pattern recognition. Long before deep connectionist architectures, Ronald Fisher, Harold Hotelling, and Nelder & Wedderburn laid the formal mathematical foundations linking linear projections to probability density estimation. Through Fisher’s Linear Discriminant Analysis (LDA) and Generalized Linear Models (GLMs), we discover how geometric projections, maximum likelihood estimation, and exponential families unite under a single mathematical paradigm.
1. Fisher’s Linear Discriminant: Maximizing the Scatter Ratio
In 1936, Ronald Fisher approached binary classification not through probabilistic modeling, but as an optimal dimensionality reduction problem: finding a linear projection vector $w in mathbb{R}^d$ such that projected scalar values $y = w^T x$ maximize separation between class centroids while minimizing intra-class dispersion.
Given class sample means $mu_1, mu_2$, the separation between projected means is $(w^T mu_1 – w^T mu_2)^2 = w^T S_B w$, where $S_B$ is the Between-Class Scatter Matrix:
$$S_B = (mu_1 – mu_2)(mu_1 – mu_2)^T$$
The total variance of the projected points within classes is $w^T S_W w$, where $S_W$ is the Within-Class Scatter Matrix:
$$S_W = sum_{x in mathcal{C}_1} (x – mu_1)(x – mu_1)^T + sum_{x in mathcal{C}_2} (x – mu_2)(x – mu_2)^T$$
Fisher formulated the objective as maximizing the Rayleigh quotient:
$$J(w) = frac{w^T S_B w}{w^T S_W w}$$
Differentiating with respect to $w$ and equating to zero yields the generalized eigenvalue equation $S_B w = lambda S_W w$. Because $S_B w$ is always in the direction of $(mu_1 – mu_2)$, the optimal projection vector possesses the magnificent analytical closed-form:
$$w^* propto S_W^{-1} (mu_1 – mu_2)$$
2. Generative vs. Discriminative Classifiers
Classification splits fundamentally into two probabilistic philosophies:
- Generative Classifiers (e.g., Gaussian Discriminant Analysis): Model the full joint distribution $P(x, y) = P(x mid y) P(y)$. By assuming class-conditional densities are multivariate Gaussians sharing a common covariance matrix $Sigma$ ($x mid y=k sim mathcal{N}(mu_k, Sigma)$), applying Bayes’ theorem to the posterior log-odds yields:
$$ln frac{P(y=1 mid x)}{P(y=0 mid x)} = w^T x + b, quad text{where} quad w = Sigma^{-1}(mu_1 – mu_0)$$
Notice that the generative Gaussian assumption recovers exactly the Fisher projection direction $w = Sigma^{-1}(mu_1 – mu_0)$!
- Discriminative Classifiers (e.g., Logistic Regression): Directly model the posterior boundary $P(y mid x) = sigma(w^T x)$ without making restrictive distributional assumptions about input density $P(x)$, maximizing conditional log-likelihood via convex cross-entropy.
3. Generalized Linear Models (GLMs) & Exponential Families
Nelder and Wedderburn (1972) unified regression, binary classification, and Poisson count modeling into Generalized Linear Models. A distribution belongs to the Exponential Family if its probability density can be written in canonical form:
$$p(y; eta) = b(y) expleft( eta^T T(y) – A(eta) right)$$
where $eta$ is the natural (canonical) parameter, $T(y)$ is the sufficient statistic, and $A(eta)$ is the log-partition function (cumulant generator). Crucially, the derivatives of $A(eta)$ analytically generate the distribution’s moments:
$$mathbb{E}[y] = nabla_eta A(eta), quad text{Var}(y) = nabla_eta^2 A(eta)$$
A GLM assumes that the target response $y$ follows an exponential family distribution whose expected value $mu = mathbb{E}[y mid x]$ is connected to a linear predictor $eta = w^T x$ via a monotonic link function $g(mu)$:
$$eta = g(mu) iff mu = g^{-1}(w^T x)$$
For Gaussian linear regression, the canonical link is the identity ($g(mu) = mu$). For Bernoulli binary classification, the canonical link is the logit function ($g(mu) = ln frac{mu}{1 – mu}$). For Poisson count models, it is the natural log ($g(mu) = ln mu$). This unified formulation guarantees strictly concave log-likelihoods, enabling deterministic Newton-Raphson convergence via Iteratively Reweighted Least Squares (IRLS).
