Shannon’s definition of entropy is not merely an intuitive heuristic; it is an inescapable mathematical necessity. In 1957, Soviet mathematician Aleksandr Khinchin demonstrated that any functional measuring the uncertainty of a probability distribution that satisfies four basic operational axioms—continuity, maximality under uniform distribution, expansibility, and additivity—must be uniquely proportional to Shannon’s formula. Expanding this measure to multiple interacting random variables generates the algebraic landscape of Joint Entropy, Conditional Entropy, and Mutual Information, establishing an information-theoretic geometry isomorphic to classical set theory.
1. The Khinchin Axiomatic Derivation
Let $Delta_n = { (p_1, dots, p_n) in mathbb{R}^n : p_i ge 0, sum_{i=1}^n p_i = 1 }$ be the simplex of probability distributions over an $n$-element event space. A measure of uncertainty $H(p_1, dots, p_n)$ must satisfy four Khinchin axioms:
- Continuity: $H(p_1, dots, p_n)$ is a continuous function of its arguments on $Delta_n$.
- Maximality: For a fixed $n$, uncertainty is strictly maximized when all outcomes are equally probable (uniform distribution):
[
H(p_1, dots, p_n) le Hleft(frac{1}{n}, dots, frac{1}{n}right)
] - Expansibility: Appending an impossible event ($p_{n+1} = 0$) leaves the uncertainty unchanged:
[
H(p_1, dots, p_n, 0) = H(p_1, dots, p_n)
] - Strong Additivity: If an experiment consists of composite stages, the total uncertainty equals the uncertainty of the first stage plus the conditional expectation of the remaining uncertainty:
[
H(X, Y) = H(X) + H(Y|X)
]
Khinchin proved that the unique class of functions satisfying all four axioms is:
[
H(p_1, dots, p_n) = -k sum_{i=1}^n p_i log_b p_i
]
where $k > 0$ is an arbitrary scaling constant. Choosing $b = 2$ and $k = 1$ establishes the standard bit-measure of Shannon entropy.
2. Bivariate Measures: Joint and Conditional Entropy
Consider two discrete random variables $X in mathcal{X}$ and $Y in mathcal{Y}$ governed by joint probability distribution $p(x, y) = mathbb{P}(X=x, Y=y)$. The marginal distributions are $p(x) = sum_y p(x, y)$ and $p(y) = sum_x p(x, y)$.
Joint Entropy
The total uncertainty associated with the pair $(X, Y)$ is the Joint Entropy:
[
H(X, Y) = -sum_{x in mathcal{X}} sum_{y in mathcal{Y}} p(x, y) log_2 p(x, y)
]
Conditional Entropy (Equivocation)
The remaining uncertainty in $X$ given that variable $Y$ has been observed is the Conditional Entropy $H(X|Y)$:
[
H(X|Y) = sum_{y in mathcal{Y}} p(y) H(X | Y=y) = -sum_{x in mathcal{X}} sum_{y in mathcal{Y}} p(x, y) log_2 p(x|y)
]
A fundamental property of information is that conditioning never increases entropy on average:
[
H(X|Y) le H(X)
]
Equality holds if and only if $X$ and $Y$ are statistically independent ($p(x, y) = p(x)p(y)$). Learning an auxiliary variable $Y$ can never make you more ignorant of $X$.
3. Mutual Information: The Shared Uncertainty Core
The central currency of communication channels is Mutual Information $I(X; Y)$. It quantifies the reduction in uncertainty of $X$ due to knowledge of $Y$ (and vice versa):
[
I(X; Y) = H(X) – H(X|Y) = H(Y) – H(Y|X)
]
Expanding the definitions reveals the symmetric expression:
[
I(X; Y) = sum_{x in mathcal{X}} sum_{y in mathcal{Y}} p(x, y) log_2 frac{p(x, y)}{p(x) p(y)}
]
Notice the exact algebraic isomorphism between Shannon information measures and classical Venn set theory:
- Set Union $A cup B iff$ Joint Entropy $H(X, Y)$
- Set Intersection $A cap B iff$ Mutual Information $I(X; Y)$
- Set Difference $A setminus B iff$ Conditional Entropy $H(X|Y)$
- Set Difference $B setminus A iff$ Conditional Entropy $H(Y|X)$
By inclusion-exclusion:
[
H(X, Y) = H(X) + H(Y) – I(X; Y) iff I(X; Y) = H(X) + H(Y) – H(X, Y)
]
Furthermore, $I(X; X) = H(X)$, demonstrating that the self-information of a random variable is simply its Shannon entropy.
4. Relative Entropy: The Kullback-Leibler Divergence
To quantify the statistical distance between two probability distributions $p(x)$ and $q(x)$ defined over the same alphabet, information theory introduces the Kullback-Leibler (KL) Divergence (or Relative Entropy):
[
D_{KL}(p parallel q) = sum_{x in mathcal{X}} p(x) log_2 frac{p(x)}{q(x)}
]
By Jensen’s inequality applied to the strictly concave logarithm function, the Information Inequality proves:
[
D_{KL}(p parallel q) ge 0
]
with equality if and only if $p(x) = q(x)$ for all $x$. Notice that mutual information is simply the KL divergence between the joint distribution $p(x, y)$ and the independent product distribution $p(x)p(y)$:
[
I(X; Y) = D_{KL}(p(x, y) parallel p(x)p(y)) ge 0
]
This guarantees that mutual information is strictly non-negative, vanishing to zero if and only if $X$ and $Y$ are completely uncorrelated.
