For nearly a century after Ludwig Boltzmann and J. Willard Gibbs founded statistical mechanics, physicists engaged in bitter, contentious debates regarding the fundamental nature of thermodynamic entropy. Was entropy an objective physical substance like mass and energy, or did it measure subjective human ignorance? In 1957, American physicist Edwin Thompson Jaynes published a revolutionary two-part paper in the Physical Review titled “Information Theory and Statistical Mechanics.” Jaynes proved that statistical mechanics is not a separate physical theory requiring ad-hoc ergodic hypotheses; rather, it is simply an application of Objective Bayesian Inference governed by Claude Shannon’s information entropy. The celebrated Boltzmann-Gibbs distribution is the unique probability assignment that maximizes Shannon entropy subject to known macroscopic physical constraints.
1. The Problem of Statistical Inference
Consider a physical system (such as a container of $10^{23}$ gas molecules) that can exist in an astronomical number of discrete quantum microstates ${1, 2, dots, N}$, each having energy $E_i$. Experimentally, we cannot observe the microscopic coordinates of individual molecules; we can measure only a single macroscopic expectation value: the total average energy $langle E rangle$.
How should a rational scientist assign a probability distribution $p = (p_1, p_2, dots, p_N)$ over these microstates when the number of unknowns ($N approx 10^{23}$) infinitely exceeds our single piece of data ($sum p_i E_i = langle E rangle$)?
Any arbitrary probability distribution that satisfies the energy constraint introduces unproven physical assumptions. Jaynes recognized that the only rationally honest, maximally unbiased distribution is the one that maximizes Shannon entropy $H(p)$ subject strictly to the known constraints, remaining completely agnostic regarding everything else.
2. Mathematical Derivation of the Canonical Ensemble via MaxEnt
We formulate the variational optimization problem using the method of Lagrange multipliers:
[
text{Maximize: } H(p) = -sum_{i=1}^N p_i ln p_i
]
subject to two constraints:
- Normalization: $sum_{i=1}^N p_i = 1$
- Average Energy: $sum_{i=1}^N p_i E_i = langle E rangle$
We construct the Lagrangian function $mathcal{L}(p_1, dots, p_N, lambda_0, beta)$:
[
mathcal{L} = -sum_{i=1}^N p_i ln p_i – lambda_0 left( sum_{i=1}^N p_i – 1 right) – beta left( sum_{i=1}^N p_i E_i – langle E rangle right)
]
Taking the partial derivative with respect to each probability $p_i$ and setting it to zero:
[
frac{partial mathcal{L}}{partial p_i} = -ln p_i – 1 – lambda_0 – beta E_i = 0
]
Solving directly for $p_i$:
[
ln p_i = -(1 + lambda_0) – beta E_i implies p_i = e^{-(1 + lambda_0)} e^{-beta E_i}
]
Enforcing the normalization constraint $sum p_i = 1$, we define the normalization denominator as the Partition Function $Z(beta)$:
[
Z(beta) = sum_{i=1}^N e^{-beta E_i}
]
which immediately yields the legendary Boltzmann-Gibbs Canonical Distribution:
[
p_i = frac{1}{Z(beta)} e^{-beta E_i}
]
3. Unifying Shannon and Gibbs Entropy
What is the maximum entropy achieved by this distribution? Substituting $p_i$ back into Shannon’s entropy formula:
[
H_{text{max}} = -sum_{i=1}^N p_i ln left( frac{e^{-beta E_i}}{Z} right) = -sum_{i=1}^N p_i (-beta E_i – ln Z) = beta sum p_i E_i + ln Z sum p_i
]
[
H_{text{max}} = beta langle E rangle + ln Z(beta)
]
In classical thermodynamics, the Helmholtz free energy is defined as $F = langle E rangle – T S_{text{thermo}}$, and statistical mechanics relates it to the partition function via $F = -k_B T ln Z$. Dividing by $k_B T$ (with thermodynamic beta $beta = frac{1}{k_B T}$):
[
frac{S_{text{thermo}}}{k_B} = beta langle E rangle + ln Z
]
Comparing these two equations establishes the definitive identity:
[
S_{text{thermo}} = k_B cdot H_{text{Shannon}}
]
Thermodynamic entropy is not an independent physical fluid; it is literally Shannon information entropy multiplied by Boltzmann’s physical constant $k_B$!
Jaynes dissolved the century-old mystery of the Second Law of Thermodynamics. The Second Law ($dS ge 0$) does not describe a physical deterioration of matter; it describes the inexorable loss of our microscopic information. When a gas expands into a vacuum, the molecules do not lose physical mass or charge; rather, the volume of phase space compatible with our macroscopic knowledge increases exponentially, multiplying our missing information.
Today, the Maximum Entropy Principle (MaxEnt) extends far beyond physics: it powers natural language processing, financial risk modeling, image reconstruction in radio astronomy, and ecology.
