The History of Computing & AI (Part 13): Knowledge Engineering & Neural Resurgence — Expert Systems, Hopfield Nets, and Backpropagation

Following the icy desert of the First AI Winter, artificial intelligence split into two violently competing camps during the 1980s. The dominant commercial empire embraced Knowledge Engineering: hardcoding human expertise into thousands of deterministic production rules. Meanwhile, an underground diaspora of renegade cognitive scientists refused to bury the connectionist dream, proving through vector calculus that multilayer neural networks could learn hidden representations from data.

1. The Rise of Expert Systems: Knowledge as Power

By 1980, AI researchers abandoned the pursuit of general, domain-independent reasoning engines like GPS. Stanford professor Edward Feigenbaum formulated a new doctrine: the Knowledge Principle. Intelligence does not stem from clever, high-speed reasoning methods, but from vast stores of highly specific, real-world knowledge.

This gave birth to Expert Systems: computer programs designed to emulate the decision-making ability of human specialists within narrow technical domains. The architecture was decoupled into two distinct components:

• The Knowledge Base: A repository of domain-specific facts and heuristic production rules elicited through exhaustive interviews with human experts: IF <condition> THEN <action / conclusion>.
• The Inference Engine: A domain-independent logic interpreter that chained rules together via Forward Chaining (data-driven: deriving new facts from observed symptoms) or Backward Chaining (goal-driven: evaluating hypotheses backward to evidence).

MYCIN and XCON: Commercial Triumphs

Developed at Stanford by Edward Shortliffe, MYCIN (1976) diagnosed infectious blood diseases (such as bacteremia and meningitis) and recommended personalized antibiotic dosages. Because medical diagnoses are inherently probabilistic, MYCIN introduced Certainty Factors (CF) ranging from $-1.0$ (definitely false) to $+1.0$ (definitely true) to handle uncertain clinical evidence:

$$CF(h, e) = MB(h, e) – MD(h, e)$$

In formal clinical trials, MYCIN’s therapeutic recommendations achieved a 65% acceptability rate from independent infectious disease specialists—outperforming Stanford medical faculty physicians who averaged 42% to 62%.

In 1980, Digital Equipment Corporation (DEC) deployed John McDermott’s XCON (eXpert CONfigurer) to automatically configure customized VAX computer hardware orders. By 1986, XCON processed 20,000 orders annually with 98% accuracy, saving DEC an estimated \$40 million per year. Corporations rushed to invest billions in AI, and Japan launched its ambitious \$850 million Fifth Generation Computer Systems (FGCS) initiative to build parallel logic inference machines running Prolog.

2. John Hopfield and Energy Landscapes

While the corporate world poured billions into symbolic expert systems, theoretical physicist John Hopfield presented a revolutionary paper to the National Academy of Sciences in 1982: Neural networks and physical systems with emergent collective computational abilities.

Hopfield recognized that a network of symmetrically interconnected threshold binary neurons was mathematically equivalent to an Ising spin glass model in statistical physics. He defined a global scalar Energy Function ($E$) over the network’s state space:

$$E = -\frac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} w_{ij} s_i s_j + \sum_{i=1}^{N} \theta_i s_i, \quad \text{with } w_{ij} = w_{ji}, \, w_{ii} = 0$$

Hopfield proved through Lyapunov stability that if units update their states asynchronously according to the local threshold rule $s_i = \text{sign}(\sum w_{ij} s_j – \theta_i)$, the global energy $E$ decreases monotonically or remains constant ($\Delta E \le 0$). The network inexorably slides downhill along its high-dimensional energy landscape into the nearest local energy minimum (an attractor basin).

A Hopfield network was an associative (content-addressable) memory: by storing memories as energy minima via Hebbian weighting, feeding the network a noisy, incomplete, or corrupted fragment of an image caused the network to reconstruct the full, pristine original pattern autonomously!

3. The Backpropagation Breakthrough (1986)

Despite Hopfield’s breakthrough, neural networks remained crippled by Marvin Minsky and Seymour Papert’s 1969 proof: single-layer perceptrons could not compute non-linear functions like XOR, and nobody knew how to train the hidden layers of a multilayer network.

In 1986, cognitive psychologist David Rumelhart, computer scientist Geoffrey Hinton, and mathematician Ronald Williams published an epochal, three-page paper in Nature: Learning representations by back-propagating errors. Although variants of continuous adjoint automatic differentiation had been formulated earlier by Seppo Linnainmaa (1970) and Paul Werbos (1974), Rumelhart, Hinton, and Williams demonstrated its transformative power on internal representations in neural networks.

The Mathematics of Backpropagation

The breakthrough required two conceptual shifts:

1. Replace the discontinuous, non-differentiable step function of the Rosenblatt perceptron with a continuous, smooth non-linear activation: the logistic sigmoid function $\sigma(z) = \frac{1}{1 + e^{-z}}$, whose derivative is elegantly self-referential: $\sigma'(z) = \sigma(z)(1 – \sigma(z))$.
2. Apply the calculus chain rule to propagate the error gradient backward from the output layer through all intermediate hidden layers.

For an output unit $k$ with target $y_k$ and prediction $\hat{y}_k$, the mean squared error loss is $\mathcal{L} = \frac{1}{2} \sum (y_k – \hat{y}_k)^2$. The output error term is:

$$\delta_k = \frac{\partial \mathcal{L}}{\partial z_k} = (\hat{y}_k – y_k) \cdot \sigma'(z_k) = (\hat{y}_k – y_k) \hat{y}_k (1 – \hat{y}_k)$$

For an arbitrary hidden unit $j$, its error signal is computed recursively by backpropagating the weighted sum of error terms from all connected units in the subsequent layer:

$$\delta_j = \frac{\partial \mathcal{L}}{\partial z_j} = \left( \sum_{k} \delta_k w_{jk} \right) \cdot \sigma'(z_j) = \left( \sum_{k} \delta_k w_{jk} \right) a_j (1 – a_j)$$

The weight update across any layer follows gradient descent:

$$\Delta w_{ij} = -\eta \frac{\partial \mathcal{L}}{\partial w_{ij}} = -\eta \cdot \delta_j \cdot a_i$$

Rumelhart, Hinton, and Williams showed that when presented with the dreaded XOR problem, a simple network with two hidden units running backpropagation autonomously discovered non-linear internal representations: one hidden neuron learned the sub-feature $x_1 \vee x_2$, the second learned $\neg (x_1 \wedge x_2)$, and the output neuron merged them into a clean, flawless XOR classification. The 17-year Minsky-Papert curse was broken forever.

4. The Collapse of the Expert Systems Bubble: Second AI Winter

As connectionism was reborn in academic laboratories, the commercial expert systems industry drove off a cliff in 1987. Rule-based systems suffered from fatal structural vulnerabilities:

• The Brittleness Problem: If an input fell even slightly outside the hardcoded rule set, the system failed catastrophically without graceful degradation.
• Combinatorial Rule Conflicts: As knowledge bases expanded past 5,000 rules, maintaining consistency became impossible. Adding rule 5,001 frequently contradicted rule 427, causing deadlocks.
• The Commodity Hardware Squeeze: In the early 1980s, companies like Symbolics and Lisp Machines Inc. sold custom, microcoded Lisp hardware for \$100,000 per workstation. By 1987, general-purpose desktop workstations powered by standard Intel 386 and Sun SPARC processors ran software faster and at one-tenth the cost.

The specialized AI hardware market crashed overnight; dozens of commercial AI startups went bankrupt, and government funding dried up. The second AI winter settled in. Yet beneath the ice, the mathematical seeds of statistical learning and backpropagation were preparing to bloom.

---