Sequential temporal processes—speech, natural language, financial time series, and biophysical kinetics—defy static feed-forward mappings: their outputs depend critically on an unbounded history of past observations. Recurrent Neural Networks (RNNs) formalize this sequence dependence by maintaining a latent dynamical state $h_t in mathbb{R}^d$. However, standard RNNs historically suffered from the Vanishing and Exploding Gradient Problem, which bounded temporal credit assignment to a handful of steps. In 1997, Sepp Hochreiter and Jürgen Schmidhuber solved this with the Long Short-Term Memory (LSTM) architecture.
1. The Mathematics of Temporal Unrolling and BPTT
A standard Elman recurrent network evolves according to the non-linear recurrence:
$$h_t = sigma(W_{hh} h_{t-1} + W_{xh} x_t + b_h)$$
To train parameters using gradient descent, the network is unrolled across $T$ discrete time steps—a procedure known as Backpropagation Through Time (BPTT). Evaluating the gradient of the loss at step $T$ with respect to parameters $W_{hh}$ requires applying the chain rule across the recurrent sequence:
$$frac{partial L_T}{partial W_{hh}} = sum_{t=1}^T frac{partial L_T}{partial h_T} frac{partial h_T}{partial h_t} frac{partial h_t}{partial W_{hh}}$$
The temporal credit assignment bottleneck resides in the temporal Jacobian product:
$$frac{partial h_T}{partial h_t} = prod_{k=t+1}^T frac{partial h_k}{partial h_{k-1}} = prod_{k=t+1}^T text{diag}(sigma'(z_k)) W_{hh}^T$$
2. Bengio’s Spectral Radius Theorem
In 1994, Yoshua Bengio et al. proved the fundamental limitation of recurrent credit assignment: let $rho(W_{hh})$ be the spectral radius (largest absolute eigenvalue) of weight matrix $W_{hh}$, and let $|sigma'(z)| le gamma$. The norm of the Jacobian product is bounded by:
$$left| frac{partial h_T}{partial h_t} right| le (gamma rho(W_{hh}))^{T – t}$$
- Vanishing Gradient Regime ($gamma rho < 1$): The gradient norm decays exponentially with temporal distance $Delta t = T – t$. For $T – t ge 50$, gradients shrink below floating-point precision ($10^{-16}$), rendering the model incapable of learning long-range dependencies.
- Exploding Gradient Regime ($gamma rho > 1$): Gradients explode exponentially toward numerical infinity ($text{NaN}$), causing destructive parameter divergence (mitigated heuristically via norm gradient clipping $|nabla| le tau$).
3. The Long Short-Term Memory (LSTM) Solution
Hochreiter and Schmidhuber resolved this dilemma by introducing the Constant Error Carousel (CEC): enforcing a linear self-connected unit where the Jacobian equals the identity matrix ($frac{partial c_t}{partial c_{t-1}} = I$).
Modern LSTMs (enhanced with Felix Gers’s 2000 forget gate) govern internal information routing via four interacting affine gates:
$$begin{bmatrix} f_t \ i_t \ o_t \ tilde{c}_t end{bmatrix} = begin{bmatrix} sigma \ sigma \ sigma \ tanh end{bmatrix} left( W begin{bmatrix} h_{t-1} \ x_t end{bmatrix} + b right)$$
The internal cell state updates additively:
$$c_t = f_t odot c_{t-1} + i_t odot tilde{c}_t$$
$$h_t = o_t odot tanh(c_t)$$
Because the update to cell state $c_t$ is purely additive, the gradient $frac{partial c_T}{partial c_t} = prod_{k=t+1}^T f_k$ does not undergo repeated matrix multiplications with weight matrices. When forget gate $f_k approx 1$, error signals propagate across hundreds of steps without attenuation.
4. The Gated Recurrent Unit (GRU)
In 2014, Kyunghyun Cho et al. streamlined the LSTM into the Gated Recurrent Unit (GRU), eliminating the separate cell state $c_t$ and coupling the forget and input gates into a single update gate $z_t$:
$$z_t = sigma(W_z [h_{t-1}, x_t]), quad r_t = sigma(W_r [h_{t-1}, x_t])$$
$$h_t = (1 – z_t) odot h_{t-1} + z_t odot tilde{h}_t$$
GRUs reduce parameter count by $25%$ while matching LSTM expressivity, establishing the standard sequential recurrent baseline prior to the Transformer revolution.
