Prior to 2015, the deep learning community confronted an unexpected empirical crisis: the Degradation Problem. As networks grew beyond 20 layers, training error—not just test error—began to increase dramatically. Crucially, this was not caused by overfitting (which would exhibit lower training error alongside higher test error), but by an optimization failure: deeper architectures failed to learn even an identity mapping! In late 2015, Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun resolved this barrier with Deep Residual Learning (ResNet), enabling the stable optimization of networks with 152 to over 1,000 layers.
1. The Degradation Paradox and Residual Formulations
Consider a network with $L$ stacked non-linear layers. Suppose there exists an optimal shallow mapping $mathcal{H}^*(x)$ at depth $l < L$. In theory, if subsequent layers $l+1 to L$ simply learn the identity function $mathcal{H}(x) = x$, the deeper network must achieve training error no worse than the shallow model. Yet empirically, non-linear stacks of layers $sigma(W x + b)$ struggle to approximate identity mappings due to the curvature of activation manifolds.
He et al. hypothesized that instead of forcing stacked layers to fit the underlying mapping $mathcal{H}(x)$, they should explicitly fit a Residual Mapping:
$$mathcal{F}(x) equiv mathcal{H}(x) – x implies mathcal{H}(x) = mathcal{F}(x) + x$$
If an identity mapping is optimal, the optimization dynamics simply drive weights toward zero ($mathcal{F}(x) to 0$), which is trivial for standard weight decay regularizers.
2. Mathematical Derivation of Unimpeded Gradient Flow
Consider a chain of residual blocks where each block computes $x_{l+1} = x_l + mathcal{F}(x_l, mathcal{W}_l)$. Recursively expanding this recurrence from layer $l$ to arbitrary deeper layer $L > l$ yields the fundamental identity:
$$x_L = x_l + sum_{i=l}^{L-1} mathcal{F}(x_i, mathcal{W}_i)$$
This formulation exhibits profound properties: any deep representation $x_L$ is the linear sum of the early representation $x_l$ plus an additive aggregation of residual corrections. Applying the chain rule to evaluate the gradient of scalar loss $mathcal{E}$ with respect to activation $x_l$:
$$frac{partial mathcal{E}}{partial x_l} = frac{partial mathcal{E}}{partial x_L} frac{partial x_L}{partial x_l} = frac{partial mathcal{E}}{partial x_L} left( I + frac{partial}{partial x_l} sum_{i=l}^{L-1} mathcal{F}(x_i, mathcal{W}_i) right)$$
This gradient decomposition reveals why vanishing gradients are eliminated: the term $I$ acts as an unattenuated Gradient Highway. Even if the Jacobian term $frac{partial}{partial x_l} sum_i mathcal{F}_i$ vanishes or fluctuates chaotically, the identity term guarantees that $frac{partial mathcal{E}}{partial x_L}$ propagates directly back to layer $l$ without exponential decay.
3. The Unraveled Ensemble Interpretation
In 2016, Andreas Veit, Michael Wilber, and Serge Belongie proved that ResNets do not behave like a single monolithic ultra-deep network; rather, they behave like an Unraveled Ensemble of Exponentially Many Shallow Networks.
A ResNet with $n$ residual blocks contains $2^n$ distinct computational paths from input to output. Veit et al. showed that dropping individual layers during inference (or training via Stochastic Depth) barely affects performance, whereas deleting a single layer in a plain VGG network completely obliterates performance. Furthermore, gradient norm analyses demonstrate that the effective path lengths contributing to gradient updates remain around $O(sqrt{n})$ to $O(n/2)$, establishing ResNets as parallel implicit ensembles.
4. Modern Descendants: DenseNet, ResNeXt, and ConvNeXt
The residual paradigm spawned a lineage of foundational visual architectures:
- DenseNet (Huang et al., 2017): Connects all layers directly ($x_l = H_l([x_0, x_1, dots, x_{l-1}])$), maximizing feature reuse via concatenation.
- ResNeXt (Xie et al., 2017): Introduces Cardinality—splitting residual transformations into multi-branch grouped convolutions, demonstrating that topological cardinality is more effective than sheer depth or width.
- ConvNeXt (Liu et al., 2022): Modernizes pure convolutional networks with $7 times 7$ depthwise convolutions, inverted bottlenecks, and LayerNorm, proving that classical convolutions match vision transformers when architected under modern residual principles.
