Preprint
Machine Learning

Understanding deep learning (still) requires rethinking generalization

Chiyuan Zhang(Google (United States)), Samy Bengio(Google (United States)), Moritz Hardt(University of California, Berkeley), Benjamin Recht(University of California, Berkeley), Oriol Vinyals(Google DeepMind (United Kingdom))
February 22, 2021Communications of the ACM2,785 citations
PDFDOI

2.8k

Citations

105

Influential Citations

Communications of the ACM

Venue

2021

Year

Abstract

Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models. We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.

Analysis

Why This Paper Matters

This paper fundamentally challenges the classical bias-variance tradeoff and traditional notions of overfitting in deep learning. By showing that large neural networks can memorize random labels and noise while still generalizing well on real data, it forces the community to rethink why deep networks work. The findings have profound implications for understanding generalization, regularization, and the role of model capacity.

The work is highly cited (2785 citations) and has become a cornerstone in the debate on implicit regularization. It highlights that explicit regularization (e.g., weight decay, dropout) is not the primary driver of generalization, shifting focus to properties of the optimization algorithm (SGD) and the data distribution.

Technical Contributions

  • Empirical demonstration: ConvNets trained with SGD achieve near-zero training error on completely random labels and unstructured noise (e.g., random pixels).
  • Insensitivity to regularization: Adding explicit regularizers does not prevent fitting of random data, showing that generalization gap is not controlled by traditional complexity measures.
  • Theoretical construction: Proves that depth-two networks with more parameters than data points can perfectly fit any labeling, establishing finite-sample expressivity.
  • Comparison with classical models: Contrasts with kernel methods and linear models that cannot fit random labels without severe overfitting, highlighting unique properties of deep networks.

Results

The key result is that a standard ConvNet (e.g., AlexNet) can achieve 100% training accuracy on CIFAR-10 with randomly shuffled labels, while test accuracy remains near chance (10%). This holds even when images are replaced by random noise. Explicit regularization (weight decay, dropout, data augmentation) does not qualitatively change this behavior. The theoretical result shows that a depth-two network with ReLU activations and d parameters can shatter any set of n points if d >= 2n, matching the empirical regime.

Significance

This paper has reshaped the field's understanding of generalization in deep learning. It motivated extensive research into implicit regularization (e.g., SGD's tendency to find low-norm solutions), the lottery ticket hypothesis, and the role of data structure. It also spurred new theoretical frameworks such as neural tangent kernels and the study of overparameterized models. Practitioners now recognize that large models can memorize but still generalize, leading to practical guidelines like using more data and careful tuning of optimization rather than heavy regularization.

deep-learninggeneralizationoverfittingconvolutional-networksstochastic-gradient-descentexpressivityregularization
Neural Networks

Related Papers