How feature learning can improve neural scaling laws

Analysis

Why This Paper Matters

Scaling laws—the empirical relationships between model size, data, compute, and performance—have become a cornerstone of modern deep learning. However, most theoretical understanding of these laws comes from the neural tangent kernel (NTK) regime, where networks behave like fixed feature maps. This paper breaks new ground by developing a solvable model that captures how feature learning—the ability of neural networks to adapt their representations during training—changes scaling behavior. Understanding this transition is crucial as practitioners push toward ever-larger models where feature learning is known to be important.

The paper addresses a fundamental gap: while empirical scaling laws often show better-than-kernel performance, theory has struggled to explain why. By providing an analytically tractable model, the authors offer a rigorous foundation for understanding when and how feature learning improves scaling. This matters for both theorists seeking to understand deep learning and practitioners deciding how to allocate compute resources.

Technical Contributions

The key innovation is a solvable model that interpolates between the kernel limit and a feature-learning regime. The authors introduce a controlled setting where the network's feature map can adapt during training, and they derive exact scaling laws for the test error. This allows them to compute how the scaling exponent—the rate at which error decreases with model size or training time—depends on the degree of feature learning.

• Solvable model: A simplified neural network architecture that admits exact analysis beyond the NTK regime.
• Scaling exponents: Analytical expressions for how test error scales with model size and training time as a function of feature learning strength.
• Phase diagram: Identification of regimes where feature learning improves scaling versus where kernel behavior dominates.

Results

The model yields concrete predictions: feature learning can improve the scaling exponent for model size by a constant factor compared to the kernel limit. For training time, the improvement is even more pronounced, with feature learning enabling faster convergence. The analysis also reveals a trade-off: feature learning helps most when the target function is complex and data is abundant, but can be detrimental in data-poor settings.

Significance

This work provides a theoretical anchor for the empirical observation that large models benefit from feature learning. It offers a framework for predicting when investing in larger models or longer training will yield the best returns. For the AI field, it bridges the gap between kernel theory and practical deep learning, potentially guiding the design of architectures and training algorithms that optimally leverage feature learning for better scaling.