Scaling laws and spectra of shallow neural networks in the feature learning regime

Analysis

Why This Paper Matters

Neural scaling laws have become a cornerstone for predicting performance as model size and data grow, yet most theoretical work assumes the random features regime where only the last layer adapts. This paper breaks that assumption by studying the empirical risk minimization (ERM) solution in the teacher-student setting for shallow neural networks, explicitly allowing feature learning. Understanding scaling in this regime is crucial because real-world training often involves learning representations, not just linear readouts.

The work addresses a gap between theory and practice: while random features models yield clean power laws, they fail to explain phenomena like feature learning and the resulting changes in scaling exponents. By analyzing the ERM minimizer, the authors aim to uncover how feature learning alters scaling behavior and spectral properties. This could refine our theoretical toolkit for predicting when and why larger models help.

Given the surge in large-scale models, a theory that accounts for feature learning is timely. It may inform decisions about model width, depth, and optimization strategies, moving beyond heuristics.

Technical Contributions

• Beyond Random Features: The paper extends scaling law analysis to the feature learning regime, where hidden-layer weights adapt during training.
• Teacher-Student Framework: It formalizes the problem as a teacher-student setup, enabling precise asymptotic analysis.
• ERM Minimizer Focus: By studying the exact ERM solution, the work provides a rigorous baseline for understanding generalization.
• Spectral Analysis: It examines the spectra of shallow networks, linking eigenvalue distributions to scaling behavior.
• Scaling Laws Derivation: The authors derive how risk scales with sample size and network width in the presence of feature learning.

Results

The abstract does not provide concrete metrics, but the paper likely presents scaling exponents that differ from random features predictions. For instance, it may show that feature learning leads to faster decay of risk with more data or wider networks, or that spectral properties exhibit distinct power-law behaviors. Comparisons with random features baselines would highlight the impact of feature learning. Without specific numbers, the key takeaway is qualitative: feature learning changes the scaling picture.

Significance

This work pushes neural scaling theory closer to realistic training by incorporating feature learning. It could inspire further studies on deep networks and non-asymptotic regimes. Practically, it may help practitioners anticipate when feature learning yields better returns on scale. The spectral insights might also connect to optimization and generalization, offering a more unified view. Overall, it strengthens the theoretical foundation for why large neural networks work and how to design them efficiently.