Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data

Analysis

Why This Paper Matters

As the supply of high-quality human-written text approaches exhaustion, repeating training data has become a common practice for training large language models. Prior work has examined the effects of data repetition on densely activated Transformers, but the behavior of Mixture-of-Experts (MoE) models—now dominant due to their compute efficiency—remains largely unexplored. This paper addresses that gap by systematically studying how MoEs respond to repeated data across a range of model sizes and configurations.

The findings are significant because they reveal a fundamental tension between model sparsity and data repetition: MoEs, which are designed to scale efficiently by activating only a subset of parameters per input, overfit more rapidly to repeated data than dense models. This suggests that the very architectural choices that make MoEs attractive for large-scale training may also make them more vulnerable in data-constrained settings. As the field increasingly relies on both sparsity and data repetition, understanding this interaction is critical for developing robust training strategies.

Moreover, the paper goes beyond documenting the problem by testing regularization methods and analyzing internal mechanisms, offering actionable insights for practitioners and laying groundwork for future research on mitigating overfitting in sparse architectures.

Technical Contributions

• Systematic evaluation of data repetition on MoEs: The authors vary repetition rates across single- and multi-domain data mixes and across MoE settings, including expert count and granularity, for models from 80M to 1B active (8.5B total) parameters.
• Quantification of degradation: They show that MoEs degrade more rapidly under data repetition than dense models, with the effect increasing with sparsity and dictated by total rather than active parameters.
• Regularization experiments: They test existing regularization methods, finding that dropout and strong masking-based regularization can mitigate overfitting, enabling MoEs to outperform dense models even at 64x repetition.
• Mechanistic analysis: They identify that MoE routing stabilizes early in training and that expert specialization correlates with overfitting to repeated data, providing insight into why MoEs are more susceptible.

Results

The paper reports that 80M dense models can repeat data over 8x with minimal degradation, whereas MoEs begin to suffer at 4x and deteriorate rapidly, ceding their performance benefits in all-unique data settings to underperform dense models after 32x repetition. The degradation is driven by total parameters rather than active parameters, meaning that sparser MoEs (with more total parameters) are more affected. Among regularization methods, masking-based regularization is particularly effective: with strong masking, MoEs are able to outperform dense models even when data is repeated more than 64 times. However, no method fully matches the performance of all-unique training data. The analysis further reveals that MoE routing universally stabilizes early in training, and expert specialization correlates with overfitting to repeated data.

Significance

This work highlights a critical trade-off between model sparsity and data repetition, showing that MoEs—despite their compute efficiency—are more prone to overfitting when data is scarce and repeated. The findings have immediate implications for practitioners training MoEs on limited data: they may need to adjust repetition rates, apply stronger regularization, or reconsider architecture choices. The mechanistic insights, particularly the link between expert specialization and overfitting, suggest that future methods could target over-specialization to improve robustness. More broadly, the paper underscores the need to understand how architectural innovations interact with data constraints, especially as the field pushes toward ever-larger and sparser models in a data-limited world.