Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
Atindra Jha, Margaret Li, Jure Leskovec, et al.
Mixture-of-Experts models overfit more to repeated training data than dense models, with degradation increasing with sparsity; masking-based regularization can mitigate but not eliminate the gap.