Mixture-of-experts with expert choice routing

Analysis

Why This Paper Matters

Mixture-of-experts (MoE) models have become a cornerstone for scaling large language models without proportional compute increase. However, traditional token choice routing, where each token selects a subset of experts, often suffers from load imbalance and training instability. This paper introduces expert choice routing, a paradigm shift that flips the selection process: instead of tokens picking experts, each expert picks its top tokens. This simple inversion addresses the load balancing problem more naturally and leads to significant training speedups and quality improvements.

The paper is significant because it challenges the dominant routing paradigm and offers a practical alternative that is easy to implement within existing transformer architectures. By demonstrating consistent gains across various model sizes and benchmarks, it provides strong evidence that expert choice routing could become a standard component in future MoE models. This is particularly relevant as the AI community pushes toward more efficient training of trillion-parameter models.

Technical Contributions

• Expert Choice Routing: The core innovation is to have each expert select the top-k tokens based on gating scores, rather than tokens selecting experts. This ensures that each expert receives a fixed number of tokens, naturally balancing the load.
• No Token Dropping: Unlike token choice routing, which may drop tokens when experts are overloaded, expert choice routing processes all tokens, preserving information.
• Improved Training Dynamics: The fixed capacity per expert simplifies the training loop and reduces the need for auxiliary load-balancing losses, which are often required in token choice routing.
• Compatibility: The method is applied to the FFN layer of transformers, similar to Switch Transformer and GShard, making it a drop-in replacement for existing MoE layers.

Results

The paper reports that expert choice routing achieves up to 2x faster training convergence compared to token choice routing while maintaining or improving final model quality. For instance, on language modeling benchmarks, models using expert choice routing consistently outperform their token choice counterparts at the same compute budget. The method also shows better scaling behavior, with performance gains widening as model size increases. Additionally, the paper demonstrates that expert choice routing is more robust to hyperparameter choices, reducing the need for careful tuning of auxiliary losses.

Significance

This work has the potential to reshape how sparse models are designed. By offering a simpler and more efficient routing mechanism, it lowers the barrier to training large MoE models. The principles could extend beyond language models to other domains like vision and multimodal learning. Moreover, the idea of reversing the selection process might inspire other innovations in dynamic computation. As the field moves toward more efficient and scalable AI, expert choice routing provides a timely and impactful contribution.