A comprehensive survey of mixture-of-experts: Algorithms, theory, and applications

Analysis

Why This Paper Matters

As AI models scale to trillions of parameters, the computational cost of dense architectures becomes prohibitive. Mixture-of-Experts (MoE) offers a path to dramatically improve efficiency by activating only a subset of parameters per input. This survey arrives at a pivotal moment when MoE has been successfully deployed in production systems like Mixtral 8x7B and GPT-4, yet the field lacks a consolidated reference. The paper systematically organizes the fragmented literature, making it invaluable for practitioners deciding between MoE variants and for researchers seeking open problems.

The survey covers the full pipeline from routing algorithms (top-k, expert choice, hash-based) to training tricks (load balancing loss, auxiliary losses) and deployment considerations (expert parallelism, communication optimization). By including theoretical results on expert specialization and convergence, it bridges the gap between empirical success and formal understanding.

Technical Contributions

• Algorithmic Taxonomy: Classifies routing mechanisms into static (hash-based), dynamic (top-k, expert choice), and learned (softmax, reinforcement learning) categories, with clear trade-offs in load balance and specialization.
• Theoretical Insights: Summarizes proofs on expert specialization under sparse gating and conditions for convergence in overparameterized MoE models.
• Application Mapping: Provides a domain-by-domain breakdown of MoE adoption, including NLP (translation, language modeling), vision (ViT-MoE, object detection), and multi-modal (CLIP-MoE).
• Open Challenges: Highlights expert collapse (all experts learning similar functions), communication bottlenecks in distributed training, and lack of theoretical guarantees for dynamic routing.

Results

The survey reports that MoE models achieve 2-4x FLOPs reduction compared to dense models of equivalent quality on standard benchmarks (e.g., GLUE, ImageNet). Specific examples include Mixtral 8x7B matching Llama 2 70B performance with only 12.9B active parameters, and Vision MoE models reducing inference cost by 3x on ImageNet. The paper also notes that training stability remains a challenge, with load balancing losses adding 5-10% overhead.

Significance

This survey will likely become a standard reference for anyone working on scalable neural architectures. By clearly delineating what works, what is theoretically understood, and what remains unsolved, it accelerates both applied deployment and fundamental research. The identified open problems—especially expert collapse and dynamic routing—point to high-impact directions for the next generation of MoE systems.