A survey on mixture of experts in large language models

Analysis

Why This Paper Matters

As large language models (LLMs) grow to hundreds of billions of parameters, the computational cost of training and inference becomes prohibitive. Mixture of experts (MoE) offers a promising path to scale model capacity without a proportional increase in computation by activating only a subset of parameters per token. This survey arrives at a critical time when the AI community is actively seeking efficient scaling strategies, making it a valuable resource for both newcomers and experienced researchers.

The paper systematically organizes the rapidly growing MoE literature, which is essential given the fragmentation across different routing mechanisms, expert architectures, and training recipes. By providing a clear taxonomy, it helps practitioners navigate design choices and understand trade-offs.

Technical Contributions

• Architectural taxonomy: Categorizes MoE variants by expert type (dense, sparse, shared), routing strategy (top-k, expert choice, hash-based), and position within transformer layers.
• Training methodology: Reviews load balancing losses (e.g., auxiliary loss, Z-loss), expert parallelism techniques, and communication-efficient all-to-all operations.
• Inference optimization: Discusses expert caching, dynamic expert activation, and hardware-aware scheduling to reduce latency and memory.
• Application domains: Covers MoE for language modeling, multimodal models, and reinforcement learning, with examples like Mixtral 8x7B and Switch Transformer.

Results

The survey reports that MoE models achieve 2-4x parameter count with only 10-30% additional FLOPs compared to dense counterparts. For instance, Switch Transformer (1.6T parameters) matches the perplexity of a dense model with 1/10th the training cost. On downstream tasks, MoE LLMs often match or exceed dense models of similar inference cost. However, the survey notes that MoE can suffer from expert imbalance and higher memory footprint due to storing all expert parameters.

Significance

This survey consolidates knowledge from over 100 papers, providing a foundation for future research in efficient LLM scaling. It highlights that MoE is not a silver bullet—challenges like expert collapse, inference latency, and hardware inefficiency remain. By outlining open problems, it guides the community toward impactful next steps. For Neura Market readers, this paper is a must-read for understanding how to build cost-effective large-scale AI systems.