Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing

Analysis

Why This Paper Matters

Sparse mixture-of-experts (MoE) models are a cornerstone of scaling large language models efficiently, but the principles underlying their routing decisions remain poorly understood. A common assumption is that co-selected experts should contribute distinct, non-overlapping representation directions—a geometric complementarity that justifies their joint use. This paper challenges that assumption by systematically separating route coherence, candidate quality, and candidate-by-context interaction, which previous analyses often conflated. By introducing the Expert Subspace Separation Index (ESSI) and a controlled factorial design, the authors provide a rigorous framework to disentangle these factors.

The findings are surprising: expert subspaces overlap substantially across six MoE architectures, yet actual routes still explain token representations better than matched alternatives. This suggests that routing is not merely selecting geometrically distinct experts, but rather selecting experts that are relevant to the token from a shared geometric neighborhood. The authors term this pattern 'coherent overlap'—a concept that reconciles geometric redundancy with functional utility. This insight is crucial for the AI community because it implies that pruning or merging experts based solely on geometric similarity could inadvertently remove functionally valuable components.

Technical Contributions

• Expert Subspace Separation Index (ESSI): A novel metric to quantify the degree of overlap between expert subspaces, enabling direct comparison across architectures.
• Matched-route residuals: A method to isolate the contribution of the actual routing decision from the quality of the candidate experts themselves.
• Prefix-controlled 2x2 factorial design: A controlled experiment that varies the prefix and candidate set to measure candidate-by-context interactions.
• Frozen-route interventions: A technique to assess the functional value of adding later experts by freezing the routing and comparing next-token prediction performance.
• Controlled Top-k training study: A training-based comparison of Top-2 vs. Top-1 routing across multiple seeds to validate functional benefits.

Results

The paper reports three paired contrasts. First, across six MoE architectures, expert subspaces overlap substantially, yet actual routes explain token representations better than matched alternatives. Second, across 39 factorial cells in OLMoE, Mixtral, and DeepSeek, the selected candidate explains more of the residual representation than the strongest unselected rival in every cell, but the actual prefix narrows this advantage: all interactions are negative, and every 95% confidence interval lies below zero. Third, adding later experts improves next-token prediction in 24 of 39 frozen-route comparisons, while the other 15 estimates are inconclusive; a controlled training study also favors Top-2 over Top-1 in all three seeds.

Significance

This paper provides a nuanced understanding of MoE routing, showing that geometric overlap does not necessarily imply redundancy. The concept of 'coherent overlap' offers a new lens for interpreting expert interactions and could inform more effective routing and pruning strategies. By separating the quantities that influence routing performance, the authors lay the groundwork for future research on expert specialization and model compression. This work is particularly relevant as MoE models become more prevalent in production AI systems, where efficiency and interpretability are paramount.