Vimoe: An empirical study of designing vision mixture-of-experts

Analysis

Why This Paper Matters

Mixture-of-Experts (MoE) has emerged as a key technique for scaling neural networks without linearly increasing computational cost. While MoE has been extensively studied in natural language processing, its application to computer vision remains less understood. This paper fills that gap by providing a systematic empirical study of MoE design choices specifically for vision models, offering actionable insights for practitioners.

The significance lies in its focus on practical design decisions—such as how many experts to use, how to route inputs, and how to balance expert loads—rather than proposing a single new architecture. This makes the findings directly applicable to engineers building large-scale vision systems.

Technical Contributions

• Comprehensive ablation study on MoE components: number of experts (2, 4, 8, 16), routing strategies (top-k, noisy top-k, sparse routing), and capacity factors.
• Identification of optimal routing: Sparse routing with learned gating outperforms standard top-k routing in accuracy-efficiency trade-offs.
• Expert specialization analysis: Shows that experts naturally specialize in different visual features (e.g., textures, shapes) when properly trained.
• Scaling laws for vision MoE: Demonstrates that MoE models can achieve higher accuracy than dense models at the same FLOPs budget.

Results

The paper reports that a MoE model with 8 experts and sparse routing achieves 2.1% higher top-1 accuracy on ImageNet compared to a dense baseline with equivalent FLOPs. Additionally, inference cost is reduced by 40% relative to a dense model of the same total parameter count. The optimal configuration uses a capacity factor of 1.25 and a routing temperature of 0.1.

Significance

This work provides a foundational reference for researchers and engineers deploying MoE in vision tasks. By demystifying design choices, it lowers the barrier to adopting MoE for large-scale image classification and potentially other vision applications. The empirical insights could accelerate progress in efficient vision models, especially in resource-constrained environments.