A comprehensive index of artificial intelligence and machine-learning research with AI-generated summaries, citation metrics, and direct links to papers and code.
X. Shi, Shiyu Wang, Yuqi Nie, et al.
Time-MoE introduces a scalable mixture-of-experts architecture for billion-scale time series foundation models, demonstrating effective scaling laws and strong zero-shot performance.
Unknown
This paper introduces a mixture compressor for Mixture-of-Experts LLMs, aiming to improve efficiency and performance by compressing expert outputs.
Unknown
DeepSeekMoE introduces a novel MoE architecture with fine-grained expert segmentation and shared experts to achieve ultimate expert specialization in language models.
Unknown
This paper provides a closer look into mixture-of-experts (MoE) in large language models, analyzing their architecture, routing mechanisms, and performance trade-offs.
Danyang Zhang, Junhao Song, Ziqian Bi, et al.
A comprehensive review of Mixture-of-Experts (MoE) in large language models, covering architecture, routing, meta-learning, and applications.
Unknown
OpenMoE is an early effort to train and release fully open Mixture-of-Experts (MoE) based large language models to help the open-source community understand MoE LLMs.
Unknown
This paper introduces a multi-head mixture-of-experts (MH-MoE) architecture that addresses low expert activation by splitting experts into multiple heads, improving expert utilization and performance.
Huiyuan Tian, Bonan Xu, Shijian Li
This paper introduces the Expert Subspace Separation Index (ESSI) and controlled experiments to show that sparse MoE routing selects token-relevant experts from overlapping subspaces, a pattern termed 'coherent overlap' that preserves functional valu
Unknown
This paper proposes expert choice routing for mixture-of-experts, where each expert selects top tokens, improving load balance and training efficiency.
Unknown
Gemini 1.5 Pro is a compute-efficient multimodal mixture-of-experts model excelling in long-context retrieval and understanding across text, video, and audio.
Unknown
MoE-LLaVA introduces a sparse mixture-of-experts framework for vision-language models, activating only top-k experts per token to match larger model performance with fewer parameters.
Unknown
Introduces Mixtral 8x7B, a sparse mixture-of-experts language model trained on multilingual data with a 32k-token context window.