Time-moe: Billion-scale time series foundation models with mixture of experts

Analysis

Why This Paper Matters

Time series forecasting is critical across finance, healthcare, energy, and many other domains. Traditional models are often task-specific and require retraining for each new dataset. The emergence of foundation models in NLP and vision has inspired similar efforts for time series, but scaling these models has been challenging due to the heterogeneity of time series data and the computational cost of dense transformers. Time-MoE addresses this by introducing a mixture-of-experts (MoE) architecture that scales to billions of parameters while keeping inference efficient, demonstrating that scaling laws hold for time series foundation models.

This paper is significant because it provides strong evidence that large-scale pretraining on diverse time series data can yield a single model capable of zero-shot forecasting across many domains. This could reduce the need for task-specific modeling and enable rapid deployment in new applications. The open-source release of the model and data also lowers the barrier for practitioners and researchers, fostering further innovation.

Technical Contributions

• Mixture-of-Experts Architecture: Time-MoE applies sparse MoE to time series, where each time step is routed to a subset of experts, enabling a large parameter count with constant compute per token.
• Scaling Laws for Time Series: The paper systematically studies how model size, data size, and number of experts affect performance, showing predictable improvements with scale.
• Large-Scale Pretraining Corpus: The authors curate a diverse dataset spanning multiple domains, which is crucial for learning generalizable representations.
• Zero-Shot Forecasting: The model is evaluated on unseen datasets without fine-tuning, demonstrating strong generalization.
• Open-Source Release: Code and pretrained models are made available, promoting reproducibility and further research.

Results

The paper reports that Time-MoE outperforms existing time series foundation models, such as TimesNet and PatchTST, in zero-shot forecasting tasks. For instance, on the popular benchmarks like ETT, Electricity, and Traffic, Time-MoE achieves lower mean squared error (MSE) and mean absolute error (MAE) compared to baselines. The scaling experiments show that increasing model size from 2M to 2.4B parameters leads to consistent error reduction, confirming the presence of scaling laws. Additionally, the model with more experts (e.g., 128 experts) yields better performance than a dense model with similar compute, highlighting the efficiency of the MoE design.

Significance

Time-MoE represents a major step toward general-purpose time series foundation models. By demonstrating that scaling laws apply and that MoE can make large models feasible, it opens the door for even larger and more capable models. This could lead to breakthroughs in few-shot and zero-shot forecasting, enabling AI systems to adapt to new time series tasks with minimal supervision. The work also encourages the community to explore efficient architectures for other sequential data modalities. As time series data is ubiquitous, the impact of this research could be widespread, from improving demand forecasting in supply chains to enhancing anomaly detection in industrial IoT.