In-context fine-tuning for time-series foundation models

Analysis

Why This Paper Matters

Time-series forecasting is critical across industries, but traditional models often require extensive retraining for new domains. Foundation models for time-series have emerged, yet they typically operate in zero-shot mode, limiting their adaptability. This paper addresses that gap by introducing in-context fine-tuning, a method that allows a pre-trained model to adapt to new data using only context examples, without updating weights. This is significant because it bridges the gap between generic pre-training and task-specific adaptation, making foundation models more versatile and accessible.

The approach is inspired by in-context learning in large language models, where models use examples in the prompt to infer the task. Applying this to time-series is non-trivial due to the continuous nature of the data and the need to capture temporal dependencies. The paper's contribution is a practical mechanism to inject context into time-series foundation models, enabling them to adjust their predictions based on recent or similar patterns. This could democratize access to high-quality forecasting for niche domains where labeled data is scarce.

Technical Contributions

• Introduces a context encoding mechanism that converts a set of time-series examples into a fixed-size representation, which is then fed into the foundation model as additional input.
• Proposes a training strategy that simulates in-context fine-tuning during pre-training, ensuring the model learns to leverage context effectively.
• Demonstrates that the method is parameter-efficient, requiring no gradient updates at inference time, and can be applied to any existing time-series foundation model.
• Provides a comprehensive evaluation against statistical baselines (e.g., ARIMA, ETS) and other foundation models (e.g., TimesNet, PatchTST) across multiple datasets.

Results

The paper reports that in-context fine-tuning improves forecasting accuracy by up to 20% over zero-shot baselines and outperforms statistical models by a significant margin. For example, on the Electricity dataset, the method achieves a 15% reduction in MSE compared to the best baseline. The gains are consistent across different horizons and data frequencies, and the method shows robustness to noisy context examples. The authors also highlight that the method is particularly effective in few-shot scenarios, where only a handful of examples are available.

Significance

This work has the potential to shift how time-series foundation models are deployed in practice. By enabling in-context adaptation, it reduces the need for fine-tuning on each new dataset, saving computational resources and time. It also opens up new research directions, such as meta-learning for time-series and context-aware forecasting. The approach could be extended to other sequence modeling tasks, such as anomaly detection or classification, where context plays a crucial role. Overall, this paper contributes a simple yet powerful technique that enhances the flexibility of foundation models, making them more useful for real-world applications.