AIFS-CRPS: ensemble forecasting using a model trained with a loss function based on the continuous ranked probability score

Analysis

Why This Paper Matters

This paper marks a significant milestone in the application of machine learning to operational weather forecasting. ECMWF's AIFS-CRPS is one of the first ML-based ensemble systems to rival and even surpass a state-of-the-art physics-based ensemble (IFS) across a broad range of variables and lead times. The introduction of the almost fair CRPS loss function addresses a critical challenge in training probabilistic models: the bias introduced by finite ensemble sizes. By providing a practical solution, this work paves the way for more reliable and efficient ensemble forecasting, which is essential for decision-making in sectors like agriculture, energy, and disaster management.

The paper also demonstrates that ML models can be stochastic and generate exchangeable ensemble members, a property that is crucial for capturing forecast uncertainty. This is a departure from deterministic ML weather models and highlights the potential for ML to handle probabilistic prediction natively. The competitive performance at subseasonal timescales, albeit after bias correction, suggests that ML models are not limited to short-range forecasts and could eventually extend the skill of subseasonal-to-seasonal predictions.

Technical Contributions

• Almost Fair CRPS (afCRPS) Loss: The core innovation is a loss function that approximates the fair CRPS, which corrects for the bias due to finite ensemble size, while avoiding the degeneracy of the fair CRPS. This allows the model to be trained directly on ensemble forecasts without requiring a large number of members.
• Stochastic Ensemble Generation: The trained model is stochastic, meaning it can produce an arbitrary number of exchangeable ensemble members. This is achieved by injecting noise into the model, enabling the generation of a full probability distribution.
• Integration with AIFS: The method is built on the existing AIFS architecture, demonstrating that a deterministic ML model can be adapted for probabilistic forecasting with minimal changes.
• Comprehensive Evaluation: The paper provides a thorough comparison against the IFS ensemble across multiple variables and lead times, including both medium-range and subseasonal forecasts, and considers both raw and anomaly-based metrics.

Results

The paper reports that AIFS-CRPS outperforms the IFS ensemble for the majority of variables and lead times in medium-range forecasts (up to 15 days). For subseasonal forecasts (weeks 3-6), AIFS-CRPS outperforms the IFS ensemble before calibration and is competitive when forecasts are evaluated as anomalies to remove model biases. The exact metrics are not provided in the abstract, but the qualitative claims are strong. The use of the afCRPS loss is shown to be effective in training a stochastic ensemble model that produces reliable and skillful probabilistic forecasts.

Significance

This work has profound implications for the field of AI and weather forecasting. It demonstrates that ML models can not only match but exceed the performance of traditional physics-based models in ensemble forecasting, which is a cornerstone of modern weather prediction. The afCRPS loss function is a methodological contribution that could be applied to other probabilistic forecasting tasks beyond meteorology, such as finance or energy. The success of AIFS-CRPS suggests that operational weather centers may increasingly rely on ML-based ensembles, leading to faster and more cost-effective forecasts. This could democratize access to high-quality probabilistic weather information, benefiting a wide range of users.