Introduction

The ECMWF, European Centre for Medium-Range Weather Forecasts, is running an experimental research model which is exploring machine learning (ML) and artificial intelligence (AI) techniques.

Note that I am not a researcher in machine learning, artificial intellegence, or a meteorologist. All opinions expressed here are those of a layman in this field.

For anybody who has been paying attention, you will have noticed that AI and ML have been improving at an incredible rate over the last decade. If you have spent any time chatting with one of the large language models (LLM’s) and exploring their knowledge, its hard not to be astounded at their capabilities. Another shocking fact is that, the LLM’s and other ML models we are experiencing now are the worst that we will ever encounter. The LLM’s and ML models will continue to improve and evolve in the future.

Several teams have been experimenting with applying the ML techniques, and their ability to capture an understanding of patterns and how they evolve, to the Numerical Weather Prediction (NWP) area. For example, Google published a paper describing GraphCast, their version of a ML model applied to weather prediction, in November 2023. In early December 2024, Google published a new paper discussing GenCast, which applies ML to ensemble weather prediction models. This paper was noticed by several of my customers who contacted me (Craig) asking if they could gain access to this model.

The tricky piece is that a research group, such as the one at Google, developing an AI model for weather prediction is not what I need in order to provide data to the community. What was needed is for one of the large operating centers to dedicate some of their resources to exploring this area and commit to producing data, available to the public, on some regular schedule. This is what ECMWF had been doing without my realizing it.

Toward late 2024 I ran across a model on the ECMWF open data site named AIFS. AIFS is the product of a group working within ECMWF on applying ML and AI to weather prediction. The AIFS model is being used to create forecast data, four times a day, with forecasts every 6 hours out to 15 days.

Deterministic vs Machine Learning

For the vast majority of Numerical Weather Prediction models produced to date, what are called Deterministic models, the model consists of two major parts:

1) trying to capture the initial atmospheric conditions in the most accurate way possible. This is needed as a starting point from which to create the forecasts. This is done by ingesting as many observations of the atmosphere, via weather stations, buoys, satellites, commercial airplanes, weather baloons, etc, and converting these observations into a coherent picture of the atmosphere, at all of its levels. (This is a very complex, perhaps underappreciated, task!)

2) modeling the physics of how the atmosphere and all of its different motions, temperatures, composition (air, rain, ice, etc) and radiation interact to influence the current conditions in an area, or at a point, to produce a forecast for what would happen shortly in the future.

The physics in these deterministic models is necessarily an approximation of what actually happens in the real atmosphere. Having said that, these simulations have evolved into remarkably useful predictions, forecasts, of future weather conditions.

A simplistic view of running a deterministic model would be to start with the initial conditions and apply the physics to make a prediction a few seconds in the future.
Repeat this process, advancing from one prediction to the next. Every now and then, perhaps hourly, output the state of the prediction to create the forecast data.

The Machine Learning approach is not physics based at all (or if it is, it approaches the physics in a completely different manner.) I’ve read experts in the field compare ML to compression. When a ML model is trained on the weather data, it is learning how the physics works and compressing (expressing?) that knowledge into the ML weights being created. I believe you can think of the AIFS model as having learned how the physics in weather system evolution works. This is a very coarse view of what is happening. The model is also learning details about the physics of local areas, perhaps influenced by mountains, coastlines, etc.

While training a large ML model is enormously compute intensive, once a ML model, such as AIFS, has been created, it can be run, which is a process called inference, very efficiently.

I imagine that running AIFS would be similar to how a deterministic model is run: start with the initial conditions and rather than applying the physics package to advance to the next prediction, use the ML model. The AIFS ML model replaces the physics package.

While the ECMWF IFS (deterministic) model take many hours to run on large super computers, the AIFS model can be run in several minutes on much more modest hardware. The difference between these two generation times is exciting, and I look forward to seeing what creative ways the operational centers will find to use the computing resources that may eventually be freed up with this approach.

AIFS

AIFS was first announced in October 2023. In that initial announcement, AIFS was producing data at a 1 degree resolution. AIFS was updated in January 2024. The upgrade announcement is worth reading. Two notable points, the first being that the data resolution is now 0.25 degrees. The second, among many, interesting points is that they mention that this second version of AIFS has moved on from the original Google GraphCast ML techniques toward variants which they find are working better for weather prediction. There are several charts showing comparisons of the skill level between AIFS, its earlier version, IFS, and GraphCast where the new AIFS is more skillful than the others. ML appears to be a very promising approach for NWP.

For additional articles and detail on the AIFS model, please read the articles published on the ECMWF AIFS blog.

The AIFS team have also created a paper linked off of this page.

It appears that AIFS may become the preferred model for forecasts which extend further into the future. AIFS appears to be more skillful than the others at the 15 day mark, and earlier.

AIFS data and LuckGrib

AIFS is being made available through LuckGrib, with the model appearing in the Global / Research category. As ECMWF is treating this as a research project, they do not promise the same level of reliability as they do with their operational models. i.e. the data may suddenly not appear for several days - this is a risk when using non-operational data. I’m confident that ECMWF will make an attempt to provide reliable data delivery, as they have been doing.

Also, there are fewer parameters availble in the AIFS than in the deterministic ECMWF model. In particular, wind gust is currently not available, along with the heights of the various pressure levels (500mb and 250mb.)

Thanks!

Thank you to ECMWF and the AIFS team for making their data available in this early release phase. It will be interesting to watch and learn how this approach to numerical weather prediction evolves.

License terms

This data is licensed in the same manner as with the ECMWF data. Please read that page for details.