Why Did the ECMWF Forecast Joaquin So Well?

The post-mortems have begun on how well Hurricane Joaquin was predicted, and one of the key themes is why the flagship global model of the European Centre for Medium-Range Weather Forecasts (ECMWF) beat NOAA’s Global Forecast System (GFS) to the punch in forecasting that Joaquin would remain well offshore. On Wednesday, September 30, less than six days from a potential landfall, the ECMWF operational model was consistently keeping Joaquin offshore, even as the GFS and nearly all other models were bringing the hurricane into the U.S. East Coast. From late Wednesday into Thursday, the GFS and other models began to shift toward an offshore track for Joaquin, as the hurricane itself was still diving southwestward into the Bahamas. By Friday, there was virtually unanimous model agreement on the offshore track that proved accurate.


Figure 1. Satellite image of atmospheric water vapor, collected at 1915Z (3:15 pm EDT) Friday, October 2, 2015. By this point, models agreed that Hurricane Joaquin would be moving out to sea, even though it was located unnervingly close to the southeast United States and moisture from Joaquin was already flowing onto the U.S. East Coast. Image credit: NOAA/NESDIS.

The National Hurricane Center did an admirable job of threading the needle in its public forecasts. NHC’s “cone of uncertainty” gradually edged toward the U.S. coast during the middle of last week. On Wednesday, the midpoint of the cone reached the mid-Atlantic coast, and the center’s forecast discussions acknowledged that they were splitting the difference between the insistently offshore ECMWF and other model guidance. The cone then began shifting back eastward on Thursday--but again, quite gradually, since the NHC works to avoid back-and-forth, “windshield-wiper” swings in projected track that could exacerbate public confusion. (One limit to the cone as currently designed is that the width of the cone is based on average track errors in recent years, rather than the actual uncertainty for a given hurricane, so in a case like Joaquin one wouldn’t have realized from a glance at the cone that the leading models differed so starkly.)


Figure 2. Official forecast track and cones of uncertainty issued by the National Hurricane Center at (left to right) 5 am and 5 pm EDT Wednesday, September 30, and 5 am and 5 pm EDT Thursday, October 1. Image credit: NHC.

It will take in-depth analysis to fully understand why the ECMWF was the first of the leading operational models to consistently keep Joaquin offshore. In 2012, the ECMWF model gained fame for correctly leading the pack in the opposite direction, as it was the first to call for Hurricane/Superstorm Sandy to hook into the New Jersey coast rather than remaining offshore. The fact that the ECMWF caught on to Sandy’s future ahead of the GFS model gained wide attention, and last week’s repeat victory for the ECMWF hasn’t escaped notice. On Friday, the New York Times published a summary of the computational, staffing, and design challenges that have hindered U.S. medium-range modeling efforts as compared to Europe’s.

Before jumping to conclusions...

Here are a few important things to keep in mind when contemplating the Battle of the Models:

--The GFS and ECMWF are both used to predict a vast array of weather events. These models have to be robust and durable in handling all kinds of atmospheric conditions in all seasons. Hurricane forecasting is an important task, but not their sole focus. Other models have been developed specifically for hurricane prediction, such as the hurricane-oriented version of the Weather Research and Forecasting model (HWRF) and the NOAA Geophysical and Fluid Dynamics Laboratory (GFDL) hurricane model. In 2014, HWRF led all other individual models in 2- and 3-day track forecasts (see Figure 3). For track guidance beyond 3 days, forecasters tend to put the most weight on the GFS and ECMWF, along with the UK Met Office model (UKMET).


Figure 3. Skill of computer model forecasts of Atlantic named storms in 2014, compared to a "no skill" model called "CLIPER5" that uses just climatology and persistence to make a hurricane track forecast (persistence means that a storm will tend to keep going in the direction it's current going.) OFCL=Official NHC forecast; GFS=Global Forecast System model; GFDL=Geophysical Fluid Dynamic Laboratory model; HWRF=Hurricane Weather Research Forecasting model; ECMWF=European Center for Medium Range Weather Forecasting model; UKMET=United Kingdom Met Office model; TVCA=one of the consensus models that blends together up to five of the above models; CMC=Canadian Meteorological Center (GEM) model; BAMM=Beta Advection Model (Medium depth). Data taken from the National Hurricane Center 2014 verification report.


--Individual high-profile events like Sandy and Joaquin can obscure the big picture. As Jeff Masters reported last week (scroll article to “Which track model should you trust?”, the ECMWF and GFS were virtually tied as the best-performing models for hurricane tracking when averaged over the period 2011-2014. As a guiding rule, the best forecasts emerge from a consensus that include a wide variety of models, rather than by going solely with the ECMWF or any other single model. This is why the NHC track forecasts lean heavily toward multi-model consensus, as was the case last week. Official NHC track predictions are often very close to the output from a model blend called TVCA, which employs the five models mentioned above (ECMWF, UKMET, GFS, GFDL, and HWRF). See our roundup post from August on recent improvements to the GFS and ECMWF as well as other leading models used for hurricane prediction.

--Overall the GFS and ECMWF are both very capable, sophisticated global models, and both are continually being revised and improved. Multiyear analyses tend to give the ECMWF a slight edge in overall performance across the range of weather features predicted. On average, the GFS lags the ECMWF model by about half a day in predicting upper-level weather features (i.e. centers of high and low pressure at the 500-millibar level, about 4 miles high). That margin has remained more or less constant over the last decade as both models have improved. The GFS now predicts 500-mb features out to 8 days with the same level of skill as 5-day forecasts of the 1980s. “Both models are world class models. I think that often gets lost in the crossfire,” said Marshall Shepherd in a Forbes essay published on Friday.


Figure 4. The number of days of useful skill produced by various models beyond 6 days (top lines) and 8 days (bottom lines) in predicting the height of the 500-millibar surface, which translates into the location of the upper-level highs and lows that shape surface weather. In recent years, the ECMWF has provided about 14.5 days of measurable skill, while the GFS has provided about 14 days. Image credit: NOAA/NWS Environmental Modeling Center, from “Improvements in Forecast Skill of the NCEP Production Suite,” Glenn White et al., presented in June 2015 at the 27th AMS Conference on Weather Anlysis and Forecasting.


--The ECMWF was created by 18 European nations in the 1970s with a very specific mission: “to pool Europe's meteorological resources to produce accurate climate data and medium-range forecasts.” A typical definition of “medium-range” is the period from 3 to 7 days, although ECMWF now produces a variety of forecast products out to 10 days and beyond. NOAA, in contrast, issues a much larger variety of forecasts and other products for a larger, more diverse customer base. The short-term High-Resolution Rapid Refresh (HRRR) model is one example of a recent NOAA innovation that falls outside the bailiwick of ECMWF. NOAA’s huge array of responsibilities limits the amount of resources it can devote to global forecast systems like the GFS, especially in an era with relentless pressure on U.S. science budgets. In the aftermath of Sandy, NOAA did receive supplemental funding that allowed for major progress, including a doubling of GFS horizontal resolution, implemented in January 2015, and a tenfold increase in supercomputing capacity being implemented this year. At the same time, ECMWF continues to upgrade its own models and computing resources, so the race continues.

One current weakness in the GFS relative to the ECMWF, noted in the New York Times article, is its technique for data assimilation (bringing as many observations as possible into the starting point of a model run). ECMWF employs a four-dimensional data assimilation technique, while the GFS uses a 3D technique. The fourth dimension is time: the 4D system allows data from satellites and other sensors to be woven into a model run over multiple time steps, rather than being injected into the model at a single time step. In this and several other ways, including the ability to draw on a wider range of observations, the ECMWF data assimilation appears to give it the edge. A 4D data assimilation system is now being developed for the GFS, perhaps to be incorporated within the next year.

Did cumulus clouds make the difference in Joaquin’s track forecasts?
Looking back at Hurricane Sandy, some interesting research led by Nicholas Bassill (University at Albany, State University of New York) points to one explanation for the ECMWF’s triumph with that storm: the way in which the model depicted (or parameterized) cumulus clouds that were too small to be modeled directly. Bassill found that when the WRF-ARW model was run using the ECMWF’s cumulus parameterization technique, it performed similarly to the ECMWF in bringing Sandy onshore. However, when the WRF-ARW was run with the GFS cumulus parameterization, it kept Sandy offshore (see Figure 5).

The GFS parameterizations have undergone some upgrades since 2012, and we can’t immediately tell for sure if this difference in cumulus treatment mattered in the case of Joaquin. Bassill does think that large-scale latent heat release within cumulus clouds (the energy released when water vapor condenses to form cloud droplets and raindrops) probably altered the evolution of important features upstream of the upper-level trough northeast of Joaquin that helped steer it away from shore. Also, he told me, “perhaps the ECMWF was better at assimilating satellite winds in the area to better capture this feature. You'd need further study to definitively say.”

Bob Henson


Figure 5. Results from nine 7-day forecasts extending from 1200 GMT October 23 to October 30, 2012, showing the variety of solutions for Hurricane Sandy that were obtained in research mode by running the same model (WRF-ARW) nine times. The only difference between each run is the method used for cumulus parameterization (handling the cumulus clouds that are too small to be directly simulated within the model). The parameterization techniques used for the NAM and GFS models led to an offshore track, whereas the technique used in the ECMWF model brought Sandy onto the northeast U.S. coast. Image credit: Nick Bassill.