RESULTS FROM IDEALIZED TESTING OF MACHINE LEARNING ATMOSPHERE MODELS

Abstract

Artificial Intelligence (AI)-based weather models are becoming an increasingly important part of the weather and climate ecosystem due to the cheap and skillful forecasts they generate. There are also many promising applications of these models for general scientific discovery. While all AI weather models report performance in global forecasting metrics, few have so far been subjected to the degree of scrutiny applied to traditional “dynamical” models. By replicating and expanding on the results of Hakim and Masanam (2024), we demonstrate that a set of popular AI weather models’ responses to idealized tests range in physical realism from “qualitatively correct” to “physically implausible”, depending on the test. For example, all models appear to evolve towards a state of geostrophic balance when provided with an upper-level low lacking its corresponding cyclonic wind field, but most of them also produce one or more physically unrealistic features during a series of idealized tests intended to characterize each model’s ability to represent realistic wave growth in response to either an initial or recurrent perturbation. Finally, we observe that the Earth-centric designs of the Nvidia and Pangu models appear to result in fewer grid artifacts during idealized tests, and that the single-model inference pipeline of the Nvidia models allows these tests to be performed more simply.