Model evaluation and ensemble modelling of surface-level ozone in Europe and North America in the context of AQMEII
Introduction
Regional air quality (AQ) models have undergone considerable development over the past three decades, mainly driven by the increased concern regarding the impact of air pollution on human health and ecosystems (Rao et al., 2011). This is particularly true for ozone and particulate matter (e.g., Holloway et al., 2003, Jacob and Winner, 2009). Regional AQ models are now widely used for supporting emissions control policy formulation, testing the efficacy of abatement strategies, performing real-time AQ forecasts, and evaluating integrated monitoring strategies. Moreover, ozone estimates have been used in assimilation schemes to provide further information on meteorological variables such as wind speed (e.g., Eskes, 2003). The combination of outcomes predicted by several models (regardless of their field of application), in what is typically defined as ensemble modelling, has been shown to enhance skill when compared against an individual model realisation (e.g., Delle Monache and Stull, 2003, Galmarini et al., 2004, van Loon et al., 2007). Although ensemble modelling is well established (both from the applied and theoretical perspectives) and is now routinely used in weather forecasting, it is only during the last decade that a growing number of AQ modelling communities have joined their model outputs in multi-model (MM) combinations (Galmarini et al., 2001, Carmichael et al., 2003, Rao et al., 2011). The advantages of ensemble modelling versus an individual model are at least twofold: (i) the mean (or median) of the ensemble is, in effect, a new model that is expected to lower the error of the individual members due to mutual cancellation of errors; and (ii) the spread of the ensemble represents a measure of the variability of the model predictions (Galmarini et al., 2004, Mallet and Sportisse, 2006, Vautard et al., 2006, Vautard et al., 2009, van Loon et al., 2007). Potempski and Galmarini (2009) also point out the scientific consensus around MM ensemble techniques as a way of extracting information from many sources and synthetically assessing their variability. In particular, the mean and median offer enhanced performance, on average, compared with single-model (SM) realisations (Delle Monache and Stull, 2003, Galmarini et al., 2004, McKeen et al., 2005, and others).
A MM ensemble can be generated in many ways (see, e.g., Galmarini et al., 2004), including by varying some internal parameters for multiple simulations with an SM, by using different input data (e.g., emissions) for multiple simulations with an SM, or by applying several different models to the same scenario. This latter approach is the main focus of the Air Quality Model Evaluation International Initiative (AQMEII) (Rao et al., 2011), an international project aimed at joining the knowledge and experiences of AQ modelling groups in Europe and North America. Within AQMEII, standardised modelling outputs have been shared on the web-distributed ENSEMBLE system, which allows statistical and ensemble analyses to be performed by multiple groups (Bianconi et al., 2004, Galmarini et al., 2012). A joint exercise was launched for European and North American AQ modelling communities to use their own regional AQ models to simulate the entire year 2006 for the continents of Europe and North America, retrospectively. Outputs from numerous regional AQ models have been submitted in the form of both gridded, hourly concentration fields and values at specific locations, allowing for direct comparison with air quality measurements available from monitoring networks across North America and Europe (see Rao et al., 2011 for additional details). This type of evaluation, with large temporal and spatial scales, is essential to assess model performance and identify model deficiencies (Dennis et al., 2010, Rao et al., 2011).
In this study, we analyse ozone mixing ratios provided by simulations from eleven state-of-the-art regional AQ models run by eighteen independent groups from North America (NA) and Europe (EU) (while a companion study is devoted to the examination of particulate matter, Solazzo et al., 2012). Model predictions have been made available, along with observational data, to the ENSEMBLE system. The ability of the ensemble mean and median to reduce the error and bias of SM outputs is examined, and conclusions regarding the size of the ensemble and its quality are made. The level of repetition provided by each individual model to the ensemble is quantified by applying a clustering analysis to examine whether the improvement in error using the mean or median of the model ensemble is due to the increased ensemble size, or if information carried by each model contributes to the MM superiority.
The main objective of this study is to assess the statistical properties of the ensemble of models in relation to the individual model realisations for a range of air quality cases. Each model has imperfections, and it is beyond the scope of this analysis to identify the causes of model bias for each ensemble member. Several other papers examining the performance of the individual model simulation are available in the AQMEII special issue.
Section snippets
Experimental set up
In order to carry out a comprehensive evaluation of the participating regional-scale AQ models, the model estimates are compared to observations for the year of 2006, with the various modelling groups providing hourly ozone mixing ratios and concentrations of other compounds. Surface concentrations were then interpolated to the monitoring locations for the purposes of model evaluation.
Participating models
Table 1 summarises the meteorological and AQ models participating in the AQMEII intercomparison exercise and
Operational SM and ensemble statistics for the continental-wide domains
van Loon et al. (2007) showed that the ensemble mean ozone daily cycle over EU, obtained by averaging over all monitoring stations for the entire year of 2001, agrees almost perfectly with the observations, and better than any individual member of the ensemble. This result provides substantial evidence of the enhanced skill of MM predictions versus the individual SM predictions. Such a result, though, while encouraging, poses some additional questions, such as what is the role of repeated
Ensemble size
In this section we evaluate whether an ensemble built with a subset of individual models can outperform the ensemble mean of all available members, as anticipated by the theoretical analysis of Potempski and Galmarini (2009). The analysis is done for the sub-regions of EU and NA separately, using hourly ozone data for the period JJA.
Consider the distribution of some statistical measures (RMSE, PCC, MB, MGE, defined in Appendix A) of the mean of all possible combinations of available ensemble
Reduction of data complexity: a clustering approach
Results discussed in the previous section have shown that a skilful ensemble is built with an optimal number of members and often includes low-ranking skill-score members as well. In order to discern which members should be included in the ensemble, a method for clustering highly associated models and then discarding redundant information was developed using the PCC as the determining metric (we note that PCC is independent of model bias; therefore, the analysis would be the same for unbiased
Conclusions
This study collectively evaluates and analyses the performance of eleven regional AQ models and their ensembles in the context of the AQMEII inter-comparison exercise. The scale of the exercise is unprecedented, with two continent-wide domains being modelled for a full year. The focus of this study was on the collective analysis of surface ozone mixing ratios, rather than on inter-comparing metrics for each individual model. The study began with an analysis of ozone time series for sub-regions
Acknowledgments
The work carried out with the DEHM model was supported by The Danish Strategic Research Program under contract no 2104-06-0027 (CEEH). Homepage: www.ceeh.dk.
References (57)
- et al.
Overview of the atmospheric model evaluation tool (AMET) v1.1 for evaluating meteorological and air quality models
Environmental Modelling & Software
(2011) - et al.
Aerosol modeling with CHIMERE: preliminary evaluation at the continental scale
Atmospheric Environment
(2004) - et al.
Web-based system for decision support in case of emergency: ensemble modelling of long-range atmospheric dispersion of radionuclides
Environmental Modelling and Software
(2004) - et al.
An ensemble of regional ozone model uncertainty with an explicit error representation
Atmospheric Environment
(2011) - et al.
An integrated model study for Europe and North America using the Danish Eulerian hemispheric model with focus on intercontinental transport of air pollution
Atmospheric Environment
(2012) - et al.
The effects of meteorology on ozone in urban areas and their use in assessing ozone trends
Atmospheric Environment
(2007) - et al.
Measurements of sulfur dioxide, ozone and ammonia concentrations in Asia, Africa, and South America using passive samplers
Atmospheric Environment
(2003) - et al.
An ensemble air quality forecast over western Europe during an ozone episode
Atmospheric Environment
(2003) - et al.
A characterization of the spatiotemporal variability of non-urban ozone concentrations over the eastern United States
Atmospheric Environment
(1993) - et al.
Forecasting the consequences of accidental releases of radionuclides in the atmosphere from ensemble dispersion modelling
Journal of Environmental Radioactivity
(2001)
Ensemble dispersion forecasting. Part I: concept, approach and indicators
Atmospheric Environment
ENSEMBLE and AMET: two systems and approaches to a harmonised, simplified and efficient assistance to air quality model developments and evaluation
Atmospheric Environment
Cloud processing of gases and aerosols in a regional air quality model (AURAMS)
Atmospheric Research
Natural volatile organic compound emission rate estimates for US woodland landscapes
Atmospheric Environment
Diagnostic analysis of ozone concentrations simulated by two regional-scale air quality models
Atmospheric Environment
Effect of climate change on air quality
Atmospheric Environment
Modeling Europe with CAMx for the Air Quality Model Evaluation International Initiative (AQMEII)
Atmospheric Environment
Modelling the formation and atmospheric transport of secondary inorganic aerosols with special attention to regions with high ammonia emissions
Atmospheric Environment
NARSTO critical review of photochemical models and modelling
Atmospheric Environment
Impact of biogenic emissions on air quality over Europe and North America
Atmospheric Environment
A comparison of simulated and observed ozone mixing ratios for the summer of 1998 in western Europe
Atmospheric Environment
A comparative performance evaluation of the AURAMS and CMAQ air quality modelling systems
Atmospheric Environment
A dispersion modeling system SILAM and its evaluation against ETEX data
Atmospheric Environment
Operational model evaluation for particulate matter in Europe and North America in the context of the AQMEII project
Submitted for publication to Atmospheric Environment
Evaluation of long-term ozone simulations from seven regional air quality models and their ensemble average
Atmospheric Environment
Skill and uncertainty of a regional air quality model ensemble
Atmospheric Environment
Evaluation of the meteorological forcing used for AQMEII air quality simulations
Atmospheric Environment
Examination of the Community Multiscale Air Quality (CMAQ) model performance for North America and Europe for the AQMEII project
Atmospheric Environment
Cited by (153)
Dynamic evaluation of modeled ozone concentrations in Germany with four chemistry transport models
2024, Science of the Total EnvironmentAn updated model-ready emission inventory for Guangdong Province by incorporating big data and mapping onto multiple chemical mechanisms
2021, Science of the Total EnvironmentLong-range transport of Siberian biomass burning emissions to North America during FIREX-AQ
2021, Atmospheric EnvironmentHigh spatial resolution WRF-Chem model over Asia: Physics and chemistry evaluation
2021, Atmospheric Environment