ENSEMBLE and AMET: Two systems and approaches to a harmonized, simplified and efficient facility for air quality models development and evaluation
Introduction
In atmospheric dispersion and air quality modeling, the evaluation of the model performance has long been a topic of research activities and establishing best practices (e.g. Oreskes et al., 1994, Steyn and Galmarini, 2008). Before we tackle the model evaluation topic per-se, it is probably appropriate to give a couple of definitions. We will refer here to atmospheric dispersion models as the models that simulate the dispersion of a passive, decaying or reactive species released from a point source, be it a stack or a limited-area source. No limit exists to the scale of application which can extend as far as global. These are the models normally used for emergency preparedness and response applications. With air quality model we refer however to a model which deals with distributed sources of various chemical precursors and that treats the chemical transformations occurring in a volume of air that can range from the meso- to the global-scale in a three-dimensional domain. In the two cases the dynamic fields are acquired from meteorological models. The final application of these models is in the realm of support to air quality policy planning and regulation.
The verification of the model capability to adhere to experimental evidence has occupied atmospheric scientists in recent decades. The complexity of the atmospheric system in terms of spatial and temporal variability, stochasticity and scales, makes the task of evaluating a model particularly complicated. The collection of experimental evidence, representative of space and time scales, which could be used for evaluation of a time- and space-averaged model result, has always constituted the first burden in the practice of model evaluation. To date, operational networks of instruments provide organized, quality-checked information useful in this context for both atmospheric dispersion and air quality models. Routine monitoring data however are often insufficient to assess the performance of a model in depth, as they may only provide evidence of the general correspondence of model results with surface-based point measurements that are typical in such network data. These types of evaluations do not provide clear indications of the veracity of the modeled-chain of processes that leads to the model result. In other words, using a common expression, we would not know whether the right model results were obtained for the right reasons.
The problem becomes particularly complicated in the case of predictions where data are not sufficiently available. This is the case, for example, in atmospheric dispersion modeling or scenario studies in air quality. In such cases as well as in the case of a total absence of experimental evidence, a reasonable alternative to model evaluation is model inter-comparison, meaning that models are applied to a common case study and inter-compared. The relative differences in model results are useful indications of uncertainties in modeling science or data that need attention.
The Joint Research Center of the European Commission and the US-EPA over the past two decades have played an important role in promoting model evaluation and bringing together the European, US and international model communities in promoting best practices in model evaluation. These activities were intended to develop and promote standards in model evaluation (e.g. Gamarini et al., 2010). Over the last decades this practice in particular has been extremely important in bringing together communities of model developers and users and has led to very important developments beyond the singular practice of model evaluation.
Given the complexity of air quality modelling systems and diversity of information necessary to run them and to evaluate them, the availability of tools that could assist modelers in the latter is very important. On the one hand one wants to guarantee that model data produced by different sources are handled in the same way. At the same time one should consider that diverse monitoring information is assembled, organized and harmonized so that all users could use the same source of information efficiently. In this context diversity pertains not only to the nature of the information (meteorology, micro-meteorology, atmospheric chemistry), but also to the fact that same atmospheric species could be measured differently by different networks. In parallel when evaluating a model one would like to use similar metrics and a standard set of tools so that all model performances are assessed against the same ruler and terms of comparison.
Toward these ends, information technology has provided us with useful means for an easier, consistent, standardized and practical way of evaluating models. Over the last decade the European Commission Joint Research Center and US-EPA have independently invested resources in the development of two distinct systems for model evaluation, namely, the ENSEMBLE system (Galmarini et al., 2004a, Galmarini et al., 2004b) and AMET (Appel et al., 2011). While they both work toward the same goals they are different in philosophy and approaches and cover, in a complementary way, different aspects of the problem.
Similar systems exist for more specialized applications such as AeroCom (Textor et al., 2006; http://nansen.ipsl.jussieu.fr/AEROCOM), which provides an online access to consultation of global model predictions observations of various kinds relating to aerosols and aerosols components. Off line model evaluation tools were developed over the years on the footprints of AMET such as BOOT (Chang, 2002; Chang and Hanna, 2004) specialized in atmospheric dispersion models and the CITY-DELTA tool still from the Joint Research Center (Thunis et al., 2008) specialized in the assessment of differences in air quality scenarios at the mesoscale models.
ENSEMBLE and AMET have been used extensively for the AQMEII activity and without them a project of this scale could not happen in the time scale in which it was performed. In this paper the two independent systems are presented in terms of their usefulness for both model evaluation and inter-comparison application, including their future developments and synergies. The perspective presented is that of the model developer/user rather than the information technology side to give an idea on the amount of resources and rapid advancements to the atmospheric science discipline such systems can provide. A model evaluation framework adopted by the AQMEII activity (Dennis et al., 2010) that considers four types of evaluation: operational, diagnostic, dynamic, and probabilistic. Operational is intended as direct comparison of model results with observations which may lead to assessment pass or fail; diagnostic is intended as investigation on the reasons for a pass or a fail; diagnostic would lead to determining the sensitivity of model results to input data, whereas the probabilistic evaluation aims at providing an assessment of confidence in model results and uncertainty. We will show that the two system serve all these type of evaluation thus allowing a thorough and complete analysis of model performance.
Section snippets
Evaluating models through a web-distributed service: the ENSEMBLE system
ENSEMBLE was developed in 2000 as a system for the real-time acquisition and consultation of the results of several models produced by different groups on the same case study. The field of application was long range atmospheric dispersion forecasting. Details on that application can be found in Galmarini et al. (2001), Bianconi et al., 2004, Galmarini et al., 2004a, Galmarini et al., 2004b, Potempski et al., 2008, Galmarini et al., 2008.
Fig. 1 shows schematics of the concept behind the ENSEMBLE
Evaluating models through a standalone service: the AMET system
The Atmospheric Model Evaluation Tool (AMET; Appel et al., 2011) was designed to aid in evaluation of meteorological and air quality model output, and has been utilized by a number of the AQMEII modeling groups to aid in the evaluation of their model output.
The AMET software is a collection of several software packages integrated together to provide a system for storing and analyzing meteorological and air quality observations and model output. AMET uses FORTRAN and Perl code to pair model
Conclusions
Over the past couple of decades a number of modeling consortia have been established where a large number of institutions have come together with the scope of comparing the results of their models with others in the scientific community. It is from discussions within these consortia that activities of such nature can only be beneficial to promoting progress in modeling science and application, in particular in the case of air quality modeling. The progress produced by these communities in terms
Acknowledgements
The authors would like to thank Dr C. Hogrefe (Bureau of Air Quality Analysis and Research, New York State Department of Environmental Conservation) for his constant support and advice on the collection and harmonization of monitoring information and Drs R. Gilliam (US-EPA) and J. Zhang (Environment Canada) for their contribution in data preparation. The authors also acknowledge for the strong support of the European Commission, Airbus, and the Airlines (Lufthansa, Austrian, Air France) who
References (20)
- et al.
Overview of the Atmospheric Model Evaluation Tool (AMET) v1.1 for evaluating meteorological and air quality models
Environ. Modell. Soft.
(2011) - et al.
The AQMEII two-continent regional air quality model evaluation study: fueling ideas with unprecedented data
Atmos. Environ.
(2011) - et al.
Forecasting consequences of accidental releases from ensemble dispersion modelling
J. Environ. Radioact.
(2001) - et al.
Ensemble dispersion forecasting, Part II: application and evaluation
Atmos. Environ.
(2004) - et al.
Ensemble dispersion forecasting, Part I: concept, approach and Indicators
Atmos. Environ.
(2004) - et al.
Real-time monitoring data for real-time multi-model validation: coupling ENSEMBLE and EURDEP
J. Environ. Radioact.
(2008) - et al.
Multi-model ensemble analysis of the ETEX-2 experiment Atmos
Environments
(2008) - et al.
Giovanni: a Web service workflow-based data visualization and analysis system
IEEE Trans. Geosci. Remote Sens.
(2009) - et al.
A WWW-based decision support system for the management of accidental releases of radionuclides in the atmosphere
Environ. Modell. Soft.
(2004) - Chang, J. C., 2002. Methodologies for evaluating performance and assessing Uncertainty of atmospheric dispersion...
Cited by (53)
Joint Nordic nuclear research to strengthen nuclear emergency preparedness after the Fukushima accident
2018, Journal of Environmental RadioactivityCitation Excerpt :Among other things the group performs real-time exercises on atmospheric dispersion of radioactive matter. It maintains liaison with relevant international activities, including the EU ENSEMBLE platform (Galmarini et al., 2012) that can collect dispersion forecasts provided by more than 20 operational models. On the research and development level, Nordic organisations naturally also participate in wider international networks in European collaborative research and development projects, particularly under the EURATOM framework.
Simulation of European air quality by WRF-CMAQ models using AQMEII-2 infrastructure
2016, Journal of Computational and Applied MathematicsTheoretical Predictive Air Quality Models
2016, Comprehensive Analytical ChemistryFeedbacks between air pollution and weather, Part 1: Effects on weather
2015, Atmospheric Environment