Annual and seasonal mean tropical and subtropical precipitation bias in CMIP5 and CMIP6 models

The fully coupled run output from historical simulation in CMIP5 (1850–2005) (Taylor et al 2012) and CMIP6 (1850–2014) (Eyring et al 2016) is used. The historical simulation uses observed 20th century greenhouse gas, ozone, aerosol, and solar forcing. Output from CMIP5 and CMIP6 simulations, for which the ensemble member designated r1i1p1 or r1ilp2 (https://pcmdi.llnl.gov/mips/cmip5/docs/cmip5_data_reference_syntax_v0-27_clean.pdf?id=99), (under the heading 'Ensemble member') in CMIP5 and in CMIP6 (in which case it is designated r1i1p1f1 or r1i1p2f1) is available with all necessary fields for 1980 to the end of simulations, is used to produce annual means from monthly data. We use 28 CMIP5 models (see SI table S1) as a single group. Output of 23 CMIP6 models is currently available (see SI table S1) but they are divided into NOS and SON groups. The multi-model means of all CMIP5 (N = 28) and CMIP6 (N = 23) models are denoted as CM5 and CM6, respectively and they include both SON and NOS models. Although five CMIP5 models consider FIREs (CESM1-CAM5, CSIRO-Mk3-6-0, HadGEM2-AO, HadGEM2-CC and HadGEM2-ES), we consider this sample to be too small to subset so only separate CMIP6 into SON (N = 11) and NOS (N = 12).

We further split SON (see SI table S2) into two subgroups according to treatments of ice-cloud radiative properties: SON1 follows the UK Met Office global climate model (HadGEM3-GA7.1) using combined cloud ice and snow contents to calculate ice-cloud radiative properties, which we refer to as GA71 (Walter et al 2019, Williams et al 2018, Mulcahy et al 2018) and SON2 follows the National Center for Atmospheric Research CESM2-CAM6, using separate cloud ice and snow contents to calculate ice-cloud radiative properties (Gettelman et al 2010, Gettelman and Morrison 2015). Therefore, for CMIP6 where there are a sufficient number of models of each subtype we separately compare SON versus NOS, and SON1 versus SON2.

The atmospheric component (HadGEM3-GA7.1) used in the HadGEM3-GC3.1 earth system model (Walters 2019, Williams 2018, Mulcahy et al 2018) is also included in ACCESS-CM2, EC-Earth3-Veg, HadGEM3-GCM31-LL, SAM0-UNICON and UKESM1-0-LL. The cloud microphysics scheme is based on Wilson and Ballard (1999) with extensive modifications (Abel and Boutle 2012). We refer to this approach to handling cloud ice as GA71. In GA71, the scheme has only one type of prognostic frozen water, including cloud ice, aggregated snow and rimed particles. Ice is allowed to fall from layer to layer. The prognostic cloud scheme predicts ice cloud fraction (Wilson et al 2008) and the total ice content is used to calculate ice-cloud radiative properties with a single effective ice diameter.

The CESM2 atmospheric component, CAM6, contains a two-moment prognostic cloud microphysics scheme (MG2, Gettelman and Morrison 2015). The major innovation in MG2 is to carry prognostic precipitation species—falling liquid (rain) and falling ice (snow)—in addition to cloud condensates (cloud ice and cloud liquid). The SON2 subgroup includes CESM2, CESM2-WACCM, CESM2-WACCM-FV2, NorESM2-MM and E3SM-1-0 with MG2 and TaiESM with a previous generation (MG1) cloud microphysics scheme (Gettelman 2010) with diagnostic falling ice (snow). The difference between SON1 (one prognostic frozen water) and SON2 models is that SON2 models use separate ice and falling ice contents with different effective ice diameters to calculate ice-cloud radiative properties. The major distinction in treating ice cloud–radiation interactions between the NOS and SON models is that in the NOS models the falling cloud ice is not radiatively active.

The GPCP Version 2.3 data set combines precipitation observations over land and ocean from low-orbit satellite microwave data, geosynchronous-orbit satellite infrared data and, over land, surface rain gauge observations to provide gridded monthly precipitation from January 1979 to May 2019 (Adler et al 2003, 2018). The data for January 1980 to December 2014 are used in this study and we take the mean of the monthly time series for each grid cell to represent our observational estimate of precipitation climatology.

Both the GCM and observational data sets are re-gridded onto a common horizontal resolution of 2° latitude by 2° longitude for consistent model–model and model–data comparisons; model grid spacings vary from less than 0.5° to greater than 2° and direct comparison would not be possible without re-gridding. For CMIP5 and any comparisons between CMIP5 and another dataset, the average precipitation map for 1980–2005 is used. For comparisons involving CMIP6 we use the 1980–2014 mean.

We report four area-weighted spatial statistics in this study, all of which are calculated from two-dimensional multi-year-mean precipitation fields P from models, ${{{P}}_{\text{m}}}$, and GPCP ${{{P}}_{\text{o}}}$:

• (a)  
  The mean bias, $\overline {{{{P}}_{\text{m}}} - {{{P}}_{\text{o}}}}$, which is the model area-weighted mean precipitation minus the observational estimate of the same.
• (b)  
  The mean absolute bias, $\overline {\left| {{{{P}}_{\text{m}}} - {{{P}}_{\text{o}}}} \right|}$, which is calculated as the area-weighted mean of the absolute differences between the model value and the observed value at each grid cell.
• (c)  
  The area weighted standard deviation of P, which we label σ_spatial.
• (d)  
  The Pearson correlation coefficient between two maps, r_spatial.

The σ_spatial value is derived from a single field, while the other statistics are derived from a pair of maps, e.g. between a model and GPCP.

In this section we quantify and synthesize the comparative information among individual CMIP6 models and model groups. We use a Taylor diagram (Taylor 2001) which relates statistical measures of model fidelity among the CMIP6 models and for the subgroups of NOS, SON, SON1, SON2, CM6 and CM5: the spatial correlation r_spatial and the spatial standard deviation σ_spatial, normalized by the reference σ_spatial (i.e. that of the observational target, in this case GPCP). The reference point representing observations lies at (1.0,1.0) expressed in terms of (σ_spatial, r_spatial), since by definition it matches its own magnitude and pattern. The distance from the reference point to any other point is proportional to the centered root-mean-square bias (RMSB), which is RMSB normalized by the observational σ_spatial. These statistics are calculated using the 1980–2014 (except CM5 for 1980–2005) average over the study domain, which covers the tropical and subtropical Pacific and Atlantic and is limited to the latitude belt of 40°N–40°S from 240°W (120°E) to 0°W excluding land grid points. The latitude range includes the tropical and subtropical oceans and fully captures the Hadley cells where falling ice (snow) generated in the upper levels of deep convection can lead, via radiation–circulation coupling, to non-local precipitation changes as explored in Li et al (2016).

Table 1 and figure 1 show r_spatial and σ_spatial relative to GPCP for all the models. r_spatial values are between 0.69 and 0.94 with the SON group showing higher correlations (0.82–0.94) than the NOS group (0.69–0.90). Normalized σ_spatial mostly exceed 1, i.e. most models exhibit larger spatial variability than observations (CNRM models are the exception, with σ_spatial from 0.99 to 1.03). There are similar ranges for σ_spatial, 0.99–1.27 for NOS and 1.05–1.24 for SON. Table 1 also reports mean (spatially-averaged) biases whose magnitudes are generally smaller for NOS (0.17–0.68 mm d⁻¹) than SON (0.34–0.78 mm d⁻¹). The mean absolute bias, a measure of the magnitude of biases, are higher for NOS (0.80–1.34 mm d⁻¹) than SON (0.65–1.11 mm d⁻¹). The model ranking of the mean absolute biases can be found in figure 2, which shows that the best performers belong to several SON2 models.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Comparing models in the SON1 (purple) and SON2 (red) subgroups (table 1 and figure 1), all five SON1 models have lower correlations (0.82–0.87) than four of the SON2 models (full range 0.84–0.94, although four models are 0.91 or above). The SON1 mean bias range (0.44–0.78 mm d⁻¹) is shifted higher than SON2's (0.34–0.70 mm d⁻¹), and SON1 also has generally higher mean absolute biases (0.89–1.11 mm d⁻¹) than SON2 (0.65–1.00 mm d⁻¹). SON1 spans a wider range of normalized σ_spatial (1.05–1.24 for SON1 and 1.13–1.20 for SON2), implying that the shift down in mean absolute bias is largely explained by the improved r_spatial, i.e. due to better matching the shape of precipitation patterns rather than the magnitude. In terms of the patterns of precipitation in our studied ocean regions, the SON1 models therefore show worse performance than the SON2 models, and in general are comparable to the NOS models (figure 1). While it is encouraging to see the higher correlations, smaller mean biases and smaller mean absolute biases in SON2 than in SON1, one must keep in mind that all of SON2 models still exhibit considerably larger σ_spatial than observations.

Averaging over multiple models usually results in better performance (Gleckler et al 2008). For example, σ_spatial for SON2 is reduced to 0.94 while its component models span 1.12–1.20; and SON2 has the highest correlation and the lowest absolute bias among the groups (figure 1). SON is the best performing group in terms of correlation and σ_spatial but not RMSB (figure 1), resulting from averaging of the SON1 (low correlation and high spatial variability) and SON2 (high correlation and low spatial variability) subgroups, which contributes to slightly improved performance for CM6 compared to that of CM5 (also table 1) in terms of σ_spatial (1.05 vs 1.1), mean bias (0.51 vs 0.54 mm d⁻¹) and mean absolute bias (0.78 vs 0.85 mm d⁻¹).

The geographical patterns of annual mean precipitation biases for each model are shown in SI (figure S1). The three best performing models are the SON2 models NorESM2-MM, CESM2 and CESM2-WACCM. While the models within the NOS group are relatively poorer in terms of our metrics, we found that the CMIP6 multi-model mean (CM6) shows reduced bias compared to CMIP5 (CM5), and that the inclusion of more models in the SON2 group with its MG2 cloud scheme contributes to this improved performance.

In this section, we compare the spatial patterns of (1) biases for NOS, SON, SON1, SON2, CM6 and CM5 groups against GPCP observations, (2) differences between some groups and (3) changes in the absolute biases from one group to another for the annual-mean climatology. The latter is defined as abs (NOS—GPCP) minus abs (SON—GPCP) in each grid cell, for example. The mapped positive changes in the absolute biases proceeding, for example, from the first group (NOS) to the second group (SON), signify improved performance of the second group (SON) relative to that of the first group (NOS) against GPCP. Pooled t-tests (Bhattacharyya 2013) are used to identify regions with significant differences (at the 95% level) between groups, accounting for the difference in group size. These regions are hatched in figures 3(e)–(h). A number of questions can be addressed. For example, is there any improvement in spatial patterns and biases from CMIP5 to CMIP6? Does the difference in the spatial pattern of precipitation biases correspond to different treatments of ice-cloud radiative properties in the two SON subgroups?

Zoom In Zoom Out Reset image size

Figures 3(a), (b), (c) and (d) show the precipitation biases relative to GPCP for SON, SON1, SON2 and NOS, respectively. Over the convective zones (ITCZ, SPCZ and Tropical Warm Pool) all the groups underestimate precipitation, but overestimate it over the trade wind regions. Over the latter regions, in general, biases are the smallest in SON2 (less than 1.6 mm d^–1) with a slight mitigation of the double ITCZ (figure 3(c)). Otherwise, the spatial patterns of biases for SON, SON1 and NOS (figures 3(a), (b) and (d)) are similar in structure, albeit with different magnitudes.

Both NOS—SON2 and SON1—SON2 differences (figures 3(g) and (h)) are generally positive except along the equator and, for SON1—SON2, for areas west of the coast between approximately Mauritania and Sierra Leone in Africa. This means that the double ITCZ is significantly reduced in SON2 relative to either NOS or SON1. A possible explanation is that indirect changes in circulation from NOS to SON2 could affect the simulations of trade wind via radiation–circulation coupling (Li et al 2014a, 2020). This mechanism can result in substantial improvements for SON2 over NOS as seen from the patterns of changes in absolute biases from NOS to SON2 (figure 3(k)) except for the eastern Pacific ITCZ as noted in Li et al (2020). There is less improvement from NOS to SON (figures 3(e) and (i)), due to cancellation in biases in parts of the study domain between SON1 and SON2 (figures 3(f), (g), (j) and 3(k)). The deterioration of SON1 relative to SON2 will be discussed in section 4.

Next, we study changes in spatial patterns and biases from CMIP5 to CMIP6 shown in figure 4. For this, we choose CM5 as the reference and present the differences (left column) and the changes in absolute bias (right column) for NOS (figures 4(a) and (f)), SON (4(b) and (g)), SON1 (4(c) and (h)), SON2 (4(d) and (i)) and CM6 (4(e) and (j)) from CM5, respectively, using model output over 1980–2005 for all groups. Once again, hatching denotes regions where differences are significant at p <0.05 according to pooled t-tests. As discussed earlier, the spatially-averaged, mean bias and mean absolute bias over the study domain are reduced from 0.54 to 0.51 mm d⁻¹ and from 0.85 to 0.78 mm d⁻¹ from CM5 to CM6, respectively (see table 1). The changes in spatial patterns from CM5 to CM6 (figures 4(e) and (j)) resemble those between CM5 and SON (figures 4(b) and (g)) over the Pacific and south Atlantic trade wind regions, albeit with slightly smaller amplitudes. It appears that the improvement from CM5 to CM6 is contributed mostly by the SON models (figures 4(b) and (g)) and particularly by the SON2 models (figures 4(d) and (i)).

Zoom In Zoom Out Reset image size

All CM6 subgroups show reduced biases around the northward edge of the equator. Elsewhere, bias changes occur with opposite sign in different groups: SON1 and NOS both show increased biases near the Maritime Continent and in the north Pacific, offsetting the improvements in SON2. In other parts of the study domain, particularly the south Pacific and throughout the Atlantic, SON1 has degraded performance even relative to NOS, whereas SON2 once again shows greater bias magnitude reductions.

The seasonal cycles of the double ITCZ index from each model and model groups are shown for the Pacific Ocean (figures 5(a) and (b)) and Atlantic Ocean (figures 5(c) and (d)). The double ITCZ index is defined as the precipitation bias averaged over the area of (150°W–100°W, 30°S–EQ) for the Pacific Ocean and over the area of (35°W–0°, 20°S–EQ) for the Atlantic Ocean, respectively, following Bellucci et al (2010) and Hirota et al (2011). These regions are selected as a double-ITCZ metric because the observed ITCZ is located north of the equator except for a brief period in March and April (e.g. Zhang 2001, Haffke et al 2016), and excess heavy precipitation south of the equator is a consequence of models generating an unrealistic double ITCZ.

Zoom In Zoom Out Reset image size

Figure 5 shows that the amplitude of the annual mean bias is largely determined by the biases over November to June for all models and model groups although there is a small bias of approximately 0.5 mm d^–1 from July to November. Figures 5(a) and (c) show that, all other model groups exhibit a stronger double-ITCZ bias relative to GPCP than SON2 does, ranging from 1.2 to 2 mm d^–1 for the Pacific Ocean in February–May (figure 5(a)) and from 2.8 to 4.7 mm d^–1 for the Atlantic Ocean (figure 5(c)) in April and May. By contrast, the SON2 biases peak at ~1.1 mm d^–1 in the Pacific and 2.7 mm d⁻¹ in the Atlantic. In terms of the index peak, it is suggested that SON1 performs poorly over both regions while SON2 is the best performer. A degradation in performance from CM5 to CM6 occurs in the Atlantic, driven primarily by NOS prior to March and by SON1 from April–June. SON2 is the only case showing consistent improvement.

For individual models, CESM2-WACCM has the smallest index in the boreal spring (0.25 mm d^–1) over the Pacific (figure 5(b)) and 0.5 mm d^–1 over the Atlantic (figure 5(d)), followed by NorESM2-MM, CESM2 and TaiESM. The two other SON2 models (E3SM-1-0 and CESM2-WACCM-FV2) have values as large as those in NOS and SON1 models over the Pacific (figure 5(b)). The other models show a wide range of biases in spring and early summer over both oceans. Over the Atlantic, the SON2 models generally outperform the other groups in spring but their later peak in May and June results in no performance improvements then.