Published October 5, 2022 | Version v2
Software Open

Optimized version of the ecRad radiation scheme with new RRTMGP-NN gas optics

Description

This is a development version of ecRad 1.6 which includes CPU performance optimizations and ecCKD (already included in official version 1.5), RRTMGP and RRTMGP-NN gas optics schemes. This code has testable optimizations - if you don't care about this and just want the most up-to-date optimized ecRad code, see branch clean_no_opt_testing in this github repo: https://github.com/peterukk/ecrad-opt. 

The code is used in the papers

  • Implementation of a machine-learned gas optics parameterization in the ECMWF Integrated Forecasting System (GMD 2023)
  • Twelve times faster yet accurate: a new state-of-the-art in radiation schemes via performance and spectral optimization (JAMES 2024)

The offline timings in both papers were produced on the ECMWF Atos HPC, using these steps for the first paper:

  1. make PROFILE=aa_gfort SINGLE_PRECISION=1  GPTL_TIMING=1 OPTIM_CODE=2
  2. cd test/ifs
  3. run ecRad with TripleClouds and either RRTMG, RRTMGP or RRTMGP-NN:
    • make test_tripleclouds; make test_rrtmgp_tc; make test_rrtmgp-nn_tc
    • repeat the resulting runtime commands but change the input "ecrad_meridian.nc" to the big input file with 40,000 columns, e.g.:
    • ../../bin/ecrad config_rrtmgp_tc.nam ecrad_highres_40k.nc TMP_OUT_BIG.nc

This example uses the Make profile for GCC compiler and ECMWF-AA, change the argument of PROFILE for your own (location of GPTL timing library needs to be provided, otherwise compile without GPTL_TIMING=..). For the timings in the paper a SLURM batch script was used with OMP_NUM_THREADS set to 128.

The timings in the second paper were produced in a similar way. The highest level of optimization (fastest results) includes making the innermost g-point dimension a compile time constant, which makes a big difference when using ecCKD gas optics. To compile for 32-term ecCKD models:

  1. make PROFILE=aa_gfort SINGLE_PRECISION=1  GPTL_TIMING=1 OPTIM_CODE=2 NG_LW=32 NG_SW=32
  2. In test/ifs, produce the configs e.g. for ecCKD+SPARTACUS and ecCKD+TripleClouds: make_test_ecckd_spartacus; make_test_ecckd_tc
  3. ../../bin/ecrad config_ecckd_spartacus.nam ecrad_highres_40k.nc TMP_OUT_BIG.nc

In this example, attempting to run ecRad with a gas optics scheme that doesn't have 32 g-points will result in a segfault!

Configurable optimization levels:

  • All optimizations except using original expm and reflectance-transmittance kernels, and no explicit ng: OPTIM_CODE=1
  • All optimizations except not using explicit ng: OPTIM_CODE=2
  • Full optimizations: OPTIM_CODE=2 NG_LW=... NG_SW=...

Otherwise uses reference code

Files

Files (1.0 GB)

Name Size Download all
md5:6ebf3790b55e673470b02b403733726b
114.5 MB Download
md5:a3639aac4f5e95be3499c2b4858a74b8
896.4 MB Download