Demonstrating multi-country calibration of a tuberculosis model using new history matching and emulation package - hmer

Infectious disease models are widely used by epidemiologists to improve the understanding of transmission dynamics and disease natural history, and to predict the possible effects of interventions. As the complexity of such models increases, however, it becomes increasingly challenging to robustly calibrate them to empirical data. History matching with emulation is a calibration method that has been successfully applied to such models, but has not been widely used in epidemiology partly due to the lack of available software. To address this issue, we developed a new, user-friendly R package hmer to simply and efficiently perform history matching with emulation. In this paper, we demonstrate the first use of hmer for calibrating a complex deterministic model for the country-level implementation of tuberculosis vaccines to 115 low- and middle-income countries. The model was fit to 9-13 target measures, by varying 19-22 input parameters. Overall, 105 countries were successfully calibrated. Among the remaining countries, hmer visualisation tools, combined with derivative emulation methods, provided strong evidence that the models were misspecified and could not be calibrated to the target ranges. This work shows that hmer can be used to simply and rapidly calibrate a complex model to data from over 100 countries, making it a useful addition to the epidemiologist's calibration tool-kit.


Background
Complex mathematical models are widely used in infectious disease epidemiology.
Their reliability in predicting the behaviour of real-world systems strongly depends on our ability to calibrate them to the empirical data.
Despite the wide variety of calibration methods available to date, the application of calibration methods to the analysis of complex models is often lacking.Background (cont.)"For 52% of the articles, the parameter-search strategy could either not be identified or was described as an informal, non-reproducible method." "most papers only presented a single best parameter combination.""Only one-third of articles used calibration methods that quantify parameter uncertainty."

Background (cont.)
The lack of a robust calibration may lead to

• overconfident predictions
• misleading recommendations being made to policy makers, potentially costing lives.
History matching with emulation (HME): a method specifically designed to calibrate complex models.
The risk is for complex models to not be robustly calibrated.
Hmer: a new R package that allows to easily and efficiently implement HME.
We demonstrate the use of hmer for the calibration of a tuberculosis model.

Methods: the model
We developed a compartmental deterministic dynamic model of Mtb transmission and progression accounting for: We used this model to evaluate the impact of new tuberculosis vaccines in low-and middle-income countries.

Methods: the calibration task
The aim was to calibrate our model separately to each of 115 LMIC countries.
We had 9-13 target measures to fit to for each country.
On average, a model run took around 10 seconds.

Methods: history matching with emulation (HME)
HME proceeds as a series of iterations, called waves, where implausible parameter sets, i.e. parameters sets unable to match empirical data, are identified and discarded.
Emulators, computationally fast approximations of the model outputs, are used to estimate model outputs without running the model an unmanageable number of times.

Methods: HME and the hmer package
The hmer package allowed us to adopt an automated history matching with emulation process, not requiring any customisation choices across different countries or waves.
History matching with emulation allows to perform a systematic exploration of the parameter space, providing full uncertainty quantification.
This was key, since we were dealing with a large number of calibration tasks (115) and checking each calibration process manually would have not been feasible.

out of 115 countries were successfully calibrated.
A minimum of 1000 full-fitting parameter sets were identified for each country.
Computational cost: each country was set to run on a single computer node (typically processor at 2.6 GHz) for 7 full days.
On average the non-implausible space identified at the end of all waves was ×^ times smaller than the initial parameter space.

Results: 1D output plot
This plot allows us to assess how much better parameter sets at later waves perform compared to parameter sets in the initial design.
Compared to parameter sets from the first wave, parameter sets from later waves are in closer and closer agreement with our targets (in white).
Each column is a different model output

White vertical bars indicate targets
This plot shows the output values for non-implausible parameter sets at each wave for each combination of two outputs.
The main diagonal shows the distribution of each output at the end of each wave, with the vertical red lines indicating the lower and upper bounds of the target.
In the off-diagonal boxes are plots for each pair of targets, with rectangles indicating the target area where full fitting points should lie.

Results: posterior plot
This plot shows the distribution of the non-implausible space for different waves.Each row and column refers to a specific parameter.
The main diagonal shows the distribution of each parameter singularly: the distributions tend to become more and more narrow wave after wave.
In the off-diagonal boxes we have plots for all possible pairs of parameters.
Results: countries that could not be calibrated 10 out of 115 LMIC countries could not be calibrated to the available empirical data.
The hmer visualisations and derivative emulation tools provided strong evidence that the 10 uncalibratable models were misspecified and could not be calibrated to the target ranges.
Both plots present a "wall", indicating that, given the parameter ranges specified, the model was not able to simultaneously match the two targets shown.
By allowing us to calibrate a complex TB model to 105 countries, hmer made it possible to produce a robust evaluation of the potential impact of new tuberculosis vaccines in low-and middle-income countries.
Estimates of the reduction in TB incidence and mortality under a variety of vaccine scenarios can be found in Clark et al. 2023 "The impact of alternative delivery strategies for novel tuberculosis vaccines in low-income and middle-income countries: a modelling study", in The Lancet Global Health.