An expert judgment model to predict early stages of the COVID-19 pandemic in the United States
Fig 4
Forecast accuracy for expert predictions of cases and deaths.
Evaluation of forecast accuracy for forecasts of cumulative COVID-19 deaths (A) and cases (B). For both types of questions, the methods used to elicit probabilistic forecasts changed and this point is indicated by a vertical dashed line. Predictions are shown from each expert (light dots), the median expert (dark diamond), and the linear pool (dark square) compared to an “unskilled” forecaster (see Methods). Higher relative forecast skill indicates better performance than an unskilled forecaster and a zero relative forecast skill represents identical performance with an unskilled forecaster. (A). Relative forecast skill of the cumulative number of COVID-19 deaths by December 31, 2020 (see Fig 1A). Over 50% of experts made better predictions of year-end COVID-19 deaths than an unskilled forecaster on each of the five occasions this question was asked. Experts’ median relative forecast skill was higher than the linear pool forecast skill for the latest prediction of year-end deaths when asked to provide percentiles compared to the smallest, most likely, and largest number of deaths. (B.) Relative forecast skill of the number of cases to be reported by the end of the week from thirteen surveys administered between February 23 and May 17, 2020. Individual experts’ accuracy was mixed with some experts performing better than an unskilled forecaster and others scoring worse. In the first five surveys, the median expert made less skilled forecasts than the unskilled forecaster. Experts’ median relative forecast skill was smaller than the linear pool forecast skill when asked to provide a smallest, most likely, and largest number of cases and similar to a linear pool when asked to assign probabilities to a set of intervals where the true number of cases could fall.