Multiple models for outbreak decision support in the face of uncertainty

Significance During infectious disease outbreaks, uncertainty hinders our ability to forecast dynamics and to make critical decisions about management. Motivated by the COVID-19 pandemic, leading agencies have initiated efforts to prepare for future outbreaks, for example, the US Centers for Disease Control and Prevention’s National Center for Forecasting and Outbreak Analytics and the WHO’s Hub for Pandemic and Epidemic Intelligence were recently inaugurated. Critically, such efforts need to inform policy as well as provide insight into expected disease dynamics. We present a validated case study from early in the pandemic, drawing on recommendations to minimize cognitive biases and incorporate decision theory, to illustrate how a policy-focused process could work for urgent, important, time-sensitive outbreak decision making in the face of uncertainty.

2 scientific uncertainty (e.g., about epidemiological processes or parameters, or intervention efficacy, given limited data) that is relevant to policy development and decision making. In this framework, insights can be shared across modeling groups to inform the collective projections, while retaining the perspective of individual groups.
Sources of cognitive bias are many (3)(4)(5)(6), but three main biases our process guards against include: dominance or authority effects (where there is a tendency to agree with field "leaders"); starting-point bias or anchoring (focusing on suggestions raised early in the process to the detriment of other ideas); and groupthink (when a desire for cohesiveness causes collaborators to minimize conflict and reach consensus without sufficient critical evaluation).
The process involves multiple steps (Fig. S1), including two rounds of modeling with an intervening structured discussion to eliminate unwanted biases and uncertainty (including semantic or linguistic uncertainty), increase consistency in modeling of interventions, share critical insights, and generate a comprehensive picture of relevant uncertainty. The projections from the second round of modeling are then used to generate aggregate results under different interventions that encapsulate scientific uncertainty about epidemiological processes and management interventions (7). We stress that this process is designed primarily to inform decision making, rather than to provide quantitative projections of epidemic trajectory (as in ongoing forecasting challenges (8)), though such results are also obtained. The multi-model, multi-step process is expected to generate better calibrated projections than individual models. That is, the aggregate distributional forecast will be more consistent with future observations than individual forecasts. More importantly, this process is also expected to produce more robust insights about the ranking of intervention options that improve management outcomes. In short, we use multiple models to generate better expressions of uncertainty while guarding against cognitive biases to provide decision support. The COVID-19 pandemic offers a unique opportunity to apply this structured framework.
We also stress that our approach encourages an integration of science and policy-makingefforts that are often separated to the detriment of public health, economic, and environmental outcomes when semantic uncertainties cannot be clarified and may thus interfere with success. Modelers intend their forecasts to 'inform management decisions,' yet the common separation of model outputs from the decision context increases the chance of misunderstandings and errors. Continued efforts to foster collaboration and streamline communication between modelers and decision makers, as well as to shift the focus from solely providing projections to evaluating proposed interventions, are essential steps towards effectively leveraging modeling efforts to inform decisions.
Resolution of linguistic uncertainty in structured discussion between rounds 1 and 2 The group discussion between modeling rounds identified numerous sources of linguistic uncertainty arising from different interpretations of the objectives and the nature of interventions from the problem setting. For example, the wording on reopening '2 weeks after peak' engendered considerable confusion in the first round of modeling. How is a peak defined? Is it in reported deaths or cases? Is it measured on a daily or a moving-average basis? Likewise, how 3 should a model determine whether 2 weeks have passed since the peak? A continuous monotonic decline was never seen; should a moving average be used? And, if so, for 7 or 14 days?
As well as allowing a common definition of "peak" and other terms, other sources of unanticipated uncertainty were resolved. For example, one modeling group asked for clarification on the definition of 'death.' There was a thorough discussion of the options that different groups had considered or used (reported only; reported plus probable; reported, probable and co-morbidities; or, also indirect deaths, such as those from unrelated causes in patients choosing not to go to the ER during a pandemic). We agreed as a group to use all deaths due to COVID-19 disease-induced mortality, regardless of reporting. This way of counting deaths is based on infection status, not testing status, and can include comorbidities but not indirect deaths, as we are only focusing on people who have been infected with SARS-CoV-2 and died from their infection.
The first round also provided some important checks and balances on the consistency of objective and intervention interpretations across groups, i.e., were the same definitions of workplace closures used? In the first round, some groups used the May 15 to November 15 timeframe, others based start dates on declarations of a State of Emergency or stay-at-home orders, and one group implemented a weighting for essential and non-essential business closures and associated compliance issues explicitly (Fig. S11). Including a metric that should be consistent across models allowed us to check for and remove linguistic uncertainty in round 2 submissions that would have limited our ability to compare the rankings of interventions between models and objectives. Clear guidelines developed during and after the group meeting removed this uncertainty from round 2 projections, improving the comparability of intervention rankings across models. Even so, there was still considerable variation across modeling groups in how these openings were triggered, in part because the triggering events were sensitive to how daily variation in the projections was handled (Fig. S12B). The MMODS Process is deliberate in explicitly planning for and appropriately managing this process, so that all groups are equally informed and use the same interpretations. Formally building the discussion phase into the modeling and decision-making process manages decision-maker expectations. Modeling teams also commented that they found the well-defined structure in Fig. S1 to be valuable.
In addition to resolving linguistic uncertainty, the first round provided information on the utility of the interventions themselves. We initially requested results for reopening after declining to 1% of peak. Round 1 results suggested this condition would rarely, if ever, be met, so that results for this intervention were not meaningfully different from those of the closed intervention, and thus we altered the intervention to trigger at 5% of peak instead. Typically, such changes in interventions would be made in consultation with decision makers (as part of Fig. S1, loop A).
Deliberately, consensus on scientific uncertainty was not required. In fact, model results were presented anonymously to reduce the pressure to conform to other groups' expectations and hence to avoid 'groupthink,' and other cognitive biases, engendering a more comprehensive expression of legitimate scientific uncertainty. We thus encouraged modeling groups to adjust their models to reflect unknown aspects of the transmission and intervention implementation process to more fully express genuine scientific and logistical uncertainty. A fuller expression of uncertainty, captured in individual models and the aggregate, allows for more robust risk quantification. If for example, a local official or hospital administrator had to rely on only a single model to estimate the exceedance risk for hospital bed capacity, they might mis-estimate the risk and possibly over-or under-prepare.
Due to the opposing effects of decreasing linguistic uncertainty and maintaining or increasing expression of scientific uncertainty, it was difficult to draw conclusions about the source of model-level changes in expressions of uncertainty between rounds 1 and 2. To begin to assess model-level changes, we compared the lengths of inter-quartile ranges (IQRs) (Figs. S13-S15) within groups by round as well as the ratio of IQR length between each model and the corresponding aggregate distribution. The clearest examples of model incorporation of additional scientific uncertainty in round 2 were the models that provided point estimates in round 1 (length of IQR = 0) that subsequently expanded these estimates to distributions in round 2. Requiring distributions rather than point estimates necessarily increased the degree of expressed uncertainty. However, even in these models, we observed decreases in uncertainty (presumably in linguistic uncertainty) as the point estimates account for the majority of outliers in round 1 (Fig. S13).
For each objective-intervention pair considered in both rounds, the length of the aggregate IQR was greater than the median length of the corresponding model IQRs (Fig. S14). The degree of uncertainty (as measured by IQR lengths) for the majority of models increased towards that of the corresponding objective-intervention aggregate distribution from round 1 to round 2 (see the clustering of points near the orange dashed line in Fig. S14 round 2). Implementation of the open and closed interventions did not rely on a definition of "peak". In Fig. S15, we observed that the ratio of IQRs (IQR(model)/IQR(aggregate)) between rounds tended to be closer to one than the 2-week intervention, which required a definition of peak (Fig.  S15). We also note that decreases in the IQR length for the aggregate distribution were observed for all objectives in the 2-week scenario (i.e. aggregate ratio of IQRR2/IQRR1 <1). Changes observed in the open scenario (cumulative infections, cumulative deaths, and peak hospitalization ratios observed are 1.20, 1.02, and 0.949 respectively) were moderate compared to those in the closed scenario (cumulative infections, cumulative deaths, and peak hospitalization ratios observed are 1.93, 1.92, and 1.54 respectively). Note that an analogous comparison for the alternative peak intervention was not possible, given the change from 1percent of the peak to 5-percent of the peak between rounds.
Tradeoff between public health and economic objectives Balancing public health outcomes and economic considerations is an important aspect of pandemic decision making, but needs a more nuanced treatment, particularly on the economic side. If we compare each of our four interventions to a hypothetical "no disease" scenario, we can identify four broad groups of costs from multiple perspectives: (a) financial costs caused by the disease itself (e.g., reduced economic output due to absence from work due either to illness or voluntary isolation, lower productivity at work or when 5 working from home, direct and indirect costs of medical treatment, or costs associated with funerals); (b) non-financial costs caused by the disease itself (e.g., mortality, morbidity, long-term health impacts); c) financial costs caused by the strategic response to the disease (e.g., reduced economic output due to a lockdown, costs of monitoring and enforcing a lockdown or of quarantining incoming travelers) or by individual responses that go beyond local policy; and (d) non-financial costs caused by strategic or personal responses (e.g., mental health challenges due to isolation and stress, relationship stressors and breakdowns, family violence, reduced access to opportunities for recreation).
The magnitudes of these costs will depend on a wealth of factors, potentially including behavioral factors (e.g., forgoing preventive medical care such as routine childhood vaccinations), economic factors (e.g., poverty causing a need to work despite disease risks), and policy factors (e.g., constraints such as lockdowns). There may also be direct or indirect economic benefits associated with mitigation activities. For example, some firms have found that having staff work from home can increase efficiency and reduce operational costs (9), which could have ongoing benefits. Air pollution was reduced by reduced transport during lockdowns. The successful use of online video software to hold meetings including people from different locations is likely to result in a greater reliance on that approach post-pandemic, leading to reduced costs of travel, accommodation and lost work time and reduced emissions of greenhouse gases. The likelihood that people will adapt creatively to the constraints imposed by a pandemic or by a management strategy increases the difficulty of estimating the costs, because the nature and success of such adaptations are somewhat unpredictable.
There may be important dynamic trade-offs in the benefits and costs that arise. For example, in some circumstances it may be worth incurring higher financial costs in the short term (a more extreme lockdown) in order to achieve a more rapid opening up of the economy following successful containment of the disease. New Zealand provides an example where this strategy was successfully followed.
In principle, the optimal strategy would be that which minimizes the sum of all these costs, less any benefits. Our analysis does not quantify all of these costs and benefits but does provide evidence that could be used as key inputs to a comprehensive economic analysis. This analysis also starkly illustrates the tensions between economic and public health goals seen worldwide and suggests that strategies that only consider the timing of re-opening, or focus on a single type of intervention, may not be nuanced enough to manage these trade-offs. Feedback to decision makers from this process may lead to refined, possibly multi-criteria, objectives (via loop A in Fig. S1).

Insights from individual models
The median probability of an outbreak increased to 100% for all intervention scenarios other than closed. Even relatively stringent re-opening guidelines were insufficient to guarantee success; complete cessation of community spread of the virus was unlikely even with long-term non-essential workplace closure, i.e., non-essential workplace closures alone would be insufficient to manage COVID-19 at a county-level). Either additional stay-at-home orders would be required, or other non-pharmaceutical interventions (e.g., testing, contact tracing and 6 isolation, or wearing masks) or pharmaceutical interventions (e.g., vaccination) would be needed to stop transmission while allowing workplace re-opening.
Three models ranked the closed and 5-percent interventions as identical for several metrics; 6 groups reported that for at least some simulations the 5-percent reopening criterion was never met in the 6-month period. Two models ranked the 5-percent intervention as better than the closed intervention based on the medians of health outcomes; both models had wide priors on parameters governing compliance with interventions. Three of the 17 models ranked the 5percent intervention worse than the 2-week intervention for public health measures (Fig. S12A), a result that was driven by different timing in the triggering of re-opening (Fig. S5). Another notable result is that the ranking of the 2-week intervention for the peak hospitalization metric spanned the gamut from worst rank (in submission M) to first-tied rank (in submission A) ( Fig.  1). Otherwise, rankings were remarkably consistent overall.
Disagreements between models, or between models and the aggregate, were examined retrospectively, and include a range of reasons: genuine scientific disagreement about processes to include in the model given the massive uncertainty about SARS-CoV-2 at the time; stochasticity (especially in the case of very close or "tied" results); differences in calibration approaches; residual linguistic uncertainty (language uncertainty was drastically reduced but not completely eliminated by group discussions); inclusion of assumptions groups would later choose to revise if more time was available (especially with the benefit of hindsight).

Comparison of county death and case data with aggregate model results
The modeling exercise was motivated by a U.S. county representative of mid-sized counties with populations of approximately 100,000 people, with limited mobility and stay at home orders in place until at least May 15, 2020. Here, we compare aggregate model results with reported data from U.S. counties meeting the target county description.
We first selected the 98 U.S. counties with population sizes between 90,000 and 110,000 using data from the Johns Hopkins University COVID-19 dashboard (9). From this subset, we then selected counties with stay at home orders in place until at least May 15, 2020 (data from (10-12)), and changes in mobility in line with stay at home orders, i.e., less than 50% increase from baseline retail mobility, less than 25% increase in baseline work mobility, and less than 5% decline from baseline residential mobility (data from (13)). This resulted in a subset of 85 counties. Finally, from this subset, we identified 18 counties implementing a fully 'closed' intervention (with stay at home orders in place and unmodified from May 15, 2020 to November 15, 2020 and mobility patterns suggesting those orders were followed). Relaxing the definition of 'closed' to counties with any stay at home orders in place (including unmodified, modified, partial or stay safe at home orders) from May 15, 2020 to November 15, 2020, the subset of counties considered to be 'closed' increases to 84 (data from (14)). One county was found to be fully open during this period, and it was not possible to determine if any counties implemented the '2-week' or '5-percent' interventions. We compared aggregate cumulative reported deaths and cases with modeled cumulative deaths and infections (all COVID-19 deaths and infections, both reported and undetected) under the closed intervention for the 18 counties following the 'closed' intervention according to the strict definition of closed (including only full stay at home orders) and the 84 counties following the 'closed' intervention according to the relaxed definition of closed (including full and partial stay at home orders). Cumulative reported deaths and cases for the two groups of counties under the closed intervention were summarized in 100 quantiles, the same format requested from model groups (See Figs. S16 -S17, below).
Note that we are comparing reported deaths and cases (from data) with all COVID-19 deaths and cases from model results (which captures reported and undetected infections). We did not assume a reporting or detection rate, but perforce expect a higher number of model-predicted cumulative deaths and cases. Crucially, our results represent the realization of one pandemic across multiple counties in comparison to multiple model realizations across a wide range of uncertainty. Thus, the model uncertainty will necessarily be higher than the observed uncertainty. The model mean will likely also be higher, as the right-skewed uncertainty will increase the mean.  (4) immediately relax all current restrictions on non-essential workplaces. Loop B coordinates modeling groups to reduce bias and linguistic uncertainty. First, loop B involves independent (round 1) model Projections of all objectiveinteraction combinations. A structured, facilitated group discussion reduces unwanted uncertainty and also prompts information on additional sources of data used, methods used to incorporate uncertainty, and assumptions made by individual groups, so that the whole collaborative can improve their models. Retention of the remaining model differences allows for a more comprehensive expression of legitimate scientific uncertainty; consensus is not required. Modelling groups then provide updated (round 2) model projections. Loop A provides an opportunity for model groups to interact with decision makers to clarify or update objectives or interventions, i.e., to reduce linguistic uncertainty. Decision Analysis is used to aggregate and analyze the model outputs to rank interventions. If decisions are implemented, then there is also an opportunity for modeling teams to learn from Implementation data and results (loop C)  Each colored line shows the quantile distribution for a single model. Each aggregate CDF is shown in black with medians, 50% PIs, and 90% PIs indicated as red points, thick lines, and thin lines respectively.

Fig. S12. Comparison between the 2-week and 5-percent interventions. A) Medians (points)
and 50% PIs (lines) displayed pairwise by intervention and for the following objectives: i) cumulative infections, ii) cumulative deaths, iii) peak hospitalizations, and iv) days closed for each model. B) Comparison of intervention start dates for 2-week (grey) vs. 5-percent (black) interventions for each model, where the start date is computed as the number of days from May 15 until the intervention is enacted. Intervention start times of 184 days indicate that the intervention was never triggered in that model. All plots display median (points) and 5 th to 95 th quantiles (lines) for each intervention. The 2-week intervention trigger to open is the first day for which the 7-day trailing moving average of the number of new daily reported cases has been lower than the maximum for at least 14 days, and has shown a day-to-day decline in smoothed case data for at least 10 of the last 14 days (or, there have been 7 days without any new cases). The 5-percent intervention trigger to open is the first day for which the 7-day trailing moving average of the number of new daily reported cases drops below 5% of the maximum. 20 Fig. S13: Team and aggregate values for each intervention and objective pair. Round 1 and round 2 results displayed in red and blue respectively. Since the 1-percent intervention from round 1 was updated to a 5-percent intervention in round 2, results for these interventions have been omitted from this comparison. Also note that two models were excluded from this analysis, as they submitted incomplete results in round 1. After the discussion between rounds 1 and 2, these groups were able to provide complete and comparable results. Additionally, in at least one case, some of the differences can be attributed to model error fixes between rounds.  Because the 1-percent intervention from round 1 was changed to a 5-percent intervention in round 2, the corresponding results have been omitted from this comparison. The dashed orange line highlights the point at which there is no difference between the model-specific IQR lengths between rounds 1 and 2 (points to the left indicate a lower R2 IQR than that of the corresponding group's R1 submission, and vice versa for points to the right).   Participants were asked to indicate which of the provided datasets were used for any part of the model (e.g., for calibration, training, fitting etc.) as part of the submission checklist. All but one model used at least two of the provided data. Model F used only external data sources (provided data was used solely to better understand the intent of the exercise).

Notre Dame-FRED
Description Agent-based model, FRED (Framework for Reconstructing Epidemic Dynamics), with updated epidemiological parameters based on studies to date. FRED explicitly models transmission dynamics of a pathogen in a realistic population, and allows for the impacts of NPIs to be modeled explicitly (e.g., school closure results in agents representing children staying home). Code available: https://github.com/confunguido/covid19_ND_forecasting Calibration Disease specific parameters were calibrated to the number of daily deaths in PA. Adams County was then simulated to estimate the rate of importations from the state incidence and a scaling factor to google mobility trends. Parameters were uniformly sampled for each step of the calibration using a sobol design sequence (pomp package in R). Then, the likelihood based on the daily number of deaths was calculated.

Calibration
Transfer learning was used to let the LSTM learn how the epidemic would eventually end from another region, explore the RNN structure and hyperparameters, and apply them to tune the model for the modeled region Additional data sources used COVID-19 data from another region where the epidemic has (presumably) ended.

Description
Stochastic, age-and risk-structured compartmental model that includes susceptible, exposed, presymptomatic, asymptomatic, symptomatic, hospitalized, and recovered states (SEPAYHR). The model is simulated using a hybrid approach with a deterministic initial phase (up to 20 total symptomatic cases) followed by a stochastic phase. Diagram See Fig. A1 in (31) Calibration Basic reproductive number (Rt) was estimated using provided and transmission probability was estimated using a next-generation matrix approach based on the model structure and Rt. Epidemic start date was based on the time to first death implied by the estimated Rt and transmission probability. Transmission reduction due to social distancing was estimated with a nonlinear least squares fitting procedure in the SciPy/Python package. Detection rate was estimated using the provided data and published estimates of age-structured infection fatality ratios.  The original project description and request for contributions to MMODS Elicitation 1 is appended to this supplement. The request includes a detailed description of the MMODS process as well as information on the setting, interventions, and objectives to be considered during the first MMODS collaboration exercise.
The submission form and checklist of data required for round 2 models is appended to this supplement. It comprised a detailed web form requiring the reporting of information characterizing the model being submitted. This form was hosted behind a login portal for participating model groups on the MIDAS website.
Movie S1. MMODS_Elicitation1_InterventionRankResults.mp4 Model-specific intervention rank results evaluated for each objective in round 2 of the MMODS process. Results are displayed in video format by quantile (1 through 100). Colors indicating ranks and rank-ties are as specified for the median rank result figures in main text and range from single best intervention (dark blue) to single worst intervention (dark red).
Data S1. MMODS_Elicitation1-ProvidedData.xlsx Data for a generic U.S. county were provided to modeling groups to inform their model specifications. Provided data include: epidemiological data on daily and cumulative cases and deaths from January 22 to May 15, 2020; demographic information on age-and sex-distribution; testing and mobility; timing on the release of a stay-at-home order and a State of Emergency declaration. Data adapted from (9-11, 13, 32, 33).

Data S2. MMODS_Elicitation1-OutputData. xlsx
Anonymized results for individual models and the aggregate. Results are provided in a standard format, including 100 quantiles for each model-objective-intervention (or aggregate-objectiveintervention) combination (i.e., the probability distribution for each outcome for each intervention, via the cumulative distribution function (CDF) in 100 quantiles).

Harnessing Multiple Models for Outbreak Management Exercise I. Relaxation of social distancing
The Problem The profusion of models for COVID-19, with differing structures, varied epidemiological scenarios, parameters and presentation, and sometimes conflicting projections, is a challenge for decision-makers. In a recent paper, we proposed a method for harnessing the power of multiple models by drawing from tools in decision analysis, expert judgment, and model aggregation (Shea et al. 2020). This project is meant to implement that proposal in the context of COVID-19. We aim to generate unbiased and well-calibrated aggregate projections under different interventions, that encapsulate scientific and logistical uncertainty, to better inform management decisions. In this framework, insights can be shared across groups to inform the same decision, while retaining the perspective of individual groups as part of the full expression of uncertainty.
The overall goals of this project are to implement these procedures for a series of COVID-19 decisions; engage a diverse set of modeling groups with expertise in structured, collaborative, ensemble projections; and develop efficient logistical processes for managing our broad communal effort. This is a complementary effort to the COVID-19 forecasting hub developed by Nick Reich and colleagues, with an explicit focus on interventions and decisions.
Specifically, we will run multiple projection exercises to address key decisions facing managers of COVID-19, including when and how to relax key social distancing interventions (exercise I).
In later exercises, we will use model assemblages to assess more nuanced partial reopening strategies, intervention decisions at state and country levels, where best to trial vaccines and drugs, how to prioritize testing and how to optimize the roll-out of medical interventions. We will request, from each participating group, one or more models that encapsulate their group's best understanding of the current pandemic (that is, we will treat each model as an hypothesis about the current outbreak). All participants will be invited as co-authors on resultant publications. Results will be kept confidential within the group until presentation to decisionmakers or in publication(s). When presented outside the group, only model participation will be disclosed (individual model results will be anonymized).

Procedure
We will use principles of decision analysis to help structure model projections and analysis, and adopt well-established methods from the expert judgment literature so that the results from multiple modeling groups can all contribute to insights about the same decision context and contribute to a synthetic and long term resolution to the current pandemic.
For each exercise, we will take the following steps: 1. Setting. We will present a decision setting, specifying the background epidemiology (location, outbreak trajectory), the targets of the decision maker (e.g., minimizing deaths, epidemic duration, etc.), and the intervention(s) to examine. Relevant epidemiologic and demographic data will be shared. 2. Individual Projections 1. We will ask each modeling group to independently estimate the desired outcomes under the alternative interventions, with particular attention to expressing uncertainty. We will ask for probability distributions for each outcome and intervention scenario. 3. Group Discussion. We will compile the results from the multiple modeling groups and display them (anonymously) in a format that permits ready comparison. We will convene a group discussion with all the modeling groups to explore the commonalities and differences, to share insights, and to discuss sources of uncertainty. 4. Individual Projections 2. We will then ask each modeling group to independently project the same targets under the alternative interventions again, taking into account the insights from the group discussion to the extent they find them compelling. We will again ask for probability distributions. 5. Aggregation and Analysis. We will then aggregate the second round of results into a set of ensemble projections that captures the uncertainty within and across modeling groups. We will also conduct a value-of-information analysis to identify sources of uncertainty that most affect the choice of an intervention. The summary of this work should be an analysis that conveys to the decision maker the expected performance of each of the interventions, using the ensemble projections, with an understanding of the role of uncertainty.

Setting and initial conditions:
We ask that you consider the setting of a US county of 100,000 people, with an age structure typical of the age structure across the US, that pre-emptively initiated, and adhered to, stringent social distancing guidelines (i.e., full lockdown with workplace and school closures) until May 15 th , 2020. As of 15 th May 2020, the town has recorded 180 confirmed cumulative cases and 6 total deaths (time series for both provided). Please assume current (i.e., partial) travel restrictions remain in place throughout the exercise, so that no international importation is allowed and domestic importations are limited.
The decision maker is the county executive, who has authority to specify guidance for opening workplaces. The focus is on decisions regarding social distancing and re-opening over the next few months, prior to the onset of the influenza season.

Projection outcomes/objectives:
The county executive has indicated they are interested in weighing the trade-offs among a number of outcomes, including the impact of the disease on public health, hospital resources, and the local economy. To reflect these objectives, we ask participating modeling groups to address 5 outcomes (metrics): 1) cumulative number of infected individuals through November 15 2) cumulative number of deaths through November 15 3) peak hospitalizations through November 15 4) probability of a new local outbreak (more than 10 new cases/day) before November 15 5) total number of days workplaces closed through November 15

Interventions
In this first exercise, we will only consider relaxation related to workplaces. We request that you provide model projections for the following 4 intervention scenarios: For now, please assume no local testing/contact tracing and isolation of infected individuals; we will return to evaluate this in a future elicitation. You are however free to define and present results for any other relaxation process you feel is relevant or interesting.
Models should provide a full probability distribution of outcomes for each intervention, such that tail probabilities for the 2 nd and 98 th quantiles are relatively stable. Specifically, we want the probability distribution for each outcome for each intervention, by specifying the cumulative distribution function (i.e., with 100 quantiles). We will provide a submission template.
We request submissions by 9 June 2020. Please provide your contact information in the Google Spreadsheet if you plan to participate, and we will give you more detailed submission information. In case of questions, email Dr. Katriona Shea (k-shea@psu.edu).
Background information on your model: Please provide a short write up of your model, including assumptions made about key epidemiological parameters, with parametric uncertainty (e.g., transmission, recovery, R0, serial interval). Please document all sources of variation in your model using the checklist provided. We are looking for full expression of uncertainty in these projections. For example, uncertainty may be structural (e.g., should asymptomatic carriers be modeled explicitly?), or parametric with respect to the biology (e.g., what is the expected time between sequential cases in a chain of transmission?), the setting (what is the assumed rate of domestic importations?) or the interventions (e.g., what is the expected impact of social distancing?) or there may be other sources of stochasticity. Other key uncertainties you might scan across might include: controllability of social distancing, probability of novel incursions that might lead to a second wave of local infections, etc. Details of any model calibration or inference framework used should be provided (checklist will be provided). If we do not specify something, please use your best judgment and include that in your modeling of uncertainty (and please let us know in your short model description and in the checklist). Do not hesitate to send questions, and please provide any other information you feel is pertinent so we can update our checklist for future exercises. Save your work, or load work that you've saved! Load answers from a previously saved or submitted model:

Load Clear Answers
Save Progress Note: Submission Files aren't saved as they shouldn't be re-used.
Please be sure to save your work by clicking the "Save Progress" button. Any unsaved work will be lost if you refresh or leave this page. Submissions will be automatically saved upon successful submission.
All 2elds marked * are required for submission.
Please be sure that your submission 1le meets the requirements speci1ed in the submission template, and upload your 1le here. In order for your submission to be considered, please 1ll out the form below and then click the submit button when 1nished (allowed 1letypes are csv, xlsx, and xls): no 1le selected