Decision analysis support for evaluating transmission risk of COVID-19 in places where people gather

Background: The coronavirus diseases 2019 (COVID-19) pandemic has presented an unprecedented public health challenge. Prior to vaccination, non-pharmaceutical interventions, including closures, were necessary to help control the epidemic. With the arrival of variants of concern and insufficient population vaccination coverage, ongoing evaluation of transmission risk in settings and the use of non-pharmaceutical interventions are necessary to help control the epidemic. This study aimed to produce a framework for evaluating transmission risk in settings where individuals gather and inform decision-making. Methods: A multi-criteria decision analysis process was used to structure the framework. Fifteen criteria were identified as important to consider for COVID-19 transmission risk based on the literature. This list was ranked by experts and then categorized. The analysis was structured by the consensus list of criteria and relative positioning of each criteria within the list to produce sets of factors to consider when assessing transmission risk at gatherings. Results: Fifteen experts from across Canada participated in ranking the criteria. Strong consensus was found on the relative importance of criteria and this relative consensus was used to create four categories: critical (3 criteria); important (6 criteria); good to consider (5 criteria); and if time permits (1 criterion). Conclusion: The resulting consensus list and categories constitutes a set of important elements that can be applied to any setting as an objective and transparent framework to assess transmission risk in the venue. In conjunction with further consideration of the local epidemiology of COVID-19, an overall risk of transmission assessment can be established and uniformly implemented.


Introduction
The emergence of the novel coronavirus severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) and the associated disease (coronavirus disease 2019; COVID-19) was initially observed as an outbreak in Wuhan, China in late 2019, and resulted in the ongoing pandemic (1). The virus was first detected in Canada in early 2020 and has caused 1,670,241 cases and 28,367 deaths (as of October 13, 2021) (2). The SARS-CoV-2 is a highly transmissible respiratory virus that can cause a range of symptoms from none to mild or severe disease and death (3). This has created an unprecedented disease management challenge for public health and numerous public health measures have been implemented with variable stringency in an attempt to slow the epidemic and reduce its impact. These include increased personal physical distancing and non-pharmaceutical interventions, such as case detection and isolation, tracing of contacts and quarantine and community masking (4), to reduce transmission opportunities in the community. However, when and where transmission is high, a range of restrictive closures have been imposed by provincial and local governments, including closures of schools, universities and non-essential businesses, bans or limitations on gatherings, limitations on travel within and between jurisdictions and encouragement of teleworking in OVERVIEW an attempt to limit transmission opportunities. Together these actions helped to minimize person-to-person contacts in Canada and resulted in the epidemic coming under control with low reported case incidence during summer 2020. However, the fall 2020 return to school and re-opening of businesses in many regions of Canada resulted in a resurgence of the epidemic and a second wave peaking higher in total cases, hospitalizations and deaths than the spring wave (2). As capacity for control of the epidemic by testing and tracing alone was surpassed, re-implementation of some levels of restrictive closures was deemed necessary to help reduce rates of contact between Canadians, regain previous levels of epidemic control and limit the risk of exceeding healthcare capacity. Closures of schools and businesses have important social and economic impacts on society. The challenge facing decision-makers is how to navigate the trade-off between preventing COVID-19 transmission and the negative potential health, social and economic impacts of restrictive measures (5)(6)(7)(8)(9)(10)(11)(12). A full assessment that includes negative impacts of closures is outside the scope of this work, as at this time the focus is restricted to informing assessment of transmission risk.
The arrival of vaccine doses in Canada in December 2020 raised hopes that restrictive measures could be eased. However, the recent emergence of new, more transmissible and in some cases more virulent variants of concern (VOC) meant that caution was and will be needed when lifting restrictive measures and re-opening businesses and places where people gatherparticularly until sufficient vaccination coverage and natural immunity of the Canadian population has been achieved. Even then, the capacity to inform decisions on restrictive closures will remain relevant with the continuing threat of immune escape VOC and the potential of waning immunity. This project began in late 2020, prior to vaccine arrival in Canada, with the aim to explore available evidence on COVID-19 transmission in different settings and contribute to informing decision-making around closures. Settings are meant to broadly encompass all locations presenting a transmission risk for COVID-19 that a decision-maker may wish to assess. These include transmission at private gatherings in people's homes, as well as transmission in public places such as schools, grocery stores, retail stores, concerts and bars among others. The explicit consideration of high-risk populations can also be included in the assessment. While the potential cascading effects of closures are numerous and still being studied, characteristics of a setting contributing to transmission risk remains independently important to evaluate, even as vaccination is being rolled out since the presence of variants of concern continues to pose an important transmission risk. The objective of this project was to produce a framework to assist in ranking settings by the risk they pose for COVID-19 transmission, and potentially identify areas where mitigating measures can be targeted to help reduce transmission risk in these same settings in order to help inform decision-making.

Methods
Multi-criteria decision analysis (MCDA) is part of a family of decision aid tools from the field of operations research and used in numerous sectors to systematically evaluate alternatives over multiple criteria (13). Multi-criteria decision analysis approaches help to structure reflection around a decision problem by allowing the integration of multiple types of evaluations and the highlighting of strengths and weaknesses of the alternatives under evaluation. Participatory MCDA processes generally consist of a multi-step problem structuring phase where the problem is defined, stakeholders identified, criteria defined and weighted, items to be ranked identified and evaluated over the criteria, followed by a decision analysis phase where the multi-criteria analysis is carried out along with a sensitivity analysis and interpretation of results ( Table 1). In this article, a "rapid and light" version of a participatory MCDA process was used to structure a framework for the evaluation of settings at risk for COVID-19 transmission, while taking into account time constraints of stakeholders and considerable data gaps in the literature. The objective was to identify criteria and indicators that would be most informative for assessing transmission risk in settings and produce a consensus ranking of these criteria by experts. The results of this exercise are presented along with a discussion of how the results could be used to help assess settings for transmission risk.
Four steps from a participatory MCDA methodology were adapted to allow for the construction of an expert-ranked consensus list of criteria that could be used as a decision-aid. A quick scan of the literature was conducted to search for broad Interpretation of results a Steps included in the "light" process are marked with an "X". Dashes, "-", indicate step not included b "Alternatives" in this context would be the settings (e.g. bars, indoor concerts, etc.) under evaluation factors contributing to COVID-19 transmission risk and produce a preliminary list of key criteria. The scan was conducted in an evergreen database of COVID-19 literature maintained within the Public Health Agency of Canada that compiles citations daily from seven databases. The search was used to draft an initial list of twenty-three criteria, and indicators for their rapid evaluation. A limited number of studies were available and consisted primarily of rapid reviews of reports where clusters had occurred with many of the early outbreaks reported having occurred before widespread use of public health measures. Preliminary criteria related to common factors present in settings where outbreaks had occurred. Droplet and aerosol transmission was thought to account for most transmissions and pointed towards elements favoring close contact in closed and crowded spaces as primary drivers of transmission.
The preliminary list of criteria was presented to a group of 62 provincial public health experts for review and comment. The list was condensed to 15 criteria, including 10 site and event characteristics, one participant-level characteristic and four potential mitigation measures ( Table 2). In order to keep the final number of criteria manageable, a number of criteria from the original list were combined (e.g. indoor/outdoor location and ventilation) and criteria to evaluate secondary activities at settings were not included (e.g. shared dining or break rooms). This list was then presented to a group of experts within the Pan-Canadian Public Health Network involved in the COVID-19 response, for ranking during December 2020 via an online tool created explicitly for this purpose. The individual expert-ranked lists were combined using general Mallows models (14) to produce a consensus ranking (see Appendix for more details on the general Mallows models). The R package PerMallows (15) was used to analyze the rankings. -Random mixing events (e.g. public transport, bars and sporting events) are higher risk because of mixing from many small networks.
-Could also include settings where exposed to multiple clients (for example transport workers, sales people, cleaners). A review of workplace related transmission risk found that drivers and transport workers, service and sales workers, food industry, personal care occupations, food production, preschool occupations, community and social services, construction and related trades occupations and public safety workers were most at risk of infection (these groups are highly exposed to random individuals/clients in their line of work).
Ski resorts due to their attraction of global travellers (26)(27)(28) Mixing of participants (circulation and mixing of participants within the event) 1. None 2. Moderate

High
In a review of outdoor transmission events, crowding (number and density) was a common factor among outbreaks, but circulation (mixing) of participants, close range interactions with loud conversations, shouting or singing and duration were important factors.  Large crowd size increases the probability that an infected individual is present, increases crowding, contact and thus transmission likelihood, even in outdoor settings.
Of fifty-five studies reviewed in recent PHAC evidence brief, clear relationship found between increased gathering size and risk, but size threshold was inconsistent. When physical distancing breached, density is high, circulation of participants occurs and gathering takes place over extended duration of time, risk of transmission increases.
An ecological study estimated a 36% reduction in R o if the cut-off for gathering size was 10 people, compared to 21% if it was 100 people, and a 2% reduction in R o if the cut-off for gathering size was 1,000 people. In an evaluation of NPIs at a global scale, Esra et al. estimated overall 10% reduction in infections associated with gathering size restrictions.
In indoor environments in particular, larger numbers of individuals increases the potential concentration of airborne virus-carrying particles and number of individuals that can be exposed at any given time.

Results
Fifteen characteristics (i.e. criteria) were presented to a set of experts for ranking from highest to lowest level of importance when evaluating transmission risk of COVID-19 in various settings. Fifteen experts returned rankings by the deadline, and while the number of respondents was low, the respondents represented the geographic regions in Canada most affected by COVID-19 at the time.

Consensus ranking by means of generalized Mallows models
A generally good consensus emerged among experts on the relative importance of the criteria with some individual variations in specific ranking positions (Figure 1). Table 3 shows the rankings created by each participant. The consensus ranking resulting from the generalized Mallows models with the Kendall and Hamming distance is shown in Table 4. While there was broad agreement between the two consensus rankings, differences emerged as a result of wider variation in respondent rankings for some criteria.

OVERVIEW
Location and ventilation and ease of contact tracing were the criteria on which experts most strongly agreed in terms of absolute rank ordering (i.e. most important and least important criteria). Location and ventilation appears exclusively in the first seven ranks and almost always (n=14/15 times) in the first four ranks. Criteria with stronger disagreement among experts regarding absolute rank position were number of households or individuals and level of expelled air.

Criteria categorization based on combined consensus rankings
Criteria were placed into categories based on their level of agreement between experts and combined Kendall and Hamming rank orderings (Table 4). Four categories were created as a result of these ranking agreements among experts: "critical"; "important"; "good to consider"; and "if time permits".

Categories 1 and 2: Critical and important criteria
The first set of criteria ("critical") consisted of three criteria that were consistently ranked within the first few positions by experts: 1) density of the crowd; 2) contact between participants; and 3) location and ventilation.
The second set of criteria ("important") were almost consistently ranked within the top half of the ranking by experts (with the exception of level of expelled air), though some variability in specific rank positioning was observed among experts: 1) number of households; 2) level of expelled air (of activity); 3) duration of the event or activity; 4) use of personal protective equipment; 5) mixing of networks; and 6) mixing of participants. While level of expelled air showed wide variation in expert ranking (present in the top half for some experts and the bottom half for others); given available literature data on this criterion, it was placed in the "important" category.

Categories 3 and 4: Good to consider and if time permits criteria
The third set ("good to consider") consisted of five criteria. Experts generally ranked these criteria in the lower half of their rankings, though relative importance attributed to each criteria varied between experts. This set included the following: 1) the use of engineering controls and environmental cleaning; 2) related activity; 3) administrative scheduling; 4) age structure of participants; and 5) shared equipment or surfaces. Finally, given very strong agreement for the rank positioning of the last criterion, ease of contact tracing, which was almost consistently ranked last among experts, this criterion was placed in the "if time permits" category.

Discussion
This project aimed to identify key factors (criteria) to consider when making decisions around COVID-19 transmission risk in various settings where people gather. The use of generalized Mallows models allowed for the analysis and quantification of the consensus among experts on the rank importance of different transmission risk factors (criteria). A lack of approximate consensus on a given criterion can lead to large differences between models with different metrics; however, using the Kendall and Hamming metrics, broad consensus was found among the most important and least important criteria.
The consensus-ranked list of transmission criteria and corresponding categories resulting from this exercise contribute to a framework for ranking settings for COVID-19 transmission risk based on criteria identified from both the literature and expert opinion. How a setting is evaluated or scored with respect to a specific criterion will depend on a range of factors specific to the local community where the evaluation is undertaken, including local transmission, public health measures in place, current adoption of those measures by the local population and setting-specific characteristics.
Although this framework is intended to assist in evaluating transmission risk, all risk assessments should be performed in the context of the local epidemiology and with consideration of the specific characteristics of the gathering/event/venue being evaluated. A ranking of transmission risk of settings produced in one geographical location will not necessarily be the same as that performed in another geographical location due to local epidemiological variation even if the same criteria are used.
Based on the formative research conducted, the consensus list captures elements that are most directly related to transmission risk. When evaluating settings and their risk for transmission, it is important to keep in mind the activities performed on site and their related contexts since related activities may affect transmission risk; e.g. shared transport to the setting or shared accommodations. These related activities may present additional opportunities for transmission that may be important to consider for inclusion in the risk assessment.
Many of the criteria presented are inter-related or synergistic and, as a result, may be difficult to evaluate individually (e.g. the number of participants at an event and the density of the crowd). The use of scenarios may help tease apart some of these factors. For example, a scenario could be defined to evaluate transmission risk when a certain percentage of the population is vaccinated, and a separate scenario defined to consider a different target vaccination percentage. Alternate scenarios could be defined to consider different levels of community transmission as local prevalence of COVID-19 will change the likelihood of encountering an infected individual. Expert review and discussion of the evaluations is also important as it will promote cross-examination and consideration of a broad set of local factors.
Operationalization of this list and resulting categories is left to the discretion of regional public health experts, though some suggestions are discussed. Assessment as part of a multi-step MCDA process would enable a systematic evaluation between settings; however, a full illustration of this approach is beyond the scope of this article.

Variant of concern considerations
Transmission risk evaluation will continue to be necessary until sufficient vaccination coverage can be reached to achieve herd immunity. With the emergence of VOCs across the country, additional waves of cases may continue to threaten healthcare capacity in Canada despite vaccine rollout. As such, our experts were consulted once again in March 2021 to see if their rankings of the criteria would change given the emergence of VOCs. Given a lack of evidence that VOCs affect transmission risk differently, the experts that responded (n=10/15) left their rankings unchanged. This should be monitored as further knowledge is gathered on the subject.

Using the criteria for evaluation of settings and gatherings and limitations
Although a full multi-criteria evaluation of settings is outside of the scope of the current paper, some guidelines on the use of the criteria for evaluating settings are suggested below. Software packages exist for MCDA analysis, including a recently developed free package for R (https://cran.r-project. org/web/packages/MCDA/MCDA.pdf) and other academic or paid software options. These software packages allow MCDA analysis without the need for statistical experts to carry out the evaluations.
Before undertaking an evaluation of settings, the scope and scale of the assessment should be clearly defined. For example, it is important to define whether the evaluation is being performed to assess the daily exposure of individuals to transmission at any given setting versus assessment of the daily exposure of individuals working all day at a setting since a setting may pose different risks to a casual visitor versus an employee who is exposed over several hours. It is important to consider whether specific subgroups are to be considered in the assessment; for example, are clinically-vulnerable individuals included in the scope or is a separate assessment required to properly assess this subgroup? As settings are considered for inclusion, creating a description of the setting in the context being assessed is useful (e.g. in a grocery store-where a typical visit generally lasts approximately 15 minutes-masks are currently mandatory). Additional variations of settings can be added to assess variations that may be relevant to consider (e.g. variations where mask use is not mandatory, etc.). A quick review of the criteria should be undertaken by the experts participating in the process to assess whether all proposed criteria remain relevant to the local context (e.g. a criterion for which all settings have the same score is not discriminating and may be omitted from the evaluation).
Depending on the data available to a decision-maker for assessing settings, and taking into account levels of uncertainty, variability and missing data, the essential and important criteria should be evaluated where possible as a first level of assessment of COVID-19 transmission risk between settings. Expert judgement and opinion can be used to fill in missing data. If sufficient information is available or can be appropriately assessed by experts, a more complete MCDA-style assessment of settings can be undertaken. A systematic evaluation process, such as offered by an MCDA evaluation, can be used to better understand the relative transmission risk between settings and, in particular, to highlight the strongest contributing factors as well as strongest protective factors for transmission risk between settings. This type of evaluation could help inform where mitigation measures should be considered to help reduce transmission risk. A setting that has criteria that score as poor or insufficient should be considered for mitigation and potential monitoring of transmission risk. As previously suggested, the use of scenarios can also be used to consider the changing epidemiology and its impact on transmission. For example, scenarios with different levels of vaccination, new levels of dominance of a VOC and levels of community transmission can be defined and used to evaluate how they may affect relative transmission risk of settings.
Depending on the data available and levels of uncertainty around these data, any resulting ranking will not represent a strict absolute assessment or ranking of settings, but rather a working local evaluation that reflects the information available and the relevant experts participating in the process.
As a reminder, the use of this framework is meant to help inform decision-making around transmission risk rather than make decisions, since other factors should be considered in a decision-making process around closures. To conduct a more complete assessment of closures/re-openings, additional dimensions beyond transmission risk factors alone such as social considerations, economic and other health factors could be considered for inclusion in a multi-step participatory MCDA process.

Conclusion
This project drew upon the latest evidence concerning transmission risk factors for COVID-19 in venues from which criteria for the evaluation of transmission risk was developed and then evaluated by experts. The resulting consensus list constitutes a set of important generic elements that can be applied to any setting as an objective and transparent framework to assess transmission risk in the venue. With further OVERVIEW consideration of the local epidemiology of COVID-19, an overall risk of transmission assessment can be established. This work focused on the factors most directly related to COVID-19 transmission as a first level of concern in evaluating settings. Depending on the decision-making context (e.g. decisions around closures or re-openings) additional factors should be considered for inclusion in the decision-making process, including economic and social impacts. Additional layers of information could be added to the participatory MCDA process to include economic, social and health criteria so that tradeoffs could be more fully examined, allowing for more informed decisions by decision-makers around closures and re-openings to reduce the transmission risk of COVID-19.
Imagine that we have a set of N rankings over n choices. In our case, N represents the number of experts and n denotes the criteria (items). The problem is to find the consensus ranking among the experts, which best agrees with the N rankings offered by the experts.
Consensus ranking finds application in social welfare analysis.
In 1950, Kenneth Arrow showed (55) that if a decision-making body consists of at least two members with at least three options to decide among, then it is impossible to design a social welfare function that simultaneously satisfies all the reasonable requirements of a fair system: • If every voter prefers alternative X over alternative Y, then the group prefers X over Y • If every voter's preference between X and Y remains unchanged, then the group's preference between X and Y will also remain unchanged (even if voters' preferences between other pairs like X and Z, Y and Z, or Z and W change) • There is no dictator: no single voter possesses the power to always determine the group's preference Arrow's impossibility theorem has several technical conditions in its formal statement (see section A3) which defines the "fair" system. While the theorem tells us that no deterministic preferential voting system exists which satisfies the technical fairness requirements, in practice all systems do not work poorly at all times. The impossibility theorem finds application in the study of voting systems and important results can be found in (14,56).
Rankings consist of bijections of the set of integers {1,2,3, ..., n} onto themselves. We will denote rankings with the symbols π and σ. For example, the ranking π ={2,4,1,3} means that item 1 is ranked second, which we denote as π(1) = 2; item two is ranked fourth, π(2) = 4. Every ranking has an inverse π -1 which gives the items in terms of the ranks: π · π -1 = e = {1,2,3, ..., n}. Given a set of rankings, we would like to find the center or the consensus of the set over some distance measure between rankings. There are many distance metric for rankings, but in this article we will focus on two of the most popular: the Kendall distance and the Hamming distance. For any distance measure d(·,·) we have d(σ,π) = d(σπ -1 , e). When the reference ranking is the identity ranking e, we use the notion d(σ,e) = d(σ).
The Kendall distance between two ranking π and σ is defined by, Equation 1: The notation l ≺ π j means that item l precedes j in ranking π. The Kendall distance counts the number of pairwise discrepancies between rankings. With n items, the largest Kendall distance between any two rankings is n(n -1)/2. On the other hand, the Hamming distance d h (π, σ) counts the number of positions that disagree between two rankings, OVERVIEW Given a metric for computing distances between rankings, we can build a probability measure over the space. Mallows model (14) is an exponential location probability model over the rankings defined by a central ranking, σ 0 , and a dispersion parameter, θ, namely, where ψ(θ) is a normalization constant. In a sense, the Mallows model is the extension of the Gaussian distribution to rankings. When θ > 0, the ranking σ 0 is the mode of the distribution-the consensus-and as θ increases the distribution becomes more sharply peaked around σ 0 . If θ > 0, σ 0 becomes the anti-mode.
We see that in the Mallows model, all rankings with the same distance from σ 0 are degenerate in probability. With distance measures that decomposed as a sum like those in equation 3, we can break the degeneracy by attaching θ j to each component of the sum (57) where θ = (θ1,θ2, ⋯). The central ranking σ 0 and θ can be estimated by maximum likelihood or other approximate techniques.
The value of V j (σ) in equation 3 gives the number of items in j + 1:n which are ranked before j in σ. Therefore, the parameters θ j reflect the strength of a ranking around the consensus σ 0 (j) = i in that the larger θ j the larger the probability that π (j) ≤ i. That is, large θ j in the generalized Mallows model with the Kendall distance implies that item j is ranking in the first i positions with high probability across all the rankings. Similarly, H j (σ) of the Hamming distance counts the rank discrepancies. Thus, the parameter θ j corresponds to the strength of consensus at rank j; large θ j implies high agreement on the item at the j-th rank.

A2: Hamming parameters
To better see the strength of consensus in the rankings across items, the Hamming θ j parameters against the cross-entropy of a criterion's rank is shown in Figure A1. The cross-entropy measures the amount of impurity in the ranks, where j is the item label, i denotes the rank, and p ij gives the probability of the i-th rank for item j. Criteria with large rank dispersion have high cross-entropy. The Hamming θ j parameters also measure the strength of consensus at a given rank. Figure A1 shows criteria separating into three basic clusters with increasing consensus appearing towards the southeast corner of the plot. Under the Hamming model, the respondents have particularity strong agreement on the criteria at rank 1, 2 and 15.

A3: Formal conditions of Arrow's Impossibility Theorem
Suppose that we are asked to extract a preference order on a given set of options for society. Each individual provides a preference order on the set of outcomes. We desire a ranked voting electoral system, the preference aggregation rule or social welfare function, which transforms the set of preferences into a single global societal preference order. Arrow's theorem says that if there are at least two members in the society and at least three options to decide among, then it is impossible to design a preference aggregation rule that satisfies all of the conditions below at once (conditions assumed to define a "fair system"): •