The value of mass-produced COVID-19 scenarios: A quality evaluation of development processes and scenario content

Hundreds of scenarios were developed across the world in 2020, aimed at generating forward-looking conversations, better understanding for COVID-19 transmission rates, trialling economic outcomes, and stress-testing existing systems in light of the developing pandemic. In response, Cairns & Wright (2020) questioned the value of these mass-produced scenarios created retroactively to existing crises. We address their concerns by evaluating 213 COVID-19 scenarios developed in the first wave of the pandemic. We use two yardsticks as guiding maps against which we plot each scenario's profile and test for values of high-quality process and content. Our analyses reveal various points of high and low qualities, in both process and content. Though most reported processes fell towards lower quality standards, and content largely carried generic applications, the prolific levels of exploratory narratives reflected a mixture of high and low-quality values. Together, our papers develop and reinforce the message that scenario interventions, especially in times of crisis, should reflect more proactive efforts and ensure powerful stakeholders, decision-makers, and affected community members are included in the development of scenarios.


Introduction
Scenario planning has been described as "a Swiss pocket knife of multiple users, or a magic wand that is often waved by inexperienced and unskilled consultants and professionals" (Masini and Vasquez, 2000, p. 49). As it stands, most techniques discussed in the extant literature are highly prescriptive in nature. While the practitioner literature refers to scenario planning as a practice in expertise, Whaley (2008, p. 310) counters that the "hard facts of what is done to create the scenarios, what data [are] processed and how" is not usually discussed. Too many scenario development techniques in the literature are poorly defined, impractical, contradict each other, and lack theoretical justification and/or adequate testing (Cairns and Wright, 2020;Varum and Melo, 2010). These issues may be non-starters if, as many suggest, scenario development is a relatively simple and straightforward task. Yet, such suggestions distort "the considerable skills required by its practitioners" (Grinyer, 2000, p. 32). As Van Asselt et al. (2010), p. 11) discussed, basic procedural descriptions in the literature fail to report the "choices, considerations, discussions, struggles, compromises, unproductive steps, flaws, practical adjustments, experiments, difficulties, challenges and local solutions". Chermack (2003) appears to recognise similar by remarking how practitioners are becoming more explicit about the importance and intentions of their scenario approaches, compared to the nature of the practice half a century ago.
Even considering common transparency pitfalls in the literature, several reviews have discovered similar themes regarding the application of scenario approaches. Varum and Melo (2010) found consensus in the literature on three benefits to a scenario approach, namely improvement of the learning process, identification of new issues and problems, and improvement of the decision-making process. Wright et al.'s (2013) review of the Intuitive Logics literature concluded with three main objectives similar to Varum & Melo's discovery: 1) Enhancing understanding: of the causal processes, connections and logical sequences underlying eventsthus uncovering how a future state of the world may unfold 2) Challenging conventional thinking: to reframe perceptions and change the mindsets of those within organizations 3) Improving decision making: to inform strategy development.
Where Wright et al.'s (2013) three objectives are interlinked, we find echoes of the same sentiment in Docherty & McKiernan's (2008, p. 10) earlier work, "the greatest contribution of scenario planning lies in its active engagement of actors in its process and its power to enable them to think about complexity and uncertainty in external contexts, and then how they might shape the external environment to their own strategic ends".
By contrast, recently, Cairns and Wright (2020) reflected on, what they saw as, the fast-paced "mass production" of scenarios generated in response to the 2020 COVID-19 pandemic. One conclusion they reached was that such "quickly-produced scenarios are not embedded in the realities of affected communities" (p 1). Others have also reflected on similar issues, bringing into question the value of scenarios created retroactively, in response to contemporary crises (Inayatullah, 2009;Millett, 2003). However, none of these conceptual studies included a systematic review and analysis of extant scenarios. In this paper, we evaluate a large sample of scenarios that were created during the first six months of the 2020 COVID-19 pandemic. We take an in-depth, evaluative, look at the quality of both their scenario development processes used and the scenario content produced across 213 scenarios. Are the underpinning scenario development processes generally of high quality and, if so, how is this quality reflected in the content of the resultant scenarios? If the development processes and content are not, generally, of high quality then in what ways do particular scenario activities fall short? Can situations in which both high-and low-quality scenario activities occur be linked to enabling conditionssuch as the type of organizational sponsor or underpinning resource for the activity? To make this evaluation we apply two yardsticks, one focussed on the quality of the scenario development process and the second focussed on the quality of the resultant scenario content.

Evaluating quality
Three propositions are developed in Cairns and Wright's (2020) paper, with respect to then-burgeoning corpus of COVID-19 scenario sets. First, that these scenarios are largely artifacts reflecting the realities of the time. They are, in essence, frozen pictures of the practitioners' then-knowns and then-unknowns and fail to engage, instead, an "ongoing refinement and adaptation of perspectives on the future" (p 2). Molitor (2009, p. 81) goes so far as to claim that such reactive scenario developments "merely reinforce what participants already basically knew," concluding that the effort is little more than a parlour game.
Second, Cairns and Wright propose that the scenarios take a global perspective, failing to incorporate information that is valuable at the local level of governments or communities.
Third, Cairns and Wright propose that the mode of scenario delivery is to a general audience, failing to either engage or address the subtleties of particular affected communities. The developed scenario sets lack the flexibility to illustrate and understand the impact of plausible local actions in response to unfolding events. Specifically, self-interested actions of the powerful, such as governmental actions and directives to promote social isolation, i.e., lockdown and social distancing, are omitted from scenario storylines and citizens' reactions to these directives are also missed.
Overall, several foci for our empirical investigation emerge from our discussion of the literature, above.
Regarding quality process: high quality process will be indicated by: (i) inclusion of affected stakeholders in the scenario development process, rather than desk-based "arm's length" development of scenarios separate from affected communities (ii) application of a replicable, defensible, structured development method (iii) ongoing refinement of the scenario storylines, as unexpected events emerge in real time Regarding quality content: high quality content will be indicated by: (i) development of several scenarios rather than a single scenario (ii) development of scenario storylines that include interactions of driving forces from across the PESTEL dimensions and incorporates the self-interested actions of powerful stakeholders to unfolding scenario storylines (iii) inclusion of implications for action by those communities affected by events within the scenario. These communities are also, potentially, the clients for the scenario development exercise.
Our main question asks whether quickly generated, mass-produced COVID-19 scenarios provided value to the affected communities for which they were developed. We propose to answer this by evaluating the profiles of COVID-19 scenarios produced within the first six months of 2020 (i.e., "the first wave"), then use features of their profiles to help determine the value of the scenario sets to their target communities and organizations. Based on our discoveries, we will offer guidance for improvements in the quality of future scenario planning efforts through illustrations of high quality COVID-19 scenarios and critical discussions on low quality scenario features. Our recommendations will help provide more robust methods for measuring value and impact.

CSI framework
The Comprehensive Scenario Intervention (CSI) typology will be used to guide our review and analysis of COVID-19 scenarios (Crawford, 2019). We chose this tool because it provides a systematic guide for identifying and working with structural and qualitative uncertainties. Uncertainties, in particular, that become exacerbated during times of global disruption. The CSI typology is divided into four overarching thematic sectionsproject goals, process design, scenario content, and scenario impact. Each section is divided into levels of sub-sections, creating over 100 possible dimensions by which to profile a single scenario. Based on the aims of the paper, we will focus on a distilled version of the CSI typology as a guiding map against which we plot each scenario profile. The distilled version includes CSI dimensions that develop two yardsticks for COVID-19 scenario planning evaluation, one for quality process and another for quality content.

Quality process
Analysing qualities of scenario planning processes, we look at the CSI typological dimensions of practitioners involved in the process, the role of the decision-makers played, and the information sourced to build the scenarios. Along with these three dimensions, we included an additional dimension relevant to pandemic-conditions. We determined whether scenarios were revisited by the authors/practitioners for updating.
Who created the scenarios? A concern was that scenarios were developed by "external agents who are not embedded in or acculturated to the communities that are the subject" (Cairns and Wright, 2020, p. 2) The CSI typology recognizes seven common types of agents who could participate in scenario planning. An external agent can be considered an "expert", such as an industry expert. If not an expert, participating agents can also be facilitators, problem owners, employees, stakeholders, community, and cross-populations. Facilitators are often scenario experts who guide the process and keep the project on track. Sometimes they may participate in the process, but largely serve as experienced guides or agents. Problem owners are those who hold responsibility for the outcome of the intervention. Employees, stakeholders, and community members are all sourced locally, whether from within the target organization or affected region.
To gain insight into the type of audience for which COVID-19 scenarios were intended, it is important to know whether "decisionmakers" from the effected communities participated in the process.
These people may have been general practitioners, organizational or sectoral members, representatives, or a collaboration of representatives that involves scenario practitioners consulting with industry experts, but not necessarily including them in any scenario planning (Crawford, 2019).
Information can be gathered through several methods. Participatory methods are active and include interactive sessions with other practitioners, often group-based. Sessions can include interviews, brainstorming sessions, think-tanks, group discussions, surveys, workshops, Delphi-style ranking scores, incasting, role playing, storytelling, intuitive logic, visioning, and/or focus groups (Bishop et al., 2007;Dator, 2009;de Vasconcelos et al., 2012;Grevi et al., 2013;Teufel et al., 2013;van de Riet et al., 2008). Desk research, on the other hand, is a passive method, that can include literature research, data mining, clustering, and computer simulations, and modelling (Allington et al., 2018;Johnson and Sieber, 2011;Van Notten et al., 2003). As well, any combination of information or data collection can be employed for scenario planning.
The rapid pace and quantity of new information that continuously emerged during the first wave of the pandemic left little time for sustained strategies. As many have stated, each scenario is not considered an end in itself, but rather serves to highlight crucial uncertainties, and through the active engagement in scenario planning (that results in multiple scenarios) the quality of executive decision making can be improved (O'Brien and Meadows, 2013;Postma and Liebl, 2005;Wilson, 2000;Wright et al., 2018). Repetitive revisits to previously published scenarios help any resulting strategies keep pace with rapidly emerging knowledge and changing landscapes.
There are certain methods that have shown to produce higher quality scenario planning, as measured by clarity and confidence of practitioners, post hoc, and articulated action within affected communities (Cairns et al., 2016;Cairns et al., 2017;Kuhn and Sniezek, 1996;Ö nkal et al., 2013;Phadnis et al., 2014;Schnaars and Topol, 1987). Suggestions are given for conducting scenario planning as interactive group sessions that consult a heterogeneous group of practitioners who bring a variety of knowledge and expertise that can be challenged. Decisionmakers from the effected communities should be involved at some stage in the process, to increase the chances of articulated action, and by extension, impact. This last feature is particularly important in a time of great disruption, like the pandemic, when information is rapidly changing and barriers to communication can cause equally rapid failures, both for human lives and systems.

Quality content
Analysing qualities of scenario content, we look at the CSI typological dimensions of the quantity of different scenarios produced in a single publication, scenario application, the values they reflect, temporal nature and complexity, and the nature of the information reflected within the scenarios. Additionally, we try to determine where scenarios include implications for actions, particularly for their target audiences.
The quantity of different scenarios developed around various COVID-19 futures speaks directly to the variety of perspectives resulting from the process. Some organizations developed a single scenario, reflecting a single theme, some developed dichotomous scenarios that reflect simple 'good vs bad' themes, while other authors developed a broad selection of niche scenarios. Developing scenarios that imagine different futures has arguably higher value during times of great disruption, since it is in these moments that our list of known knowns diminishes, and critical uncertainties expand. Cairns and Wright (2020) raised the concern that first-wave COVID-19 scenarios may have been too global in nature and broadcasted to a general audience, rather developed within effected communities. The CSI typology allows us to explore this feature, in part, through the application characteristic. How will the organization want to apply the outcomes of the intervention? Scenarios developed for general use are considered generic. Generic scenarios can help inform subsequent, more focused, and specific analysis of the implications for particular sectors or organizations, including further scenario iterations. Scenarios developed with a defined focus are considered specific by design, e.g., contextual developments that will affect a particular industry sector, such as hospitality or tourism.
To help understand if COVID-19 scenarios were addressed to general audiences, we will analyze the intent in the process. Normative scenarios focus on futures that reflect the desires, interests, and motivations of the practitioners or their intended audience (van de Riet et al., 2008). These scenarios can include prospective, strategy, policy and intervention scenarios (van Notten et al., 2003). By contrast, descriptive scenarios present hypothetical futures with little to no prior accounting for levels of desirability of those outcomes. When scenarios incorporate descriptive, hypothetical futures with normative desires in equal measure, they are present dynamic narrative. Scenario information, regardless of style, will include specific implications for action for affected communities, or these passages will be absent.
Within a particular timeline, the treatment or understanding of unfolding events that propagate each scenario may be detailed or cursory. Snapshot scenarios present outcomes at a single point in time (Biggs et al., 2007;Godet and Roubelat, 1996). Chain scenarios present a continuous storyline that includes developing relationships between events in the scenario storyline. Varied scenarios present a zoom-in-andout effort, snapshots offer in-depth attention to an evolving future at a particular point-in-time, as a scenario continues to unfold. Understanding the temporal view within a scenario is not meaningful information, on its own. Higher quality scenarios can take in-depth, explorative perspectives of a single moment in time (i.e. snapshot) and/ or elaboratively woven cause-effect connections that unfold across congruent timelines (i.e. chain or varied). The key to quality measures is the level of elaboration with in the scenario timeline. When combined with the complexity dimension, richer analyses can be applied. Organizations have more variables to work with for later impact analyses, stress testing, and policy development. High complexity scenarios tend to be those that present elaborative developments of relationships between factors, possibly crossing several disciplines (Crawford, 2019). Low complexity scenarios are more simplistic narratives, with fewer factors incorporated into them. Complexity overlaps with the data dimension. Information utilized in the process and presented within the scenarios can be either quantitative (e.g. percentage changes in projected GDP) or qualitative in nature (e.g. intuitive interpretations of the impacts between driving forces).
As with process, the presence of specific scenario features is more likely to produce higher quality scenarios. Qualities are measured in such behaviours as in-house repetitive references and applicability to ongoing strategic dialogues, including subsequent scenario planning (Kahane, 1992;Ratcliffe, 2003). Scenario features relevant to this paper's analyses are elaborated below. Producing more than two scenarios can help the scenario team break from tunnel vision perspectives that often dominate perceptions and can lead to more normative scenarios, at the sake of exploratory ones that challenge assumptions. Aiming for more than two also helps break practitioners from the "all bad" vs "all good" dichotomy, for reality is rarely, if ever, "all" one quality. Scenarios that include targeted narratives where the interests and textures of affected communities are reflected can lead to efficacious efforts, for example from policy makers, that account for real stakeholders as opposed to conceptual targets (Lehoux et al., 2020). Higher complex scenarios are generally the outcome ofand requirement forachieving targeted narratives, whether as an in-depth snapshot of a single event acting upon a group or organization. They will incorporate, at minimum, qualitative data in the form of interactions of driving forces from across several dimensions. Increased complexity in scenarios could include additional elements such as quantitative elements like trends and forecasts blended with imagination and causal analysis (Inayatullah, 2009).

Method
We used a mixed-method approach to transform qualitative data and gain richer understandings of the scenarios in our dataset. To begin, a framework of dimensions was developed from a distilled version of the CSI typology. Quality process encompasses the dimensions of practitioners, involvement of decision-makers, information collected, and scenario revision. Quality content encompasses the dimensions of quantity, application of value/reality, temporal nature and complexity, style of information presented within scenarios, and presence of actionable implications.
While building each scenario's profile, three dimensions were added alongside the CSI typology, to help fully address Cairns and Wright's (2020) original propositions. The first dimension expands the practitioner characteristic with the category Arm's length, which encompasses consulting professionals, journalists, and scenario authors who did not fit within the available categories. We designated this group Arm's length because these authors worked independently of the organizations and stakeholders targeted by their scenarios, were not identified as experts in the industries, fields, or sectors their scenarios encompassed, nor were they scenario experts facilitating a workshop or intervention. The second dimension identifies publishers who Revisited their previously published scenarios. The third additional dimension evaluates implications for actions, particularly for stakeholders and affected communities. Table 1 presents the framework of dimensions used to develop each scenario's profile, which is a distilled version of the fuller CSI typology. Overall, our framework allows for a 11-point profile to be developed for the process and content of each analyzed scenario.

Search criteria
Our aim was to locate as many scenarios as possible within the available timeframe. The search effort took place between May and September 2020. Initially, the date range of COVID-19 scenario publications were set at January -June 2020. However, this date range was later expanded to allow for related scenarios published earlier than January 2020, i.e., before the current COVID outbreak.
COVID-19 scenarios were first sourced from an existing repository of scenarios, as part of a then on-going data collection initiative at Strathclyde Business School. In conjunction with their repository, we conducted searches using Google, government sources, contacting public and private organizations, and re-establishing existing connections within the research team (that resulted in a few scenarios being emailed directly to the team). For online searches, the terms "scenario" or "scenario planning" were crossed with "covid", "covid19", "covid-19", "covid 19", "coronavirus", "pandemic", and/or "health crisis". Results were initially collected without curation. Scenarios published prior to January 2020 located through our second criteria included any combination of the search terms "coronavirus", "pandemic", "health crisis", and "2020", along with any the scenario search terms.
During the search phase, we discovered that some online publicly accessible scenarios (and their related information) had been replaced with updated scenarios and/or current COVID-19 information in a way that appeared to delete prior published work. To increase the probabilities that we accessed any potential past and present COVID-19 scenarios from the same sources, we emailed authors as well as used the Internet Archive, a constantly updated digital library of internet sites and other cultural artifacts in digital form, by use of the Wayback Machine (1996). The resulting dataset included 64 publications from 46 organizations that produced a total of 262 potential scenarios.

First round
The first round of curation selected original scenarios, including those secondary sources that reported others' original scenarios, and deleted repeated publications. We started with Spaniol and Rowland's (2018) definition of a scenario to separate publications that included scenarios from those that included scenario-like content, but were otherwise not scenarios. Spaniol & Rowland provide a summary table and flowchart to be used individually or together as diagnostic tools to identify scenarios. Their criteria for a publication to be a scenario are future oriented, reference external forces, a narrative description, possibly plausible, a systematized set, and comparatively different. In practice, we looked for publications within our search results that included future-focused scenarios with any level of horizon (short-to long-term), as opposed to past/present focused. The latter publications were largely concerned with exploring scenario-relevant elements such as key uncertainties and driving forces of the moment, exit strategies, Note: We replaced the Data category "Complimenting" from the original CSI typology with the label "Mixture (qualitative and quantitative)" to relay the meaning behind our analyses more clearly.
promotional materials, and past-to-present trends. 1 Locating scenarios that referenced external forces was a less arduous criterion to search for, given that all scenarios referenced COVID-19 as an active or passive force impacting any number of realities and actors within a structure (social, political, organizational, or regional). Narrative elements were required with each scenario, whether a brief paragraph or several pages of story-telling with charts and other supporting visual elements. Pure modelling forecasts absent of narratives were removed. 2 The last three criteria in Spaniol & Rowland's definition were given the greatest allowance in the curating process. Scenarios are necessarily fiction, however, as long as the publication referenced known-knowns (e.g. the target industry, organization, and stakeholders) and stated causal relations that could be followed, then it was considered possibly plausible (Ramirez and Selin, 2014;Spaniol and Rowland, 2018). We did not want to bias our dataset by eliminating stand-alone scenarios due to the regular occurrence of single publications, therefore, we expanded the "systematized set" criterion to include sets of one. By extension, this meant stand-alone scenarios lacked comparability with simultaneously published scenarios. First-round curation reduced the dataset to 43 unique publications from 42 organizations that produced a total of 216 scenarios, where 17 % were from in-house publications, the remaining from publicly accessible websites. In the case of the Economist Intelligence Unit (EIU: 2020), each scenario in a single month's release was published with a unique URL and not explicitly linked with the other scenarios published in the same month. However, we profiled their scenarios as a systematized set because they provided a summary table of Global Assumptions and a summary list of key changes for the month's collection of scenarios. The rest of our sourced organizations published all their scenarios from a single session into one publication. Our dataset resulted in.

Second round
The second round of curation applied and developed our 11-point profile for each scenario. As discussed, COVID-19 scenario profiles were constructed from the selection of CSI typological categories that best described the scenario content. Each scenario profile reflected only one indicative major categorical dimension, even though it was sometimes possible for some scenarios to cross more than one category.
Coding was conducted by two independent raters. The purpose for using independent raters was to help decrease the chances of bias in analyses by the authors. The raters coded all scenarios across the CSI dimensions. Each rater was familiar with scenario planning, but unfamiliar with the present research and the authors' previous publications (i.e. Mass Production of Scenarios and CSI typology). The raters were not informed of the propositions, nor any previous coding outcomes by the authors' earlier work. The raters were instructed on the selected CSI dimensions, their definitions, and all questions were answered by the authors. Each rater completed their coding task individually, then met as a group for several rounds of revisions until consensus was reached on all dimensions. The authors were available only to help clarify any ambiguities in the typological dimensions. The raters also recognized that one publication with seemingly three scenarios was more accurately recognized as one continuous scenario and one publisher who appeared to have produced four scenarios, only produced three. The revisions resulted in the same number of unique publications and organizations, but with 213 full, independent scenarios. Some CSI characteristics have mutually exclusive categories. The target audience's decision-makers either participated in scenario planning or they did not. Scenarios were either revisited by their authors, or not. This was determined through a variety of methods, including web scraping of previously visited publishers, contacting authors, and use of the Wayback Machine (1996). We also attempted count the number of revisions. This was only pursued for organizations that were confirmed to have revised or revisited their earlier COVID-19 scenarios. The quantity of scenarios produced in a single publication was profiled with a single integer, n = {1, 2, 3… i}. A scenario's application was either aimed towards a general audience or was designed for a specific sector or organization. Scenario content was either highly elaborativesharing dynamic, mixed data and verboseor presented a simple narrative. Implications for action for stakeholders or effective communities was either present or absent.
The remaining characteristics have the potential for overlapping categories. Practitioners can represent any combination of the seven separate homogeneous groups who could have participated in developing the scenarios. When practitioners from more than one group developed COVID-19 scenarios together, the scenario profile assumes a cross-population dimension. When a scenario presented a zoom-in-andout effort, where causally linked developments of events were occasionally paused along the timeline to serve as checks or moments of focused development in the storyline, it was categorised as having a varied temporal nature. Scenarios that reflected both descriptive and normative elements, were categorised as having dynamic value/realities. Scenarios with data collected through both isolated desk research methods and group participatory engagement were categorised as having a blended effort in development.

Limitations
There remain certain limitations that should be acknowledged at this point, in an effort to clarify the validity of our data and methods. First, our dataset could reflect some potential biases from our selection process. We accessed largely English-language primary COVID-19 scenarios. As such, our dataset may reflect a bias towards accessing only English-language COVID-19 scenarios, or it may be an accurate reflection of the corpus of COVID-19 scenarios available at the time. Firms, organizations, researchers, practitioners, and media that published their COVID-19 scenarios in other languagesreflecting any number of social, cultural, economic, and political differencesmay compare differently on Cairns and Wright's (2020) three main propositions. Clearly, it is important to acknowledge that our attempt to evaluate these authors' propositions is through the lens of mostly western, English-speaking practitioners, although, as we have documented, the focus of the scenarios included governments, organizations, populations, and cultures across the world.
Second, our selection process is also not as generalisable and repeatable as more traditional, quantitative research methods and techniques. Though web scraping is a well-established technique for gathering large amounts of data from the internet, the volatility of websites and web content makes the process difficult to repeat at future 1 Examples of such publications are the WHO's 2019 annual report, A World at Risk, published to identify the "most urgent needs and actions required to accelerate preparedness" (accessed May 2020, p. 4, https://www.gpmb.org/a nnual-reports). The Red (Team) Analysis Society published The COVID-19, Immunity and Isolation Exit Strategy to review key uncertainties about immunity and recovery and potential end strategies (accessed June 2020, https://www. redanalysis.org/2020/04/07/covid-19-scenarios-of-immunity-and-exit-s trategy/). The UN published Be Ready for COVID-19: Key Scenarios flyer to advise on following specific behaviours under the scenarios of meeting others, travelling, and staying home (accessed June 2020, https://www.who.int/emer gencies/diseases/novel-coronavirus-2019). Iberdrola electric utility company published a shareholder's bulletin for Q1 of 2020 that focused on their recovery from the crisis created by COVID-19. They only mention their financial resiliency against a "stress scenario" (accessed June 2020, p. 5, http://www.iberdr ola.com). The New York Magazine reported early estimates from the CDC of the virus' projection "if no action was taken to slow the spread", and referred to this as a scenario (13 March 2020, https://nymag.com/intelligencer/2020/03/cdc s-worst-case-coronavirus-model-210m-infected-1-7m-dead.html) 2 For example, Oliver Wyman consulting firm created the Alternative What-if Scenario Modelling, as part of their COVID-19 almanac (accessed June 2020, htt ps://pandemicnavigator.oliverwyman.com). They offer simple graphical projections of growth rates up to 3 weeks in the future.
dates. To help mitigate potential losses in data access, all original URLs are provided where scenarios were sourced, along with dates, with the aim of aiding our readers. Where sites may be missing, the Wayback Machine (1996) has proven to be a reliable tool for retrieval. Furthermore, to help with any future replicability efforts, all webpages that contained scenario data, and not already archived, were reported to the Wayback Machine for archiving. The third limitation is that most of our dataset is populated with scenarios that were developed at arm's length by external consultants. It may be the case that consulting firms were more likely to openly publish their COVID-19 scenarios than other organizationsin an attempt to generate business. However, it may also be the case that these consulting firms were, in fact, the main producers of first-wave, COVID-19 scenarios. To recall, our dataset was built from various internet searches, as well by directly contacting organizations, government offices, and executives across several sectors, requesting their willingness to share any COVID-19 scenarios their organization developed. From this multi-front approach, we were able to collect scenarios that were not publicly available at the time which make up nearly 20 % of our dataset. It remains the case, though, that we are unable to assess what we did not receive (or even whether there was any further content to receive) to gain a more accurate picture of the potential for bias in our dataset and subsequent analyses.
Our fourth limitation is unfortunately an all-too-common limitation with scenario work and research. It was not possible to discover and assess any potential impacts these scenarios had on their target audiences, effected communities, stakeholders, or unanticipated groups and policies. Scenario research, in particular, is regularly limited in assessing scope and impact outside the rare longitudinal case study, where an indepth action research approach can be taken. Such limitations should not be, however, accepted as insurmountable, but recognized and exposed so better methods, techniques, and tools can be designed to remove such limitations to knowledge.

Results
Some publications were fully informative with reporting their methods and techniques, whether within the publication alone or on their associated websites. Several lacked necessary process information, which is a recognized existing limitation in the field of scenario planning publications. Organizations and authors were contacted to help inform on all missing categorical information.

Practitioners
The first set of analyses look at who developed the scenarios and whether the authors made it clear any decision-makers participated in scenario planning. A significant majority of scenarios (77 %) were developed exclusively at arm's length, independently of the targeted organizations and stakeholders (χ 2 (3, n = 213) = 310.93, p < .000). The second largest group were heterogeneous teams (12.7 %), then exclusively experts (7 %: just one expert in the case of Honey, 2020, April; Kesson, 2020, April;Palma, 2020) and the smallest proportion developed by lone facilitators (3.3 %). No scenarios were developed by community members nor problem owners, though they were included in different ways. PricewaterhouseCoopers (PWC) used one of the most common methods, which was to gather a representative sample of citizens through online surveys "to understand some of the impacts of COVID-19". If we take into consideration that scenarios developed by a cross-population of different homogeneous groups of practitioners all included consultants at some point in scenario planning, then it is revealed that outside consultants were the main source for COVID-19 scenarios in the first wave (89.7 %).

Decision makers
Out of 213 COVID-19 related scenarios, only five publications, that produced a total of 25 scenarios, involved industry decision-makers at some point in the process. At the collaborative level, three scenarios were developed by Drewry (2020) maritime consulting firm after consulting with Chinese port operators for the container shipping industry. The four other publications were products of consulting firms, scenario experts, and industry executives working together. The Advisory Board healthcare consulting firm, members of the larger US health system, and internal healthcare experts developed 12 scenarios for US hospital preparedness (Kuchta et al., 2020, March). Members of the University of Minnesota's Board of Regents developed three scenarios in conjunction with scenario consultants in preparation for the 2020/21 academic year (Ikramuddin, 2020, May). The Minnesota Department of Health (2020, June) developed three scenarios with education experts for public and private schools (pre-K through 12th grade) in preparation for the 2020/21 academic year. Scenario experts from the University of Strathclyde, together with members of the Glasgow Chamber of Commerce, developed four scenarios for the greater Glasgow city region (McKiernan, et al., 2020, July). From our extensive search, we were only able to locate five publications that reported active engagement with decision-makers from affected communities in scenario planning work that was designed, to some extent, to target their COVID-19 affected futures. In conjunction with an arm's length process, the results indicate some levels of low-quality process.

Data collection
Practitioners, decision-makers, industry experts, and stakeholders used various scenario planning methods. Analyses show that an equivalent number of scenarios were developed through more solitary desk research methods and a blending of desk research with partial participatory group work (45.1 % and 46.5 %, respectively). A minority of scenarios were developed solely from interactive participatory groups (8.5 %). Understandably, the pandemic prevented most people from working together in the same physical space during the first wave. Even in the face of social distancing restrictions, just over half of the scenarios in our dataset appeared to have been developed through some form of participatory group work, whether virtually or face-to-face (55 %), illustrating higher quality process methods.

Revisited scenarios
The final analysis that addresses Cairns and Wright's (2020) process measures, looks at whether any COVID-19 scenarios were revisited by the authors before the end of the first wave. Out of 43 organizations, we identified only five that revisited their earlier scenarios (see Table 2). All revisited scenarios came from consulting firms and think tanks, with no in-house efforts appearing in our dataset.
Content analysis reveals that organizations who revisited their previous scenarios committed one of two revision types. The first type readdressed and revised previously published content. This includes any parts of the publication: updated statistics, scenarios, model projections, or interpretations. The second type did not readdress or revise previous content but, instead, published new content. We detail both types of revisions.

Readdressed/revised content
PWC updated their May report, "COVID-19 UK Economic Update", with new COVID-19 related data and published a second report in June (Forrest, et al., 2020, June;Forrest, et al., 2020, May). Their updated June report was shorter; However, the two scenarios they developed for their May report were not updated and were repeated verbatim from the earlier report.
The Red Team Analysis Society published regular updates of information as the pandemic progressed through the spring months and referenced their original early scenarios in each publications (Lavoix, 2020, May). They did not update their original three scenarios, "Worst Case Baseline Scenarios for the COVID-19 Pandemic", but rather used them to measure our unfolding reality against, moving from one to another as the world's infection and death rates increased.
Vision Foresight Strategy LLC published three scenarios in March, then revisited just their worst-case scenario, The Bad Scenario, in April (2020, March; April). Their motivation was to reconsider the "more challenging scenarios" (p. 12) in light of new and emerging indicators.
Deloitte released two separate scenario publications in April as part of their respond-recover-thrive resilient leadership framework (Blau & Schwartz, 2020, April;Kalish, et al., 2020, April). The first three scenarios focused on economic recovery. The second group of four scenarios focused on broader implications of thriving through the pandemic not just for economic systems, but healthcare and social systems as well, integrating their narratives together into pictures of the future.

New content
Kantar published new scenario content in their revision, focused on uniquely different industries  Content analyses were easier to verify, given that all data could be found within the scenarios (as opposed to some of the process data), however, were subject to greater scrutiny, due to the nature of qualitative analysis. As with process data, all content data were coded by independent raters who completed their coding when consensus was reached on all categories.

Quantity
To help address Cairns and Wright's (2020) concern that early COVID-19 scenarios may have been too global in perspective, we first look at the quantity of scenarios within each publication. The development of several scenarios is one tool for introducing focus and variety into planning and strategy. Between the months of January and June, practitioners developed a range of 1-48 COVID-19 focused scenarios. EIU produced the largest number of scenarios in a single publication. Regardless of whether we adjusted to the publication level or organizational level as the source for each scenario group, the most common quantity of scenarios per output was Mode = 3. Engaging with the traditional "three scenario" model is a common methodological technique in scenario planning, particularly in the Intuitive Logics school of practice. Unfortunately, the trios reflected the same commonality in themes, as pre-pandemic practices, which were 'worst', 'best', and some form of 'neutral' or 'business-as-usual'.
To help illustrate, Table 3 comparatively ranks the 10 most popular terms within the 48 generic EIU scenarios against the 10 most popular terms within the 25 specific scenarios developed with stakeholder and decision-maker participation, based on weighted percentages. 3 The summary table helps illustrate how the more common, generalised EIU scenarios lack a spotlight and offer a diffusion of global key issues, compared to the concentrated focus of stakeholder development of scenarios. EIU made several references to general geographic territories (e.g. global, regions, and countries), whereas the stakeholder-focused scenarios referenced more specific territories (e.g. China, Glasgow city, Minnesota, port, school, and university). There were no focused references to stakeholders in EIU's scenarios, yet those scenarios developed with stakeholder participation at any point in scenario planning resulted in focused references to several stakeholder groups in the narratives (e.g., patients, staff, and students). Note, especially, the terms common in the stakeholder-focused scenarios deal with stakeholder issues and so are likely to prompt consideration of actions by those affected stakeholders (cf Cairns and Wright, 2020). EIU scenarios were also published primarily through their website, to a general audience, whereas both Minnesota publications were access through academic channels, and Drewery and Glasgow Chamber of Commerce scenarios were published in-house for the clients.
There is no expectation that the majority, or all, of COVID-19 scenarios should be specifically focused on target communities or sectors. The issue is that without representation through the scenario narratives, outcomes such as the impact of decisions on effected communities and stakeholders has a greater chance of being unanticipated and undetected, which is the antithesis of scenario planning. In times of great disruption, such consequences can carry far more weight than prior environments.

Value/Reality
Given the overstated generic view employed in developing early COVID-19 scenarios, at first it seems surprising to see that the majority of the narratives focused on descriptive, exploratory futures (62.9 %), compared to dynamic (19.2 %) or normative (17.8 %), (χ 2 (1, n = 213) = 83.92, p < .000). Upon reflection of the extant literature, however, the outcomes appear to reflect the kind of high-quality content that makes scenario planning an effective tool. In the face of a global disruption and exponentially increasing uncertainties, it is not unreasonable to expect wide-spread efforts to explore futures that envision realities quite different from our past and present experiences.

Temportal nature
As stated earlier, the temporal nature of scenarios (i.e. varied, chain, or snapshot) do not necessarily have a hierarchy of quality. When paired with measures of complexity (i.e. high vs low), though, content analyses can present more meaningful results to address the issues of content quality. The vast majority of scenarios took a snapshot view of the future (85 %), where they focused on just a short timeframe, where the rest, minus one publication, provided a chain-of-events storyline (13.6 %).

Complexity
Unsurprisingly, the majority of scenarios were also developed simplistically (83.1 % low complexity), where information was presented as a single paragraph, bullet points, or an illustrated trend with little narrative. Together, 76 % of first-wave COVID-19 scenarios provided a simplistic snapshot of various plausible futures (χ 2 (2, n = 213) = 30.03, p < .000). The combination speaks directly to the level of impact the processes and contents could have possibly had on their target audiences and decision makers. How much representation and strategic dialogue can be facilitated from single paragraph scenarios that explore only a single snapshot of causally-linked future realities?

Data
With a picture building of first-wave scenarios, the next analysis looks at the kind of data the authors chose to present within their scenarios, to communicate the future possibilities and trends they developed. The majority presented exclusively qualitative, narratives (66.7 %) where futures were a continuing active pandemic or emergence from COVID-19 pandemic. Following this lead was a close split between either primarily quantitative (14.6 %) or a mixture of both (18.8 %), (χ 2 (1, n = 213) = 107.07, p < .000). Quantitative scenarios were centred around modelling different infection and mortality rates with incubation time (Kissler, Tedijanto, Goldstein, Grad, & Lipsitch, 2020, April), disease and symptom severity (Lavoix, 2020, May), and economic outcomes (Verbraken & Sampieri, 2020). The popularity of qualitative narratives should be unsurprising given that 1) the most prominent method of scenario planning is Intuitive Logics, whose methodology prioritises qualitative narratives and 2) we used Spaniol and Rowland's (2018) criterion that narrative elements partially determined scenarios from non-scenarios for our final dataset. With that said, there were no expectations that first-wave scenarios should be dominated by qualitative data, as opposed to a mixture of both data types, which may have reflected greater efforts in validity checks by the practitioners and communicate more salient factors and implications to the target audiences.

Implications
The final analysis we present, evaluates scenario content to determine whether implications for action were included. These are passages that indicate specific actions based on scenario futures. A review of the dataset revealed only 16 publications included some form of actionable implications. Though only 7.5 % of the full dataset, they cover a broad range, from hyper-focused to ambiguously vague. Several publications did not elaborate on implications, but rather suggested that implications and actions should be gleaned from their scenarios (e.g. "This document should ultimately spark a number of questions around implications and next steps for your organization" (Blau & Schwartz, 2020, April) or advertised their services to "explore the implications of these" scenarios (e.g. Kalish, et al., 2020, April). Table 4 presents a sample from each publication that expresses the key points.
The final analysis reveals an outcome in scenario planning that has been discussed extensively in the extant literature. Scenario planning was designed to serve as an intervention for change. Without including actionable implications alongside their selection of scenarios, authors, practitioners, facilitators, consultants, and the rest, risk devaluing their work and devaluing scenario planning.
In summary, the most popular profile for first-wave COVID-19 scenarios includes, ✓ Arm's length development ✓ Absent of decision-makers ✓ Participatory group work ✓ Single-shot publications ✓ Containing traditional "three scenario" model ✓ Generic applications ✓ Descriptive, exploratory futures ✓ Simplistic snapshots ✓ Qualitative narratives ✓ Without implications for action

Discussion
In response to Cairns and Wright's (2020) paper that highlighted concerns about the value mass-produced COVID-19 scenarios provided for the affected communities they were aimed at, our investigation provided an in-depth evaluation on the inherent value of 213 scenarios, published within the first wave of the pandemic. We measured value by applying two yardsticks, one focussed on quality of the scenario development process and the second focussed on quality of the resultant scenario content. We propose that scenario planning, and by extension scenarios, offer greater value to their target communities and organizations when practitioners engage in higher quality processes that develop scenarios with higher quality content. We used the CSI typology as a framework from which we defined and analyzed each scenario's value profiles. To help contextualize outputs of higher quality, it is necessary to contrast these exemplars against lower quality reflections. By contrasting and comparing a variety of qualities, we aim to help future practitioners and researchers recognise qualities to aim for as much as qualities to avoid. Based on our work, we offer some guidance below, for high quality improvements in future scenario planning at times of great disruption. The ultimate aim to help increase the plausible impacts that can result from effective, high quality scenario planning.
The first barrier we encountered was a regular lack in communication on process methods by the authors. Most of the information provided by organizations who published COVID-19 scenarios did not reveal whether practitioners used a structured method for developing their scenarios. It is understandable that private industries and consulting firms feel proprietary ownership over the techniques they develop, which stand them apart in their respective markets, and therefore are reluctant to expose their practices. From the client side, however, it can be highly informative to know whether the scenarios being broadcasted were developed with key stakeholders, reflected real-world issues, incorporated modelling, and brought relatable, valuable insights. These are factors that can be shared and maximize marketing efforts without risk of exposing proprietary information.
To recap, high quality processes are indicated by inclusion of: Table 4 Sample of "implication for action" passages.

Focus Passages Reference
Policy Implications Short-term: • "Cutting interest rates" • "Dissemination of good hygiene practices can be a low cost and highly effective response" Long-term: • "More investment in public health and development in the richest but also, and especially, in the poorest countries." (McKibbin & Fernando, 2020, March) Health Systems • "Implement business continuity plans" • "Potential for aid diversion and corruption" • "Sanitation, human contact, travel regulations, and food sourcing would all need to be reconsidered" • "Deciding which types of technologies are ethical to utilize or dealing with the ramifications of technology abuse for biological warfare" (acaps, 2020, April)

Economy
• "Impact on the economy would be considerable… We believe this is a real risk, which is why policy response is essential" (Gattiker, 2020, March) Governments and Institutions • "Transition to 'next normal' is contingent upon vaccine development" • "Serologic testing will be an impactful lever" (McKinsey & Company, 2020, May)

US Healthcare
Extensive implications linked to their four scenarios, divided into communications platforms, field forces, market access, marketing, clinical programs, and distribution.
• "Customize support to the needs of individual practices; prepare to help practices handle the backlog of patients and provide needed access and additional support to re-engaging patients" • "Support increased use of mail order/at-home use" (Carbone, Abraham, & Burdett, 2020, June) Special Operations Forces (SOF) • "Will need additional ways to collect and make sense of the signals of change" • "Operate far left of boommoving, in fact, away from the traditional OODA loop process of adaptation to a much more anticipatory predict and act approach." (Cederquist, Gibbon, & Lum, 2018, September) Energy and Natural Resources • "Oil demand is likely to be lower in the long term than seemed likely before the coronavirus hit." (Crooks, et al., 2020, May) US-Focused Geopolitical, Economic, and Social • "Keep strict measures in place" • "In the short term, both the United States and China may need each other to ensure a chance at a global recovery." • "The United States… would need to lead a G20-directed effort alongside Europe and China, … and work closely with its Asian allies and partners" (Burrows & Engelke, 2020, April)  • "Support for teaching and learning, advising, student (not to mention faculty and staff) health and well-being, and coordination and logistics will need to be reinforced" (Maloney & Kim, 2020, April) Crisis Return Checklist • "Include a 'red team' that critically evaluates all Return actions" • "Reallocate and retrain resources from divisions not soon reopening to high-priority areas that need additional capacity" (McKinsey & Company, 2020, May)

US Emerging Technologies Governance
• "U.S. government should identify those 'must win' technologies where primacy or parity with competitors is vital to national security." • "Undertake broad, sustained diplomatic engagement to advance collaboration on emerging technologies, norms, and standards setting." (Brannen, Haig, Schmidt, & Hicks, 2020, January)

Pre-K Through Grade 12 School Leaders Guidance
Extensively prescribed lists of actions for educators and staff in schools, divided along "requirements" and "recommendations", based on three scenarios.
• "Provide social distancing floor/seating markings in waiting and reception areas." • "Create a process for students/ families and staff to self-identify as high risk" (Minnesota Department of Health, 2020, June) EU • "Organise another 'Bretton Woods moment' devoted to building a revised and more inclusive system of global economic governance" • "Start working now on building a more powerful global partnership on health" (Islam, 2020, March) Leaders and Professionals Presents implications as two tables of leading questions.
• "What roles can you play to support significant new business creation when COVIDrelated restrictions begin to ease?" • "How prepared are we to respond to a second, different type of disruption right now?" M.M. Crawford and G. Wright • affected stakeholders • application of a structured development method • ongoing refinement of scenario storylines High quality content is highlighted through the use of: • comparatively different systematized scenarios • complex narrative formats that reflect key factors and values affecting the targeted audience(s) • followed with actionable implications for communities and organizations.

Scenario process
From a process perspective, our dataset reflects largely, but not entirely, low quality efforts. Lower quality efforts are shown in the fact that most scenarios were developed at arm's length from the affected communities, and all, but five scenario interventions, were absent of decision-makers that represented any of the affected communities. Those few scenarios that were developed from group work only appeared to include community members or key stakeholders in the PWC methodology published a month before their scenarios, regarding stratified surveys of the UK population.
In terms of the development process, low-quality scenario planning is typified by the absence of stakeholder voices and is often a result of largely isolated, practitioner-based, desk research. The potential consequences are that practitioners fail to benefit from knowledge sharing and challenged assumptions, and the resultant scenario content is ambiguous on the issues that most concern affected communities. This is partially illustrated in Table 3, where key terms are compared across generic and specific scenarios. Though the EIU was the most prolific with COVID-19 related scenario production, they appeared to largely engage in the process at arm's length from the many sectors they focused on, and as a result, failed to employ targeted narratives that spotlighted definitive stakeholders, while also failing to include any definitive implications for their clients, readers, or the effected communities profiled in their scenarios.
What has been stated is not remarkably different than what many have already reiterated in hundreds of scenario and foresight publications. Involvement of affected communities and decision-makers in the development of scenarios can increase the chances that relevant, local needs and issues will be accounted for, and increase the value and impact of scenario interventions. One of the major issues that might have prevented practitioners from reaching out to targeted communities could have been due to the chaotic, fast-paced changes that came with the initial spread of the pandemic and ever-increasing isolation measures. A need to help, through expert skills, met with barriers to engage with disparate, distant audiences, may have led to rationalising that any output was better than no output. Thereby leading to mass-produced scenarios, but largely disconnected from the target communities, lacking in intended value and impact. These are reasonable assumptions to make, until we review the Data Collection characteristics (participatory, desk research, or blended). The majority of scenario interventions were conducted through participatory groups, whether combined with solitary desk research (i.e. blended) or not. What this characteristic indicates is that the majority of contributing practitioners found effective ways to collaborate and engage in scenario planning at a time of great disruption, ever-increasing social distancing, and early versions of novel virtual platforms. The data show collaborative efforts were successful between potentially more familiar connections (e.g. other scenario planning practitioners and colleagues) and largely abandoned with less familiar connections (e.g. stakeholders and affected community members). This behavior is reflective of on-going issues in scenario planning for the public's good, a problem of communication density (Butts et al., 2007;Cairns et al., 2016;Crawford, 2019). Furthermore, reacting to the pandemic appears to have exacerbated the behavior, referred to as situational myopia. This is a quality of short-sightedness in foresight or intellectual insight bounded to the immediacy of the agent's knowledge in this case, scenario practitioners. Situational myopia creates an inability to perceive and relate elements of a crisis that lie outside the practitioner's existing knowledge (Roe, 2011). Butts et al. (2007) consider situational myopia inevitable under distaster circumstances. It is due to the inevitability of the behavior, however, that Öö rni, et al. (2011, p. 26) recommend, "We can't expect the unexpected, but we also fail to expect consistency in our own behavior if the short term behavior appears inconsistent even though there is a consistent long-term pattern." Conversely, high quality scenario planning will involve decisionmakers, community members, and/or stakeholders for the express purpose of ensuring their voices, concerns, insights, and opinions are integrated into a shared thinking process (Cairns and Wright, 2020). An example of this was illustrated in Drewry's (2020) scenario development, which involved consulting with Chinese port operators (i.e. experts and stakeholders) at the start of scenario planning. The co-creation activity resulted in a variety of scenarios (optimistic, medium, and worst cast), but collectively the scenarios included industry-specific key factors ("exposed the fragility of global supply chains that are overly dependent on a single manufacturing source", p 3/6) and provided actionable implications against which shipping companies could test their policies and practices ("carriers would be forced to revisit the playbook from the financial crash of a decade before…shippers will look to broaden their sourcing options as a form of insurance", p 5/6).
High quality scenario planning employs a structured process that takes the measure of time and employs reiterative processes for revision, when needed. Reality, markets, and economies move fast, but rarely do our shared realities move faster, with more volatility, than in a time of global crisis. Therefore, we would expect to have seen regular revisiting of earlier scenarios by practitioners if they were engaging in high quality practices that would maximize community value. To the contrary, analyses revealed that only five groups revisited their earlier scenarios, our web scraping methods revealed that they all revisited only once, where the majority used their original scenarios more like sounding boards, with only one group redrafting their scenarios to reflect new information. We recognise that all revisited scenarios were from consulting firms and think tanks, with no in-house efforts making it to our dataset. Outside of any methodological errors, we question how many of these single-shot efforts reflected the conditions of the time, where individuals, organizations, and communities were overwhelmed, forcing many to become even more time-poor than before. It may have been easier for consultants and small think tanks to take on the laborious task of reiterative scenario planning in their own spheres, compared to repeatedly collaborating with external clients and affected community members. This assumption is further supported by the fact that all revisited scenario efforts came from consulting or executive teams who developed simplistic, largely generic scenario narratives.

Scenario content
It is important to note that the vast majority of first-wave COVID-19 scenarios, whether generic or specific, community engaging or arm's length, focused on descriptive elements that took exploratory paths of inquiry into hundreds of different futures. Futures that span a few weeks to more than a decade. This kind of content is reflective of high-quality content. When faced with a landscape that is changing almost hourly, most certainly daily, where uncertainties are compounding and stability is dropping, scenario interventions find their value in helping practitioners explore the possibilities more than re-treading the normative, often biased past. Examples of re-treading can be found in the COVID-19 dataset, which Cairns and Wright (2020) identify as 'frozen pictures' in some of our dataset. For example, IARAN LTD, produced three scenarios in June that included the following ambiguous and time-dated content, "Lower levels of hygiene among poorer members of society drive the spread of the virus" (2020, June, p. 3). The team from the Atlantic Council provided an equally limited perspective with statements like, "For the Europeans, it is the same old battle that was fought during the eurozone crisis," in their post-COVID-19 scenarios (Burrows & Engelke, 2020, April, p. 12).
It is generally promoted that comparatively different systematized sets of scenarios are a better use of time than single, one-off scenarios. Crawford (2019, p. 17) states, "A single scenario offers highly limited parameters and no comparisons to other future realities with which to challenge mental models and alternative options." Multiple scenarios allow practitioners and powerful stakeholders to test for robustness and flexibility of a strategic plans and policy (Bunn and Salo, 1993). There is no 'optimal' number, but rather enough should be developed that challenge mental models, test policy, and inform future decisions (Wack, 1985;Schoemaker, 1993;Van der Heijden, 1997). An example of multiple scenarios developed to provide conditions for testing decisions and policies is Inside Higher Ed's April publication of 15 Fall Scenarios for US colleges and universities to consider for the fall semester (Maloney & Kim, 2020, April). Their scenarios run along a continuum of "back to normal on one end and fully remote learning on the other", from normative to descriptive. One issue their larger publication encounters is that the "models are not all distinct, and many overlap". Producing too many scenarios risks losing creative differences and fails in their job to challenge existing mental models. This may be a particularly charged issue during global crises, where uncertainties and impact are high. Providing scenario interventions where multiple perspectives can be explored and challenged offers greater probabilities to creating 'red flags' for horizon scanning in times of increased volatility, compared to a single-focus scenario.

Recommendations
Based on the results of our analyses and compatible conclusions in extant disaster literature, a series of recommendations are proposed to help practitioners achieve high quality processes and content in their future scenario interventions.

Process recommendation 1 -move towards proactivity by establishing dense channels of communication
The first recommendation is for organizations and practitioners to understand the inevitability of such biases as situational myopia and proactively create protocols that develop communication density. Develop a hierarchy of stakeholder, client, customer, and community members by perceived relevance to your organization or industry. Spanning high to low relevance builds greater density into your potential future communication channels by including members outside the norm. The ranked order provides a protocol that can be utilized as a heuristic during times of disruption and crisis. Well-developed heuristic tools are a cornerstone to disaster preparedness plans. Burt andVan der Heijden (2003, p. 1022) suggest that organizations need to "make sense of new realities earlier". There are any number of secondary effects that stem from proactive engagement efforts, as well. For example, Boston Consulting Group found companies that opened communication with their investors early in a crisis, performed significantly better than companies that waited and stayed disconnected (Reeves et al., 2019). Participation enables those in the process to gain the greatest insights (i. e. value) from the intervention, and for the process to be informed by a greater selection of information (i.e. impact) (Cairns and Wright, 2020;Crawford, 2019). Cairns & Wright (2020, p. 4) offer further guidance to help build preemptive behaviours within target communities, First, successful development of meaningful scenarios within a community requires skills to seek out and acknowledge global/ generalizable knowledge that is relevant, while embedding local knowledge and wisdom. Second, both global and local myth and misinformation ("alternative truths" and "fake news") must be recognized and countered. Third, development of the required skill sets in communities requires programs of education and training for which there may be limited resources-financial and intellectual-in a post-COVID-19 world. Fourth, any program to implement such skills nurturing will open the door to the "snake oil salesmen" of futurism and false promise. Finally, we must question whether academics will be willing/able to commit to necessary long-term engagement in communities from within their own world of shortterm, publish-or-perish, metrics-based reward?

Process recommendation 2 -capture flexibility at every step
Global disruptions are, by their nature, complex and unpredictable events. Response activities must adapt to match changing circumstances (Butts et al., 2007). As evidenced in past disaster, crisis, and pandemic, information rapidly changes, established channels of communication breakdown, resources change, and centres of power and control disappear and emerge. All of which can cause rapid failures under pandemic conditions. This is where the support of hierarchically established stakeholder and community connections begin to show their value. As well-known channels of communication breakdown or become unexpectedly irrelevant, others are on standby; Specifically ones that may have been less relevant in times of peace or reconstruction.
Flexibility in times of disaster can also be aided by establishing replicable, defensible, structured methods. Scenario planning is an interative process, for both purely qualitative methods (e.g. Intuitive Logics) and mixed-methods (e.g. Cross-Impact Analysis). Establishing and clarifying which method to use allows practitioners to expertly adjust the process to meet their changing needs.

Process recommendation 3 -revisit, review, revise… repeat
Ongoing refinements of scenario storylines support methodological and actionable flexibility, as unexpected events emerge in real time. Practitioners should be explicit on revisiting timelines. Revisit-flexibility is shown through practitioners also remaining open for disruption and revisiting existing scenarios sooner than expected. All scenarios should be produced with the understanding that they will be revisited, thoroughly reviewed with no driving forces taken as immutable, and revised in the face of new information, regardless of how much these may differ from prior scenarios. Revision efforts should also remain open for creating more or fewer scenarios.

Content recommendation 1 -develop multiple scenarios covering different futures
It cannot be stated too often, multiple scenario development brings greater value and impact to any intervention, compared to a single output. By the very nature of our reality, there will always be more unknowns than knowns. Developing multiple scenarios helps practitioners capture more potential unknowns than any single, even highly complex, scenario. The most common output is to develop three scenarios. Too often, however, the "three scenario" model leads to one normative, one best-case, and one worst-case. During global disasters, the "three scenario" model does little to provide actionable insight for decision-makers. For example, it was a given that normative scenarios early in the COVID-19 pandemic included mass global deaths. That is, after all, one of the requirements for a disease or virus to be categorised as a pandemic. There can be little value in spending time and resources developing a normative scenario to help strategise through a pandemic, compared to non-pandemic conditions. The majority of extant literature recommend up to six scenarios (see Amer et al. (2013) for a comparison of suggested scenario quantities).

Content recommendation 2 -integrate stakeholders self-interests
The purpose to our first process recommendation is to increase the chances of including self-interested actions of powerful stakeholders to unfolding scenario storylines. Without the right agents in the process, scenarios miss out on expert or unique insights, interests, and needs. Pandemics and major global disruptions exacerbate the interconnectivity of our everyday driving forces. Scenarios must be able to capture key self-interests of targeted communities in order to bring value to the efforts.

Content recommendation 3 -be explicit with implications for action
Information changes with unexpected speed during pandemics. Delays can prove detrimental, both in the short-and long-term. One feature a large proportion of COVID-19 scenarios from our dataset are missing are implications for action by those communities affected by the pandemic. These failures to help lead stakeholders and decision-makers towards supportive actions leaves the scenario intervention purposeless. Practitioners, facilitators, and scenario teams must ensure their scenario planning include a variety of implications for action so they can lead theiroften clientsto more salient articulated actions. Furthermore, dividing causally related driving forces across multiple scenarios, as opposed to one or two highly complex scenarios, helps increase comprehension of plausible futures, and aids the development of articulated action.
Our investigation into first-wave COVID-19 related scenarios revealed various points of both high and low qualities, in process and content. To achieve higher quality scenario interventions, and thereby more valuable efforts, we offer this summary of recommendations. Using guidelines, such as the yardsticks we applied in our evaluations and the typology we used for analysis, can help bring a replicable, defensible, structured development to the process. These tools can help practitioners counter potential situational myopia with proactive protocols. Our yardsticks provide measures for ensuring affected stakeholder and community member inclusion, incorporation of the selfinterested actions of decision-makers, development of several scenarios with ongoing refinement of their storylines, and inclusion of implications for action by those communities affected by events within the scenario.

Data availability
Data will be made available on request.