Using Strategic Options Development and Analysis (SODA) to understand the simulation accessibility problem

Abstract Simulation modelling is applied to a wide range of problems, including defence and healthcare. However, there is a concern within the simulation community that there is a limited use and implementation of simulation studies in practice. This suggests that despite its benefits, simulation may not be reaching its potential in making a real-world impact. The main reason for this could be that simulation tools are not widely accessible in industry. In this paper, we investigate the issues that affect simulation modelling accessibility through a workshop with simulation practitioners. We use Strategic Options Development and Analysis (SODA), a problem-structuring approach that allows for the stakeholder views to be expressed and linked in a systematic way. The causal map derived represents the emerging concepts and their effects, with the view to identifying their impact on the accessibility problem. We present our analysis of the issues and options identified. Based on our findings, we discuss the implications and recommendations for the future uptake of simulation.


Introduction
Simulation modelling is a multi-disciplinary field (Sokolowski & Banks, 2011) and a popular OR tool (Lane et al., 1993) used in a wide range of domains, among others defence, health, manufacturing, transportation, and supply chain (Mourtzis, 2020;Taylor, Eldabi, Riley, Paul, & Pidd, 2009;Tolk & € Oren, 2017).In essence, simulation modelling involves using computer models to investigate the behaviour of a given system and to experiment with different scenarios to support decision-making (Gunal, 2019;Law, 2015;Robinson, 2014).
There is a wide range of simulation techniques available with the most prominent being discreteevent simulation, system dynamics, and agent-based modelling (Jahangirian et al., 2010;Law, 2015).Simulation is a particularly useful tool as it enables the representation of real-world systems in a computer model, which in turn can be used to experiment with different scenarios to identify feasible improvements in that real-world system (Banks, 1998;Pidd, 1998;Robinson, 2014).This is particularly beneficial as one can experiment in a safe environment, especially in situations where it is impractical or even unethical to trial changes directly into real-world systems, e.g.developing evacuation plans for a major city.
Improvements in technology over time have meant that the adoption of simulation has increased over the years (Collins et al., 2022;Robinson, 2005).Despite the increased number of simulation research works and papers in the literature, the relative lack of "real world" involvement in simulation modelling research and the lack of evidence of practical benefits is noted in the literature (Taylor et al., 2009).Simulation in health has especially drawn the attention of researchers over time, voicing concerns about the lack of implementation of simulation tools designed to support healthcare systems (Roy et al., 2020).Harper (2002) states that "given the wealth of work that has already been done in this area, it is both surprising and disappointing that it has not found greater application."One potential reason for this lack of adoption is due to the accessibility of simulation modelling.We view accessibility as the extent to which simulation tools are accessible to practitioners at their place of work.This is related to the extent to which simulation tools are available and easy to use and whether one has the relevant skills to use them.Harper (2002) points out that specialist simulation (healthcare) tools have been developed but are not readily used or taken up by practitioners; the authors believe that this issue is more widespread than healthcare simulation but just has not been documented in other domains.
This paper is concerned with the use of simulation in practice.Therefore, we focus primarily on the accessibility of simulation from the perspective of practitioners, those working in the industry.Our starting position is that simulation is not as widely used in industry as it might be expected.We believe that this is related to the lack of accessibility of simulation.Hence, the aim of this paper is to understand the factors that affect the accessibility of simulation tools in practice.This, in turn, can help identify steps that the simulation community can take to address the challenges faced with the uptake of simulation.We believe that this will ultimately help to identify ways to increase the use of simulation tools in practice.Thus, our starting research question is "what affects the accessibility of simulation modelling?"Interestingly our findings show that multi-disciplinarity, that is the use of simulation in multiple disciplines, has a negative impact on accessibility.
To answer this question, we use a problem structuring method (PSM) called Strategic Options Development and Analysis (SODA) to explore the issues involved.A SODA workshop with expert simulation practitioners, who are considered a stakeholder group affected by this issue, was held at the Modelling and Simulation (MODSIM) World Conference & Expo 2018.We present the issues discussed at the workshop with the participants.As we show in this paper, the focus of the workshop moved naturally from the issue of accessibility to that of selling simulation products, such as software packages, consultancy services, etc.Our findings show that the accessibility problem is closely related to selling.There were also concerns about the use of simulation by multiple disciplines, which could cause problems with communicating the benefits of simulation to potential users and problems with simulation education in the different disciplines.
The structure of this paper is as follows.The next section explores the background of the accessibility problem in simulation, followed by a brief introduction to SODA, and the method used to structure our discussion at the workshop.We then describe the approach we followed to analyse the workshop's outputs in our endeavour to gain a better understanding of the situation and present the issues discussed.Next, the analysis of the main emerging themes is followed by a discussion of our findings and conclusions.

Exploring the accessibility of simulation
This section discusses the accessibility of Modelling and Simulation.It is our aim to provide some background to the issue of the accessibility problem in simulation, which is also the main topic we explored with simulation practitioners at the workshop.

Simulation is expanding, but is it being adopted?
Over the last sixty years, Modelling and Simulation (M&S) has been applied in various domains, including communication, defence, health, manufacturing, and transport (Taylor et al., 2009).It has been applied "across virtually all [academic] disciplines" (Cheng et al., 2016); and geographical locations across the world (Collins et al., 2022).However, most of these applications have been theoretical, which shows that real-world implementation and benefit from simulation studies is relatively low (Taylor et al., 2009).By implementation, we mean that the recommendations based on the simulation study findings lead to changes in real-life systems (Monks et al., 2016;Ranyard et al., 2015).Jahangirian et al. (2017) and Robinson and Pidd (1998) argue that due to its nature, simulation studies should aim to provide insight and ultimately inform potential real-world action.Over time, researchers have commented about the limited implementation or use of simulation in real-life applications.Over a decade ago, Taylor et al. (2009) discussed the lack of real-world applications, reporting a lack of engagement with real-world practice; they found that only 5% of papers published in Operational Research and Management Science outlets refer to real-world applications.Sadagic and Yates Jr (2015) make a similar point about the lack of adoption of simulation tools in the training and education domains.
Simulation implementation across different domains differs.For example, Melão and Pidd (2003), in a survey of practitioners, report low use of simulation for business process improvement due to a preference for simpler methods such as process mapping and spreadsheet modelling (i.e.Microsoft Excel).In a review of simulation studies in supply chain, Oliveira et al. (2016) found that only 28% of papers reported the implementation of results in real-world applications.Similarly, a generally low level of implementation of simulation is reported in the healthcare domain, despite the increase in the number of studies reported and major advances in both [simulation] software and hardware (Brailsford et al., 2017;Brailsford et al., 2009;England & Roberts, 1978;Fone et al., 2003;Jahangirian et al., 2015;Jun et al., 1999;Katsaliaki & Mustafee, 2011;Roy et al., 2020).Brailsford et al. (2009) and Katsaliaki and Mustafee (2011) found that the implementation of simulation study results was reported in only 5% of papers.In a more recent review of simulation studies modelling the flow of patients in an emergency department, Mohiuddin et al. (2017) report that only 14% of studies describe the implementation of changes in real-life situations.In a more extensive review of 238 papers showcasing simulation models of internal logistics problems in healthcare, Roy et al. (2020) found that only 2% of papers discuss the implementation of the study in the real-life setting; nevertheless, they noticed a high proportion of studies (almost 70%) focused on solving real problems, using real-life data, which shows some encouraging improvement from previous decades.
It is noted that the limited implementation and adoption of simulation in academic papers can be associated with the fact that academics are motivated to publish their theoretical/methodological contributions as soon as they are developed, which could be prior to any real-life implementation of the study.As a result, academic papers are published before the impact of the simulation study is realized.This means that the actual implementation levels of academic research are not clear.While we have little understanding of the adoption of simulation in practice, from the discussion above, we come to realise that there is a disconnect between academia and practitioners in the simulation community.Nelson (2016) and Taylor et al. (2009) advocate that there needs to be a reconnection between them.A key diver to reconnecting academics and practitioners is developing a mutual understanding of each other's needs (Muller, 2005); for example, operations research academics focus on developing new ideas to "publish or perish" (Collins & Hester, 2016), whereas practitioners are focused on solving the given problem, regardless of novelty.We next consider potential reasons for the limited adoption of simulation studies in practice based on published academic research.

Challenges for simulation studies
There are several reasons that can explain the lack of adoption of simulation.Sadagic and Yates Jr (2015) surveyed simulation users and experts on computer-based training simulations usage in the U.S. Department of Defense and found that key issues were: the users' concerns about the cost of maintaining and sustaining a simulation; the quality of simulation hardware and software; acquiring and maintaining knowledge, in a timely fashion, to use the simulation effectively; and a lack of peer advertising of simulations.
Within the healthcare simulation domain, Jahangirian et al. (2015) identify stakeholder engagement as a critical reason for the lack of success and implementation of healthcare simulation studies.They furthermore identify the key factors that contribute to poor stakeholder engagement in healthcare studies, based on a survey of expert opinions.They found that the communication gap between simulation and stakeholder groups, poor management support, poor familiarity with or awareness of simulation, difficulties with understanding and working with simulation tools, and stakeholders' feeling that the project is not producing a tangible impact on the problem at hand, being amongst the key factors that affect the success of simulation studies.Indeed, Mohiuddin et al. (2017) found that only 57% of studies reviewed report some level of interaction with the stakeholders during the simulation study.Another survey, in the healthcare simulation domain, by Kirchhof and Meseth (2012) to understand the reasons for the low adoption of simulation in healthcare found that cost, low levels of awareness of simulation's benefits, and lack of simulation skills were the key factors.In all three surveys, cost, lack of awareness, and high knowledge requirements were identified as key drivers that affect the adoption of simulation.
We believe that cost is an important factor in the decision to adopt a simulation study or its results.The cost of anything is relative to its worth, i.e.Return of Investment (ROI).Even though M&S has been recognized as a critical technology by the U.S. federal government (Forbes, 2007), there is some difficulty quantifying its benefits because there is a need to understand the ecosystems (application domains) where the simulation is applied.Oswalt et al. (2012) discuss the difficulty of showing the return on investment (ROI) of M&S within the U.S. Department of Defense, one of the historically large user groups of M&S.They point out that there is often a misalignment between the perceived benefits to different stakeholders, e.g.simulation purchasers and users; which makes it difficult to determine the ROI and in turn the decision to implement its findings.A study by Soorapanth and Young (2019) attempts to develop a method to assess the value for money of a simulation study for a care stroke delivery system based on an evaluation of the financial impact and cost-effectiveness analysis.A more recent study by Soorapanth et al. (2022) put forward a framework that can be used to evaluate the costs and benefits of simulation modelling studies in healthcare.However, neither approach has been widely adopted.

Perception of simulation
In its early days, there was a view that simulation is technically complex and should be used only as a last resort (Wagner, 1969), even though this is no longer the case (Lucas et al., 2015).Authors suggest the need to demonstrate to organizations that M&S is an indispensable tool (Cheng et al., 2016).
This negative attitude towards simulation is not helped by the fact that there are bad actors within our community.Crain et al. (1992) discuss how overenthusiastic marketing by simulation tool providers tends to oversell the technology, and fancy graphics can mislead naïve users about a simulation's capabilities (Banks & Chwif, 2011;A. J. Collins, D. Knowles Ball, & J. Romberger, 2015), especially when realistic animations are used (Law, 2015).In addition, simulation presents a simplification of real-life systems and provides estimates of their key performance indicators, which may affect the client's perception of the accuracy of model results if they are not fully aware of its capabilities as a tool.It can also be expensive to build a simulation model (Law, 2015).

Knowledge requirements
Another factor that can affect engagement with simulation is that it requires technical knowledge, and it can require a steep learning curve.Hamill (2010), when talking about agent-based modelling, points out that even using beginners tools, like Netlogo (Wilensky & Rand, 2015), can be pretty challenging to someone new to M&S.They point out that even though standards exist for describing models, like the ODD protocol (Grimm et al., 2020); it is not clear how to create or assess those descriptions.When a steep learning curve is combined with a lack of time available for gaining that knowledge (Sadagic & Yates Jr, 2015), this can affect users' engagement and access to simulation.This problem is compounded in domains like healthcare because of the complex nature of the problems to be simulated, which tend to have less evident structure, more complex systems, messier problems, and consequently, difficulties in collecting necessary data (Tako & Robinson, 2015).

Social factors
Other factors that affect the implementation of simulation studies can be intangible, such as the social situatedness of models.For example, inter-personal relationships and trust placed on the model by the stakeholders and, ultimately, the decision-makers can affect the implementation of simulation studies (A.Harper et al., 2021).This is a topic that has not been widely researched in simulation and OR literature.The study by Harper et al. (2021) is the first to identify the interacting aspects of trust between three agents: the model, modeler, and stakeholders.They furthermore identify the social factors that affect these relations throughout the simulation modelling process and suggest that modellers pay attention to these aspects to ensure that trust in simulation studies and insights derived can result in decisions to take trusted action.Facilitated and participatory approaches to simulation are identified as means of converging multiple and contradicting views amongst stakeholders; hence embedding stakeholder engagement into the study is considered beneficial (Kotiadis & Tako, 2018;Kotiadis et al., 2014;Tako & Kotiadis, 2015).This, in turn, can influence inter-personal relationships, and trust in the model and its results.
Other contextual and social factors such as leadership, power, organisational culture, openness, and willingness to change, can also influence the OR and simulation study process, hence the calls for a more reflective practice to help practitioners respond effectively to the challenges faced and the environment the studies are situated in (Ormerod, 2014(Ormerod, , 2017(Ormerod, , 2020)).It is the social context of simulation studies that add to the complexity and our ability to untangle the multitude of issues at play, a characteristic of a wicked problem (Eden, 1989;Mingers, 2011;Rosenhead & Mingers, 2001).
Another factor to be considered is the simulation community itself.Over time, the simulation community has changed.Originally, the community was highly connected with a focal-point being conferences like the Winter Simulation Conference (Schriber et al., 2017), but as its usage expanded, new communities of interest have emerged, who gather in their own conferences.For example, the Society of Simulation in Healthcare (SSIH) annual conference regularly attracts more than 4,000 attendees, significantly much bigger than the Winter Simulation Conference (WSC) and Spring Simulation Conference (SpringSim) combined.The Interservice/ Industry Training, Simulation, and Education Conference (I/ITSEC), focused on military training applications of simulations, dwarfs SSIH with 17,000 attendees in 2019.The discipline of M&S is expanding and growing both in terms of specialist application areas but also geographic location, i.e. it is not entirely USAcentric anymore (Collins et al., 2022).
In addition, M&S is expanding into new academic disciplines, which are forming their own independent communities of practice.A further example is The Computational Social Science Society of the Americas (CSSSA), which has been running for over ten years (https://computationalsocialscience.org/), and predominately focuses on Agent-based Modelling and Simulation (ABMS).The ecological simulation community has developed its own simulation standards as well (Grimm et al., 2020).Other specialist application areas with their own conferences include energy, epidemiology, meteorology, manufacturing, sustainability, and transportation.This expansion means that M&S is applied in multiple disciplines, making it a highly multi-disciplinary subject, thus impacting its accessibility, as discussed in our analysis below (section 5).

Summary of literature and the accessibility problem
Summarising on the issues identified in the literature, three main factors affect the use of simulation tools in practice: awareness, technical knowledge, and social situatedness of models.We argue that these are closely related to the accessibility of simulation studies.Based on the Cambridge definition of accessibility: "the fact of being able to be reached or obtained easily," in this paper, we consider accessibility as how easy it is for users (or potential users) to choose and use simulation as a problem-solving tool.Considering users of simulation, we distinguish two different stakeholder groups: modellers, those who develop the simulations, and consumers, those domain experts that use the insights and findings of simulation to inform their decisions.Modellers can be seasoned simulation practitioners with OR and simulation expertise or novices with no prior simulation expertise, who would like to use simulation, but find it difficult for any reason as discussed in section 2.2 above.Awareness of simulation and its benefits can affect whether any of the user groups mentioned above is able or willing to use it for problem-solving.Lack of the required knowledge affects accessibility.An OR practitioner, novice in simulation, as a potential user that finds simulation tools too technical or difficult, is unlikely to be convinced to use them, hence, affecting their access to simulation as a tool.If communication of an M&S study's results and derived insights are not understood by their consumers (domain experts), this affects their accessibility to simulation.Similarly, modellers' lack of understanding of the real-life problem represented in the simulation model based on the requirements and views of those parts of the real life system (as defined by the consumers), can diminish the impact of simulation studies.All in all, the multi-disciplinary nature of the user groups involved, crossing the simulation and non-simulation practice boundary exposes a difference in worldviews, which causes the simulation accessibility problem discussed in this paper.
This paper sets out to understand the accessibility problem using an open-ended Problem Structuring Method approach and, more specifically, Strategic Options Development and Analysis (SODA).We explore the problem within a workshop with simulation experts, primarily practitioners, who have a vested interest in the accessibility and adoption of simulation studies, but also have a lived experience of how simulation studies are being used in practice.The main constructs are mapped into cognitive maps or causal maps, depending on whether developed by one user in the former and more than one users in the latter, as the conversation unfolded during the workshop.A construct includes two contrasting (psychological opposite) concepts or ideas to contextualize and refine the understanding of the primary concept (Georgiou, 2009).As identified above, the issues at play are multiple and complex; hence using SODA as a problem structuring tool can help us untangle the issues based on stakeholders' views in a transparent and rigorous way and ultimately help us get a better understanding of the issue.We first provide the reader with a brief overview of Strategic Options Development and Analysis (SODA).

An overview of Strategic Options Development and Analysis (SODA)
Problem Structuring Methods (PSMs) are "a collection of participatory modelling approaches that aim to support the diverse collection of actors in addressing a problematic situation of shared concern" (Shaw et al., 2006, p. 757).These methods focus on assisting groups in reaching a shared understanding of a messy problem.In other words, PSMs help a group to achieve some form of consensus, taking into account stakeholders' different worldviews (Rosenhead & Mingers, 2001).Strategic Options Development and Analysis (SODA) is one of the widely-used PSMs that is used to make sense of a problematic situation (Abuabara & Paucar-Caceres, 2021;Smith & Shaw, 2019).SODA can be used for the collation, comparison, and analysis of the views of many experts in relation to an issue that is considered to be messy or that there is no straightforward agreement (Eden & Ackermann, 2001;Sørensen & Vidal, 2008;Eden & Ackermann, 2004).Cunha and Morais (2019) argue that SODA can help express the values and attitudes of multiple stakeholders and encourage discussions to learn about a problematic situation.The application of SODA is useful for messy, wicked problems (Abuabara et al., 2018;Ackermann et al., 2020).A wicked problem has "a range of stakeholders with potentially conflicting values of interests, a lack of reliable data, disagreement about the nature of the problem" (Mingers, 2011, p. 730).As elaborated earlier, the M&S accessibility problem is considered a complex and wicked problem; therefore, SODA is chosen as a suitable PSM approach to elicit the views of practitioners and simulation experts on the challenges faced with the adoption and accessibility of M&S tools.
SODA is applied to guide the group discussion with the primary purpose of reflecting on the accessibility issue and is used as a reflective or "sense making" device (Eden, 1988;Thaviphoke & Collins, 2019a).One of the goals is "to find ways to move towards sensible action that can be defended in rational terms" (Pidd, 2009, p. 123).The expected outcome of SODA intervention is an overall understanding of a particular issue which is the aggregated information from people in a group (Ackermann & Eden, 2001;Georgiou, 2011;Thaviphoke & Hester, 2018;Westcombe, 2002).SODA is a flexible PSM tool, and it can be used to achieve different outcomes, such as defining the boundary of the problem of interest, identifying evaluation criteria, or developing alternatives (Marttunen et al., 2017).SODA is considered helpful because, even being part of the same problematic situation, an individual has different ways to process information according to their own view as explained in a Theory of Personal Construct (Kelly, 1955).The theory argues that individuals will try to "make sense" of a situation by identifying means to "manage and control" their surroundings.According to Eden and Ackermann (2018), a SODA process enables participants to continuously surface possible issues and structure them through the means-ends causality.Therefore, SODA is considered an appropriate tool to explore and investigate our problem of interestaccessibility of M&S.
SODA primarily uses cognitive mapping as a tool to capture and represent individuals' perceptions of reality (Ackermann & Eden, 2001;Eden, 1989;Thaviphoke & Hester, 2018).Cognitive maps represent individuals' thoughts, ideas, and critical comments (Leonhardt Kjaergaard & Blegind Jensen, 2014).A cognitive map is made of nodes and arrows (Eden, 1988).A node represents a concept which, in SODA, is written in the form of two contrasting poles: one pole representing the positive part of the concept and the second its psychological oppositethe bipolar construct as called by Eden and Ackermann (2001).The authors continue explaining that a bipolar construct generates a clear understanding of a concept as well as a proper chain of arguments.For example, if the concept shows "Solve complex problems … Solve puzzles," it can be translated as to solve the complex problems rather than to solve puzzles.It should be noted that if the concept is clear without the psychological opposite, then one is not needed (Pidd, 2009).An arrow indicates the relationships among concepts.The direction of the arrow represents the causal direction of the relationship.These relationships can be both positive and negative.A positive sign shows a positive causal relationship (e.g.Tail A node positively affects Central and Head A nodes as shown in Figure 1).On the other hand, a negative sign indicates a negative relationship among nodes (e.g.Tail B and Central nodes negatively affect Head B node in Figure 1).It is usually taken to indicate a positive relationship when there is no sign on the arrow (Pidd, 2009).A cognitive map is usually drawn presenting means-ends relationships among concepts using arrows (Eden, 2004).A concept at the tail of an arrow is considered a possible cause or influence of the concept at the head of the arrowas shown in Figure 1.In other words, a concept that has no outgoing arrows can be considered a "head," and a node that has no incoming arrows is referred to as a "tail" (Georgiou, 2009).Head concepts represent goals or expressions of the desired outcome, and tail concepts are options or constraints.Options are considered actions that can be taken to achieve the desired outcome, while constraints could be some hurdles that might be in the way to achieving a goal.In other words, any concepts that have more incoming arrows can be considered goals or objectives, while the concepts that have more outgoing arrows usually represent possible actions or constraints (Eden, 2004;McKay & Marshall, 2005).Moreover, a concept can be considered "central" if it is descriptive of different content aspects of the problem (have both incoming and outgoing arrows).Ackermann and Eden (2001) added that the "central" concepts could be one of the issues of concern because these are busy concepts.Figure 1 shows some examples of causal linkages between key concepts in a cognitive map.
It should be noted that if one cognitive map represents an individual's perspective, merging cognitive maps could provide a group's perspective.Pidd (2009) distinguishes SODA interventions into two approaches based on how a group map is acquired, SODA I and SODA II.The first approach, SODA I, develops a group map from a collection of individual maps.Alternatively, in SODA II a group of participants and a facilitator develop a common group map simultaneously at the workshop (called a causal map).The choice between SODA I and II is made depending on the context of the problematic situation, organization style, the culture of the group, or the time constraints (Eden, 1989).Similarly, Pidd (2009) notes that ultimately the choice depends on the circumstances.
An advantage of SODA I is that an individual does not feel compromised to share their thoughts which leads to higher ownership of the problem (Franco, 2007).However, individuals will need to possess knowledge of the cognitive mapping concept, and the map-merging process takes longer than SODA II.On the other hand, one of the benefits of SODA II is that all individuals see the merged map as it is being constructed and more discussions and idea piggy-backing tend to emerge (Rouwette et al., 2011).However, individuals might feel uncomfortable expressing their thoughts due to some obligations (e.g.political aspects, organizational hierarchy, etc).There are some benefits and disbenefits from both approaches.Thaviphoke (2020), however, found that there is no significant difference between SODA I and SODA II regarding the confidence level of individuals in the groups.
In this study, we use the SODA II approach to develop the causal maps with the stakeholders using pen and paper.This is due to the participants' limited time availability and knowledge of cognitive/ causal mapping.Therefore, creating a group map together with the facilitators at the workshop was deemed more appropriate.We intentionally avoided the use of computer-assisted tools to keep the process simple.

Method
The accessibility problem is an issue that concerns the overall M&S community.We started our exploration as researchers (academics) based on our own perception of the issue.Our literature review analysis in section 2 provided some background of the challenges faced in the simulation field.To further explore the issue, we held a workshop with simulation practitioners at the Modelling and Simulation (MODSIM) World Conference & Expo 2018 in Norfolk, Virginia, USA, on April 26, 2018.An overview of the approach adopted is shown in Figure 2.This includes the main phases of a PSM inquiry: preliminary understanding of the problem (see sections 2 & 4.1), problem structuring (see sections 4.2 & 4.3), analysis (see sections 4.4 & 5), and outcomes (see section 6). the process followed approximately reflects one iteration of the approach discussed in Eden and Ackermann (2001).We next describe in sequence the main activities that took place for each phase of our inquiry: pre-workshop activities, including a description of the participant group, the workshop and outputs (model building), and data analysis.The results of our analysis are presented and discussed in sections 5 & 6.

Workshop preparation
The exploration of the literature (section 2) offers an initial background to the problem.We now consider the workshop organization and preparation.The workshop was held on the last day of the MODSIM World Conference & Expo 2018 (MODSIM).MODSIM is a practitioner-focused conference with most delegates representing both the private and public sectors, especially the military.The academic community makes up a small proportion of the attendees, but it is common for the attendees to hold advanced degrees, especially doctorates.The conference has been running annually since 2007 and is owned and operated by the National Training and Simulation Association (NTSA), an organization that represents simulation interests to the US government.MODSIM usually has around 300 delegates.
The workshop was held in the final 1.5-hour session in a flat room with round tables.The room layout was set in a horseshoe to enable the participants to face the front of the room.Flip-chart paper was used to record the conversations and to produce the workshop outputs since the hotel did not provide whiteboards.During the workshop, one of the authors (YT) facilitated the workshop while AC acted as the scribe.AC had extensive experience creating cognitive/casual maps in workshops (both in academic and industrial settings) and was aware that a short-hand note-taking approach was required to ensure the flow of conversation was not interrupted.
The workshop was advertised during the conference.The intent was to provide a discussion opportunity for the attendees at the end of the conference, which is customary in MODSIM conferences.Interested delegates were invited to provide input into the ways that the accessibility of simulation can be improved.Attendance was voluntary.The advertisement for the workshop, found in the conference schedule, is displayed in Figure 3: Approximately 20 delegates with real-world simulation experience attended the workshop (with some individuals leaving early and some arriving late).Their attendance was voluntary, meaning that this is a convenience sample that was available to us at the time.These delegates are part of the stakeholder community for the accessibility problem as, we believe, that all M&S academics and professionals are part of the stakeholder community.This also means that we, the authors and workshop facilitators, are also part of the stakeholder community.In generic terms, the stakeholders were evenly drawn from the public and private sectors, they covered a wide range of ethnicity, and there were both males and females present.As part of the initial agreement with the participants, their names and details were not recorded for confidentiality purposes, as well as to ensure that individual contributions are not attributed to a specific individual.Anonymity was offered as a means to encourage the open and truthful expression of stakeholders' views at the workshop.
The non-attributable nature of the discussion was the key to the success of the workshop, as, from our previous experience at the conference, we had seen individuals unwilling to speak due to fear of a potential backlash from their customers, especially about any comments that might be perceived as criticism of that customer.As such, no demographic information was collected about the participants, nor were their company affiliations collected; the stakeholders were also reassured that no such information would be made publicly available.This nonattributable requirement might seem extreme, but many simulation companies depend on highly competitive US government contracts, and any association with a written product that is deemed controversial to the customers could have dire consequences and even result in blacklisting them, especially if the media coverage is involved.There is no avoiding the political aspect of government contracts.Though we have made efforts to avoid controversial statements in this paper, this paragraph, in itself, could be deemed controversial.
Attendees of the MODSIM conference were chosen as suitable representatives of the stakeholder community due to the demographics noted for this conference, i.e. private and public sector professional modellers.We were particularly interested in understanding the accessibility problem from an industry perspective.We believe that the conference participants are stakeholders with a vested interest in the issue.It is also fair to state that, as practitioners in the field of M&S, they share ownership of the problem of interest and want to tackle the problem.We believe this interest and concern about the issue is demonstrated by the participants' willingness to attend a voluntary workshop at the end of the conference.Based on the facilitators' personal recollection of the workshop attendees, most had significant real-world simulation experience, which is ideal for the aims of the workshop.We would estimate that their mean work experience is about 15 years, with some workshop attendees having significantly longer experience as modellers.The age demographics focused on late middle-aged individuals.Based on the informal conversations held at the workshop, the participants were from government agencies and the private sector.

The workshop and outputs
The workshop started with a brief introduction.The facilitators (AC and YT) introduced themselves and the aims of the workshop.The ground rules were also explained, making the participants aware that they could leave at any time during the workshop and that anything they said was non-attributable.This was to ensure that people spoke freely and that participants engaged in a frank and honest discussion.
Then the workshop moved on to discuss the proposed topic and construct the causal map.The following question was posed to all participants: "How do we make M&S more accessible?"A top-down approach was used in developing the causal maps, where all attendees were invited to contribute their views on this question.The question was asked in a way that both facilitators and participants tried to learn their way into the problem and the issues that needed to be tackled, as recommended by Pidd (2009).In other words, the intention of the workshop was to explore the views of the participants in relation to the question posed.
As the workshop went on, the discussion organically changed, through the participants' deliberations, to more specific topics.As only one and half hours had been allocated to the workshop, the workshop facilitators deemed it appropriate to allow the conversation to flow in this organic manner.The causal maps were constructed over several flipcharts.As elaborated earlier, there are two different methods for obtaining a group map in SODA.SODA II method was applied in this workshop, which involves developing the cognitive map at the group level (called a causal map).The authors believe that SODA II was more suitable for two reasons: first, due to the limited time (1.5 hrs) we had available, and due to the profile of the workshop participants, who had no prior knowledge of SODA.The scanned images of the maps developed at the workshop are shown in Figure 4.
As it can be seen in Figure 4, the starting topic of "how to make M&S accessible?" is shown in the first flipchart (Figure 4 part 1).Throughout the workshop, the focus moved toward "how do we sell M&S?" as seen in the next flipcharts (Figure 4 parts 2 and 3).This change in focus is not surprising, considering that most participants came from the commercial sector.The facilitator made several attempts to bring the conversation back to the original set topic, but, ultimately, the newly discussed topics were relevant and, hence, were included in the causal map.After the workshop, the facilitator team merged all the parts into one causal map, which can be called the translation process.The translation of the causal map is needed before it is analysed and key concepts are identified.

Output translation
Part of the problem structuring activity was the output translation of a rough paper-based causal map.The translation of the notes, made in flipcharts during the workshop, was needed to improve the readability of the causal maps, as well as to verify the maps and concepts produced.This is alternatively called a tidy-up process by Ackermann and Eden (2001).A facilitator can perform the following tasks during the translation process: add links between concepts, identify key concepts, and/or identify clusters (Pidd, 2009).Pidd (2009) also argues that it is useful for a facilitator to add links between concepts to show a group that their ideas fit in a synergetic way.In fact, it is one of the facilitator's responsibilities to ensure that a group map is sensible and useful.
To translate the paper-based causal maps into an electronic format, we used the freeware Mental Modeler software package (mentalmodeler.org),developed by a consortium of US universities and industry partners.Mental Modeler was designed for use primarily with another cognitive mapping tool, called Fuzzy Cognitive Mapping (Kosko, 1986).It can also be used for SODA.The output of our translation into an electronic format is shown in Figure 5.The key purpose of the translation was to remove ambiguity in the concept definition by adding more details where possible.Additional links between concepts were also included, wherever this was considered necessary.
The translation also involved merging some of the concepts that were felt to be related.As a result, the 37 concepts found on the original workshop notes were reduced to 32 concepts in the electronic version.An example of merging can be seen with "Position self for buzzwords," which was a standalone concept, being merged into the "Good terminology" concept.It is worth noting that the "Good terminology" was linked to "Use common terms" in the electronic version and not in the paper-based causal map; this was due to these two concepts being linked during the discussion but not recorded in the flipcharts.
It should be noted that the thickness of the arrows reflects the extent that the connection between the concepts was discussed in the workshop based on the facilitators' recollection of the workshop discussion.The thickness is not the focus of our analysis below, but we have kept it in the diagram for completeness purposes.

Analysis
The next stage in our process was to gain an understanding of the problem through analysis of the causal map.To explore a causal map, we start with the concepts that have a high number of interactions (high traffic concept).These are the head, tail, and central concepts mentioned in section 3 above.After the identification of key concepts, we perform cluster analysis.According to Eden (Eden, 1988(Eden, , 1989(Eden, , 2004)), cluster analysis is one of the most widely used approaches to analyse cognitive/causal maps.In a nutshell, this involves identifying clusters that are formed by looking for similarities and interactions among concepts.Eden (1988) also argues that the manageable size of a cluster should be less than 30 conceptsdepending on the size of the problem.Lastly, we develop a feedback loop of the key concepts of the map with the view to identifying any valuable insights about the relationships between these concepts (Eden, 1994).The overall intention of the analyses is to gain a better understanding of the problematic situation.The analysis usually generates a means to identify the core elements of the problematic situation (Eden, 1988;Pidd, 2009;Rosenhead, 1996;Rosenhead & Mingers, 2001); hence it can provide clarity and help identify solutions.

Analysis of the causal map
This section presents our analysis of the artifacts developed at the workshop.As discussed in the section above, the workshop was successfully held with interesting dialogue from most of the stakeholders.This dialogue has been translated into a casual map (merged map), which forms the basis of the analysis presented in this paper.We follow a three-fold analysis approach to analyse the casual map as explained in section 4.4 above.We first explore the key concepts of the causal maps.Next, we use clustering to reduce the integrated causal map to form a simpler casual map.In the end, we analyse the feedback loops present in the clustered causal maps.

Analysis of key concepts
Analysis of key concepts involves looking for hightraffic concepts, that is, any concepts with the most incoming arrows.This is called a "head" concept.As mentioned in Section 3, the head concepts sit at the end of the chain of argument, and it can be one of the goals to consider in a problem of interest.We next identify "tail" concepts, which are concepts that have no arrows leading into them (Georgiou, 2009).A "tail" concept can be one of the options or constraints that might have a high impact on the problem.For example, the "Sell your products" and "Better awareness" concepts are head concepts because there are many arrows pointing towards them, while the "Multi-disciplinary application" is an example of a tail concept.In addition, the concepts that have both incoming and outgoing arrows could be considered "central" concepts or bridging concepts (Ackermann & Eden, 2001;Eden, 2004).
We now analyse and identify the key concepts of the causal map.Table 1 presents our identified head, tail, and central concepts.A concept with no incoming arrows defines a tail, and those with no outgoing arrows define a head concept.A central concept usually has both incoming and outgoing arrows (Eden, 2004).It should be noted that the key concepts have the most arrows associated with them.

Head concepts
As indicated in Table 1, the "Sell your products" is one of the key head concepts of the map, based on the number of incoming arrows into the concept.In addition, based on the number of incoming arrows, "Sell your products" is indeed more popular than the "Make techniques accessible" concept, which was our initial focus.We believe there is an indirect link between these concepts in that the stakeholders considered the statement "Make M&S more accessible" equivalent to "How do we sell M&S products?".This could be due to the way the stakeholders use M&S.Most workshop stakeholders are practitioners who focus on selling their products (both externally and internally in their respective organizations).We could argue that the resulting map could have been different if the stakeholders, present at the workshop, were mainly academics.
According to our interpretation, the concept "Sell your products" refers to the selling of M&S software packages and modelling solutions to new potential customers.There are ten concepts that feed into this concept, out of which the most prominent, based on the extensive discussion that took place at the workshop, include the variety of products, easiness in determining usefulness, and awareness for those individuals with money (these individuals were interpreted to mean decision-makers).This suggests that to sell M&S, there needs to be a focus not only on awareness of existing products but also on enabling potential users to understand their usefulness.Determining the usefulness of M&S is not easy, neither is determining the return on investment (ROI) of M&S (Oswalt et al., 2012), as discussed in the literature review section.
Another head concept is "Make techniques accessible."We note that this concept differs from our original concept of accessibility.From the stakeholders' perspective, accessibility means that M&S should be easy to use or, at least, it should be obvious how to develop the skills to use it.One might expect that more accessible techniques are easier to sell, and hence "Make techniques accessible" should, indirectly, feed into "Sell your products."However, in retrospect, this is not necessarily true.What matters is that the method is useful and, preferably, cheap to purchase, not that a technique is frequently used, to be accessible to users.The disconnect between the "Sell your products" and "Make techniques accessible" concepts support this claim.
Table 1 also identifies three additional central/ head concepts.These are "Better awareness of M&S," "Better education," and "Easy to determine if useful."The "Better education" concept is discussed first, alongside the "Better awareness" concept.There are two concepts that directly relate to awareness of M&S: "Better awareness" and "Better awareness to people with money."Technically, the second concept is a subset of the first; we merge both concepts in the clustering of the causal map below.By awareness, it is meant awareness of M&S by people not involved in the M&S industry.The concepts have an indirect impact on the "Sell your products" concept already discussed.They are supported by the "Better education" concept.A better educationcourses, certificates, and degreesmay be able to help enhance some level of awareness of the M&S discipline among people (with the potential of investments) outside of the community.The next important concept is "Easy to determine if useful."It has a strong direct impact on the "Sell your products," which is one of our key high-level goals.It makes the most sense since the potential customers will wish to judge the return of investment (ROI) for simulation.As already mentioned, determining the ROI of simulations is difficult (Oswalt et al., 2012).Purchasers of simulation products, not the user, might not pay attention to the process but are concerned with the outcome of a simulation use.Hence, the easier it is to determine the usefulness of the product, the better the chance of selling the method.

Tail concepts
According to the number of arrows shown in Table 1, there are two key tail concepts in the causal map: "Good (industry) use cases" and "Multi-disciplinary application."The "Good (industry) use cases" concept can be considered one of the important concepts on the map.It is because this concept is a central/tail concept (high traffic from both incoming and outgoing arrows).This type of concept is important because it bridges ideas, influences other concepts, and could be one of the key focuses in a map (Eden, 2004)."Good (industry) use cases" shows an indirect impact on "Make techniques accessible" and an indirect impact on "Sell your products," which are our two main goals.It can be interpreted that if there are more good use cases from the industry, it will elevate the success of both accessibility and selling opportunities.By good use cases, it is meant written examples of the application of simulation that clearly demonstrate its worth.Other disciplines have these use cases; for example, the Rochem case study is used in operations management as a clear example of why problem structuring methods are important (Slack et al., 2013).Moreover, this implies that investing in better use case studies might help the M&S community in disseminating the use of M&S.This point was also highlighted by Hamill (2010).
According to Eden (2004), a tail concept could be considered a constraint that might have a high impact on the problem.In our case, "Multi-disciplinary application" can have a negative impact on resolving the accessibility and selling problems.This is further considered in the discussion section below.
In this sub-section so far, we identified the key concepts in the causal map developed.Table 1 indicates 7 out of the 32 concepts from the map.The head concepts can be considered goals or important aspects of the problem, and tail concepts can be constraints.In our analysis, we identified three additional central/head concepts that feed, directly or indirectly, into the original goal concepts.These can also be called sub-goals and include better awareness of M&S, the need for good industry use cases, and ease of determining M&S usefulness.Based on this analysis, we next cluster the constructs in the casual map and present a new casual map of these clusters.

Clustering of the causal map
Since there are 32 concepts in total, it is difficult to "make sense" of the map.Miller (1956) suggests that humans can handle sevenplus or minus twoconcepts at once.To aid with understanding, there is a requirement to reduce the number of concepts in our casual map.One of the approaches to tackle this is to use clustering (Eden, 1989).The clusters were formed by looking for similarities and interactions among concepts.
Based on the key concepts (heads/tails/central) identified in section 5.1, we identified clusters that define the key strategic options within the focusing situation.Figure 6 shows the clustering of our causal map.This clustering was generated post-workshop through dialogue amongst the project team.
Our causal map was clustered into six clustered concepts: accessibility, awareness, ease of determining usefulness, education, multi-disciplinary, and selling.Accessibility refers to how easy the techniques are implemented for a novice in terms of knowledge requirements and resource requirements.Awareness is related to the awareness of potential users of M&S and its capabilities.The ease of determining usefulness is self-explanatory.Education refers to the availability and quality of educational material both in written form and through courses.Multi-disciplinary refers to how much M&S is used over multiple academic fields.Finally, selling refers to how easy it is to sell M&S, as a solution, to problem owners.
The six cluster concepts are all linked based on the connections identified at the workshop.We can see, for example, that the "Selling" cluster concept is influenced by the "Ease of determining usefulness" cluster concept because the "Easy to determine if useful" concept influences the "Sell your projects" concept in the original causal map.There remains a negative effect from the multi-disciplinary nature of M&S affecting education and awareness.As discussed, this negative effect occurs because different discipline groups are developing different guiding principles and ontologies for M&S, which are not necessarily compatible.This, in turn, can negatively affect the ease of determining the usefulness and selling of M&S.

Feedback loop analysis
One key factor in understanding a causal map is the presence of feedback loops.An analyst will look for both reinforcing and balancing feedback loops.Feedback loops in SODA focus on identifying feedback dynamics within a causal map.However, they do not calculate the magnitudes of those dynamics.SODA maps can be translated into system dynamics computerized simulation models to obtain the "behavioural changes over time and the resulting magnitude of their impact along causal chains and feedback loops" (Georgiou, 2009, p. 706).These help us gain a better understanding of the simulation accessibility problem based on emerging feedback loops in the causal map.We note, however, that this paper does not aim to determine the magnitude of the effects among these concepts.
What is noticeable about our causal maps is that there is no feedback loop in the casual map shown in Figure 5.The lack of feedback loops implies there must be some head nodes (with no outgoing arrows) within the map.The two main head concepts, which also represent the aims of the workshop, are "Sell your products" and "Make techniques accessible." Figure 7 presents a simplified causal loop diagram of the main relationships as seen in the clustered causal map (Figure 6), reproduced in Vensim.
This show one main reinforcing feedback loop between five concepts: selling, awareness, accessibility, education, and ease of determining usefulness.This could be virtuous, as an increase in "accessibility" leads to an increase in "education," which, in turn, leads to an increase in "ease of determining usefulness," "selling," then "awareness," and then back to increase in "accessibility."However, this loop could also be vicious, a decrease in "accessibility" leads to even more decrease in "accessibility."Which type of loop the system is experiencing will depend on the current state of the system now.To determine the current state of the system would require not only more data but also overcoming our biases about the system, i.e. our tendencies, as humans, to have attribution errors and focus on salient aspects of the system (Repenning & Sterman, 2002).Also, our model may be more complex than just this single feedback loop, and, as such, our analysis cannot be reduce to such a simple interpretation.
The feedback loop analysis of our causal map provides insight into the system (Sterman, 2000).It shows us that there is potential for accessibility to increase, assuming that education in modeling and simulation is increasing.However, the positive relationship between education and accessibility depends on the current state of the other concepts, including the negative impact of "multi-disciplinary." What this means is that we remain uncertain as to whether "accessibility" is currently part of a virtuous or vicious cycle.

Stakeholder reflection and validation
In this paper, we use SODAa PSM technique that offers a rigorous and relevant processto gain a common and shared understanding of the issue of concern, in our case accessibility, as perceived and experienced by the stakeholders.Stakeholders, simulation practitioners who have a stake in the progression of simulation tools and their use, were involved in this process, which aided the process of developing the theory and concepts represented in the maps.We, furthermore, use reflection and feedback to further elaborate on the methods used and actions derived as a result of our analysis.To ensure that our results are credible, we incorporate the following in the method followed: 1. designed a process with refined steps of the SODA methodology, that is relevant to the issue of concern, the accessibility problem, and 2. we also include stakeholder feedback on the causal maps developed/artifacts and resulting actions.
SODA offers a rigorous action research process that allows for the views of stakeholders to be represented in practice (Eden & Ackermann, 2018).We outline the approach we followed here so that other researchers can reproduce the steps followed in this research as described in section 4, with an overview presented in Figure 2. We also took measures to limit the biases present in our approach.For example, we did not prep the stakeholders before the workshop, and we encouraged a forum of open expression of opinions during the workshop.However, not all bias could be removed, which we discuss in the limitation section.
Secondly, we incorporated stakeholder feedback and reflection into the process to ensure that the causal maps and outcomes of our exploration of the issue are relevant.We presented the results of the analysis, including an electronic version of the casual map (and clustered casual map), at the following MODSIM conference (Thaviphoke & Collins, 2019b).A number of the stakeholders from the original workshop were in attendance and were able to review the causal maps.From our interactions with the stakeholders present at MODSIM 2019, at our formal presentation, and during informal discussions after the presentation, there was a general sense that the casual maps, and the results were representative of their views about the topic.This serves as an indication that the maps and key concepts produced from our analysis were representative and relevant.

Discussion
The final stage of our study process was to draw insights, and recommendations, based on the analysis of the causal maps; including understanding the limitations our method implies.This paper set out to understand the accessibility problem of modelling and simulation by considering the factors that affect the use of simulation.The contribution of this paper is twofold.Firstly, the study provides an understanding of the main factors that affect the usability and accessibility of simulation, as experienced by expert modellers from the industry.This is useful to both academics and practitioners in developing a common understanding of the barriers that the simulation community faces.Based on the findings of this study, we provide recommendations intended to improve the current state and future of simulation as a field, which is the focus of this section.
Secondly, the paper illustrates a novel context where SODA, a problem structuring method, is applied to help us gain a common understanding of the simulation accessibility problem.This paper has shown the value of using SODA to explore the issue of concern, that of the simulation accessibility problem, through a participative approach.It is noted that we apply SODA in a less traditional setting, such as an organisational setting, where the participants are stakeholders to the problem and have control over the resulting actions arising from the workshops.In our setting, our stakeholder base is the whole simulation community.The participants are simulation practitioners, who use simulation as part of their day-to-day job and have a vested interest in the issue, but they are only one sub-group of the stakeholders.We had to start with one group of stakeholders.We were able to adapt the traditional SODA approach to the problem and setting at hand, even though the stakeholders who were not familiar with the methodology.This was possible due to the flexibility of the approach, as discussed in Eden (1988).We believe that the approach adopted and data analysis undertaken helped us to identify a range of potential actions, a call to action for the simulation community, that will, at a strategic level, help to ensure that the field remains current and continues to flourish.The group of stakeholders, together with the facilitators, arrived at a commonly agreed casual map.The map is comprised of a systemic view of the problem, the factors that influence the accessibility, and ultimately the uptake of simulation.As shown in the paper, the outputs of the model-building process at the workshop are messy and the result of a negotiated understanding of the issues between the workshop participants and the facilitators (White, 2009).The validation of the outcomes of the exploration is embedded within the model-building process through the collaborative inquiry process (Champion & Wilson, 2010).The validity of our outcomes was reinforced by presenting and discussing these results at the following year at the same conference with a similar audience.
The process of analysis of the problem alternates between a convergent and divergent process (Smith & Shaw, 2019).Our initial exploration started with a clearly defined problem statement based on our understanding of the problem and existing literature (Preliminary Understanding -see Figure 2).The model building stage took place at the workshop whereby it started with our original problem statement.The stakeholders were invited to identify factors that affect the accessibility problem through a divergent process.A wider range of concepts emerged, which allowed the facilitators to delve deeper into the issues together with the stakeholders.A convergent process took place subsequently in the analysis stage, where the authors identify the key influencing factors and actions that can improve the current situation.Ultimately, our maps and analysis include the views and contributions of a group of industry stakeholders.The maps presented here can be further enriched by exploring them with other practitioner groups from different disciplines.The approach detailed in this paper can be further used and/or adapted to analyse the state of other OR modelling tools and techniques.We next reflect on our findings and implications for the M&S as a field, followed by recommendations based on our findings.

Reflections on insights gained from the causal maps about the accessibility of simulation
Our original aim was to consider the accessibility problem.As noted, our analysis (section 5.1) revealed that "Sell your products" was the most popular concept discussed at the workshop, followed by "Make techniques accessible."It could be argued that, to some extent, these two concepts were most popular due to the aim of our workshop.However, what was interesting was the shift of focus from the accessibility of simulation to the concept of selling.Accessibility is about making M&S products available, which can explain why selling became such an important concept; this finding is in line with the observation made by Morrill (2007) about the ability of selling PSMs, which is another OR method.This suggests that selling and accessibility are connected goals that we should aim for as a community, and we should identify means that can effectively sell the benefits of simulation tools.A better understanding of which means should be considered might be achieved through looking at the connections in our causal map.
Our analysis revealed some central/head concepts, which we called sub-goals, that enable the achievement of the two high-level goals: accessibility and selling.These were: "Better awareness," "Better education" and "Easy to determine if useful."We also found two-tail concepts, "Good industry use cases" and "Multi-disciplinary application."Out of these, "Good industry use cases" is an option that can be taken to achieve the goals identified in the map.Indeed, this was a central concept in our map, due to the high traffic of in-and out-arrows.On the other hand, the "Multi-disciplinary application" acts as a constraint or obstruction to achieving the goals, which suggests that it can negatively impact the selling of M&S and the accessibility of its techniques.

Accessibility vs. Selling
Let us first attempt to understand the shift in emphasis from accessibility to selling concepts at the workshop.This could be explained due to the fact that our workshop participants, who were senior M&S managers and practitioners, are focused on selling their M&S products.Naturally, academics and practitioners in industry have different aims.Academics focus their efforts on developing, exploring, and validating new methodologies and applications and tend to worry less about how these are implemented (Collins & Hester, 2016).Instead, practitioners in industry develop simulation products, and to convince the clients of the value-added, they rely on providing evidence of the benefits that modelling tools offer to their organisation.This is part of selling.Including the industry's voice in academic dialogue can bring up these different perspectives, and so they can be taken into account.This paper has shown one approach to making that voice heard and bringing their perspective into the dialogue.
On the other hand, it would be useful to also consider whether there is a difference in the meanings we place on the concept of "accessibility" due to our different backgrounds, as the authors of this paper are academics, and our stakeholders are practitioners.From an academic point of view, accessibility means making M&S more accessible to a wider group of people, for example, by offering M&S knowledge and skills, including open-access M&S software.This was the view the authors had when entering the workshop.However, from an industry perspective, accessibility means more people using M&S products.Since most M&S products, with the exception of freeware like Netlogo 1 , require licenses, this, in essence, can mean selling licenses or consultancy expertise to build models for a client.This observation provides evidence that there is a disconnect between industry and academia, which confirms the finding of a survey reported in Ranyard et al. (2015) that assesses the scope of OR practice.As a result of the workshop, our view and definition of accessibility as a concept have converged into a wider and richer definition.Again, this shows the value of using a problem structuring method, as shown in this paper.

Multiple-disciplinary
We now consider the impact of the multi-disciplinary concept.This concept was found to have a negative effect on simulation awareness and education in our causal map.Multi-disciplinary knowledge and experience have been advocated as a key factor for developing successful simulation studies (Balci, 1989;Robinson & Davies, 2010).Indeed, an individual simulation study benefits from multi-disciplinary knowledge.For example, team members with different expertise in the problem domain, computer science knowledge, communication, and people management skills, problem-solving skills etc., can all contribute to a successful study (Robinson & Davies, 2010).However, this is not the meaning attached to this concept here.The "Multi-disciplinary" concept used throughout the workshop refers to the fact that "multiple academic disciplines use simulation;" thus, to avoid confusion, we use the word multiple-disciplines instead from here on.
As discussed in the section 2.2.3., the simulation community is expanding into specialist application areas, but these communities are more disconnected (Collins et al., 2022).It is this disconnection that should be of concern for the M&S community because as time progresses, the ontologies become disconnected.As a result, what is an accepted standard in one area of simulation might be no longer accepted in another (Collins et al., 2015).For example, the word validation is being used to mean different things by different groups, as the years go on (Sargent & Balci, 2017).Hence, we could end up in a situation where you have two trained simulation specialists who cannot communicate due to their different educational backgrounds.
The disconnect happens because of the demands of the specialist application areas.If a simulation is going to be used in a specialist application area, then it needs to be credible to existing practitioners in that area.This means that a simulationist needs to adapt their practices to the requirements, problems, concerns, and standards of that area if their simulation is going to be accepted.For example, defining the simuland is very important in healthcare training simulation studies, whereas that is almost assumed in the general simulation studies (i.e. a couple of paragraphs are used, at most, to describe it); as such, if the general approach to describing a simuland is used for a healthcare training application, it will be rejected by the healthcare community and seen as uncredible.To exemplify this point, some readers might not have even heard the team simuland before; it means the real-world systems of interest to the simulation study (Petty, 2010).
Why should we care about the splinter of M&S into sub-specialities?We believe that the development of their own ontologies and terminologies (or, more importantly, interpretation of simulation terminology) from the sub-specialities can lead to confusions within the overall simulation community (Collins et al., 2015).For example, Augusiak et al. (2014) complete a review of the use of the word validation for decision-support models.It is interesting to note that they do not mention Osman Balci's work, not even once (a researcher with dozens of highly cited articles on the subject).One may comment that this omission is fine because they were reviewing validation of decision-support models; however, the authors explicitly reviewed studies using simulation (a dynamically changing representation of a system), especially agent-based models.This further demonstrates the mismatch in simulation terminology.They also introduced new terminology not seen in the traditional M&S community but important in the ecological community, e.g.ecological validation that is focused on whether the data collected for use in the model was collected in a manner that is representative of the real-world system (e.g.opinion data collected on how an individual would react to a certain situation is different if they are present in that situation or merrily just been presented with a prose description of the situation).The irony is that Augusiak et al. wrote  The use of simulation by multiple disciplines is a double-edged sword.On the one hand, it increases the use of M&S through new disciplines, and multidisciplinary knowledge is useful for a successful simulation study.On the other hand, it makes it harder for newcomers to simulation to easily access M&S for the reasons discussed above (e.g.different definitions of terms, etc.).Further discussion on the splitting of the M&S community can be found in Collins et al. (2022).Finally, it has been suggested that hybrid simulation, an emerging simulation methodology, might help overcome the barriers between disciplines (Tolk et al., 2021).

Recommendations for improving the uptake of simulation
In what follows, we attempt to make recommendations that are based on the findings of our analysis which could improve the uptake of simulation.Our analysis has identified four central concepts in the map; these are "Better awareness," "Better education," "Easy to determine if useful," and "Good industry cases."These concepts are also connected to each other in a reinforcing loop, which strengthens their impact (see section 5.3).Thus, we base our recommendations on these themes.

Improve awareness of simulation
Creating awareness about simulation and its use could offer a basis for a better understanding of simulation.This would also mean creating awareness about the uses of simulation to a wider base, from organisations to lay individuals.This would create a wider basis of understanding of its benefits and uses.This could include creating easy-to-understand tools that those not knowledgeable about simulation can use.Simple models have been advocated to be useful to support learning and understanding (Penn et al., 2020;Tako et al., 2020).More could be done to expand the use of simulation in everyday contexts.
Research in participative and facilitated simulation, for example, that aims to involve stakeholders in the simulation study, can help in this direction (Kotiadis & Tako, 2018;Robinson et al., 2014;Tako & Kotiadis, 2015).The Simtegr8 study (Tako et al., 2019), in particular, makes some initial attempts to create models that are simple and easier to understand by those with limited technical knowledge in simulation.These are called user-mode models.They are reduced versions of normal simulation software to help workshop participants understand the model in relation to the context modelled.Their success is mixed, and more research is required to ensure that the right level of detail is included.This creates the basis of our claim that using simple models can help with improving awareness of simulation.
Another avenue for improved awareness that could be pursued is to create widely available and open-source materials that summarise the applications of M&S across its application areas and the key differences between them.This would provide an understanding of the disparities of the different M&S applications.It is hoped that understanding each other's points of view will help bridge the divide occurring in the M&S disciplines.This is furthermore linked to helping with determining the usefulness of simulation and offering support, especially for novices in simulation.
It has been argued that M&S non-experts view the outputs of a simulation project through visualization (Collins & Knowles Ball, 2013).Since our novice would evaluate a simulation, and by extension M&S as a whole, through its visuals, it highlights the importance of visualization.However, Simulation visualization is usually given secondary importance in a simulation project (A.J. Collins, D. a. Knowles Ball, & J. Romberger, 2015).Worse still, simulation visualizations have "a mesmerizing effect on simulation novices" (Banks & Chwif, 2011) and could be used to misrepresent, or over-represent, the capabilities of the simulation (Roman, 2005).This shows the importance of creating awareness about key characteristics of simulation, such as multiple replications or randomness in simple-to-understand terms by a non-technical audience.The point also links to the "Ease of determining usefulness" of M&S and highlights that determining the usefulness of M&S is not easy and requires help.The outputs of our workshop suggest that this help comes from the education of M&S.

Improve simulation education offering
The clustered causal map, shown in Figure 6, indicates that the education of M&S is key to its ease of determining usefulness, accessibility, and awareness; which, by extension, education is key to the selling of M&S.As such, we believe that a focus on better education of M&S is critical to its growth and continued future.
What do we mean by education?This involves distilling the knowledge that we already have about M&S in easily understood formats that are accessible to a wide array of individuals with varying knowledge and skill levels in M&S.An example of this approach would be Wired magazine's "5 Levels" series (https://www.wired.com/video/series/5-levels).This means that teaching simulation becomes part of a wider curriculum starting with school and then at university.In addition, it would mean teaching simulation to a wider student base at undergraduate and graduate levels, and in various disciplines such as geography, marketing, social sciences, etc.
Another source of educational material is the use of case studies.A number of simulation education panels at the Winter Simulation Conference call for the use of good case studies to be shared between industry and academia (van der Zee et al., 2010;van der Zee et al., 2018).Equally, the need for good case-studies in PSM has been argued in Collins et al. (2019).

Use commonly accepted reporting standards
We argue that establishing and using widely accepted reporting standards of simulation models can help to improve the reporting of simulation models and to create the relevant evidence base that supports the use of simulation.One such example is the Strengthening The Reporting of Empirical Simulation Studies (STRESS) guidelines (Monks et al., 2019).Better reporting of empirical simulation studies can help consolidate previous work and help modellers gain a better picture of the model and the reliability of the results.Adopting commonly accepted standards amongst the different disciplines would help to overcome the barriers faced by the multiple-disciplinary use of simulation by establishing commonly accepted standards between the different disciplines.However, as discussed in Collins et al. (2015), achieving such a universally accepted standard would be difficult.We next consider the limitations of the work carried out in this research.
The final step in the SODA process is to discuss the analysis and resultant recommendations from the workshop with the stakeholders.Though we were able to discuss this with some of the stakeholders at the following year's MODSIM World conference, this only represented a small fraction of the stakeholders of this problem as, we believe that, the stakeholders are all of the M&S community.As such, this paper represents a first attempt to engage the M&S community with our findings, and hope that this paper provides a catalyst for future discussion on the accessibility problem.By making the simulation community aware of this problem, we hope this paper will act as a catalyst for efforts to bridge the divide between the different simulation user groups, those that currently use simulation and those that would like to use it.

Limitations
The findings of our work emerge from the application of a problem structuring method to exploring wicked problems.Due to the multiple-disciplinary use of simulation, we can confidently say that the accessibility and selling of simulation is a wicked and ill-defined problem because there is a range of stakeholders with potentially conflicting values of interests.This leads us to consider the reliability of our findings.Obviously, the output of our study is the result of a participative and collaborative process providing insights based on the group of stakeholders that attended our workshop.We only had access to a small group of stakeholders who attended the workshop at the MODSIM conference, and they attended our workshop on a voluntary basis.We, furthermore, note that the views expressed are representative of our participant base, who were simulation practitioners working, primarily, for the US Department of Defence (DoD) and, as such, could be more United States-centric views.As a result of this, we believe that if a different cohort of stakeholders was used, then the workshop would have generated a different causal map.Future research could include conducting a similar workshop with different stakeholders working in other domains or disciplines (healthcare, manufacturing, etc.) and also different countries (the United Kingdom, Europe, China, etc.).It would be interesting to see whether the dynamics between the factors identified in our map would differ.
To ensure our study received an Internal Review Board (IRB) exception, which is common in US university's ethical approval process, no personal information was collected from any of the workshop participants, and all participation was voluntary.As a result of this, we are unable to provide demographic information on the stakeholders (though the authors knew most of the stakeholders and would summarise that they are mid-level M&S managers in government and industry).
We need to also consider the influence of the researchers on the outcomes of the study.First, we note the impact of the facilitators on the actual modelling process at the workshop.Facilitators may not be neutral in the exploration of the problem and might have their own political and social interpretations of the issues.As discussed in Ackermann (1996), with a given set of information, different facilitators may influence the process of the meeting differently.Furthermore, the influence of the authors as academics (two of the authors are researchers experienced in simulation projects in practice) on the analysis could have been affected by our subjective interpretation of the concepts and their interrelations between them in the resulting causal maps produced.As a result of these limitations, we must be careful in any generalisations drawn from this research.As such, our findings should be viewed more as suggestions than definitive conclusions.

Future research
To enhance the insights gained through the research carried out in this paper, several different follow-on studies could be conducted to further enhance and confirm our findings.These follow-on studies could overcome any subjectivity present in our study.The approach followed in this paper is helpful in performing an initial exploratory evaluation of the problem.Next, to further confirm or refute the finding of the work presented in this paper, a survey can be used to collect the opinions of a wider practitioner base.
We started our exploration with one group of simulation practitioners that voluntarily attended our workshop at the MODSIM conference; we acknowledge that they represent a limited group of stakeholder basis that have a vested interest in the simulation accessibility problem.It would be interesting to reach out to additional stakeholder groups, including practitioners and academics, representing specific application domains and to explore further the issues identified in our analysis from their perspective.This could be achieved by running similar workshops to the one presented here or questionnaire surveys.Different groups of stakeholders could have produce different maps.For example, a wellknown problem related to difficulties faced in simulation with data collection did not come up at this workshop, which different participants could have brought up.Having a mixture of practitioners and academics in the stakeholder group could have been useful in collecting more diverse views.
The next step of the enquiry presented in this paper is to present the findings of the workshop to the wider M&S community (academia and professionals), to elicit further feedback and to reach a commonly agreed-upon actions to address the simulation accessibility problem.This paper is, in itself, an attempt to do just that.We hope that the paper provides a call to action to initiate the dialogue on the accessibility problem!

Conclusion
This study provides our initial exploration of the simulation accessibility problem.We use SODA, a problem structuring method, to analyse the accessibility problem in a systematic way at a simulation community level.The process followed has helped us to gain commonly shared views and insights into the problem and allowed us to offer recommendations in the form of actions that the community should take.Based on the analysis undertaken, we pose that to solve the accessibility problem, the simulation community should aim to improve the selling of simulation techniques and to open up its accessibility to a wider user base.This could be achieved through better awareness of simulation through the availability of educational resources, among others.Furthermore, we highlighted the multiple-disciplinary problem and its effect on the wider simulation community.We hope that this research will serve as a call to arms for the community to come together and to think strategically about the future of simulation as a field.

Figure 1 .
Figure 1.Example of causal links between key concepts, where colours of the arrows refer to the type of relationship (blue refers to positive and red to a negative relationship among nodes).

Figure 2 .
Figure 2. Overview of the steps of the study.

Figure 3 .
Figure 3. Advert for the workshop found in the conference schedule.

Figure 4 .
Figure 4. Scanned images of flipcharts from the workshop, shown in order of creation.

Figure 5 .
Figure 5.The electronic version of the casual map developed post-workshop (adapted from the map found in Thaviphoke and Collins (2019a)).

Figure 6 .
Figure 6.Version of the causal map where the concepts have been clustered (adapted from the map found in Thaviphoke and Collins (2019a)).

Figure 7 .
Figure 7.A Vensim diagram of the main reinforcing feedback loop between the concepts in the clustered causal map.

Table 1 .
Key concepts extracted from the causal map.
in their paper that "confusing terminology is one of the main obstacles to get a good understanding of what model validation is, how it works, and what it can deliver."