Skip to main content

Conceptualising the value of simulation modelling for public engagement with policy: a critical literature review

Abstract

As we face complex and dynamically changing public health and environmental challenges, simulation modelling has come to occupy an increasingly central role in public engagements with policy. Shifts are occurring not only in terms of wider public understandings of modelling, but also in how the value of modelling is conceptualised within scientific modelling communities. We undertook a critical literature review to synthesise the underlying epistemic, theoretical and methodological assumptions about the role and value of simulation modelling within the literature across a range of fields (e.g., health, social science and environmental management) that engage with participatory modelling approaches. We identified four cross-cutting narrative conceptualisations of the value of modelling across different research traditions: (1) models simulate and help solve complex problems; (2) models as tools for community engagement; (3) models as tools for consensus building; (4) models as volatile technologies that generate social effects. Exploring how these ideas of ‘value’ overlap and what they offer one another has implications for how participatory simulation modelling approaches are designed, evaluated and communicated to diverse audiences. Deeper appreciation of the conditions under which simulation modelling can catalyse multiple social effects is recommended.

Peer Review reports

Background

The COVID-19 pandemic has placed simulation models and modelling practice in the spotlight. Diverse publics are increasingly engaging with simulation modelling on issues ranging from health to the environment, and demands for transparency and access to model results are growing. The purpose of models and how they are made useful are now pressing topics of both scientific discussion [1, 2] and public conversation [3]. This has also prompted calls for greater transparency and scrutiny of models [4] and burgeoning commentary on ‘best practices’ for responsible, transparent modelling to ensure that these tools serve society’s best interests [5,6,7].

Subsequently, simulation models and their communication have been transforming. In the early days of the COVID pandemic, as governments and publics tried to make sense of the new virus, ‘Flatten the Curve’ was introduced alongside hand-washing as a key public health message. Flattening or bending the curve became a familiar visual, representing modelled projections of the effects of slowing the spread of an infectious disease outbreak. By April 2020, Forbes magazine declared ‘Flatten the curve’ an historic disease image [8]. A form of societal activism coalesced around modelling, for example, citizen scientists developed models as decision aids for office managers wondering whether and when they should send employees home [9]. With the availability of COVID vaccines in 2021, public interest in modelling turned to the question of how to safely open up alongside vaccine rollouts. Modelling practice responded to the escalating public attention by exploring new avenues for making model results publicly accessible [10]. Meanwhile, questions began to emerge around whether we ought to have other ways of framing and narrating the COVID crisis beyond the ‘obscure predicates and designs’ of modelling [11]. In the midst of such questions and efforts toward transparency, however, broader public understandings and interpretations of models and modelling practice remained somewhat unclear.

The terms 'model' and 'modelling' are used to refer to a wide range of practices including statistical modelling, systems modelling, microsimulation and economic modelling, and modelling techniques of data science. At times, the term ‘model’ may simply be used as shorthand for something that is perceived as an ideal example of a phenomenon, for example, a policy or desirable behaviour [12]. In this article, we do not focus on any particular type of model or modelling practice, however, we are interested in the broad category of simulation models aimed at forecasting future trajectories and exploring potential impacts of policy options. Simulation modelling in this context includes analytic methods of complex systems science such as system dynamics modelling, agent-based modelling, and discrete event simulation. These methods have been widely applied to sectors such as engineering, economics, defence, ecology, and business since the mid-1950s, and more recently to healthcare and population health [13,14,15,16,17,18]. Particularly, we are interested in how these models are conceptualised in relation to networks of social relations that are harnessed to respond to future national and global challenges.

The social dynamics of modelling are explored in detail in the field of participatory modelling. Participatory approaches to modelling practice are varied, but research in this area tends to be oriented toward the engagement of stakeholders and community members in modelling practice [19]. Emphasis is placed on effective communication between modellers and stakeholders, as well as among diverse stakeholder groups [20]. Understanding how core participants—as the ‘first audience’ of a model—engage with and understand models may provide clues as to how models are communicated and used with broader audiences. Policymakers engaged in participatory modelling, for example, may require models to provide concise advice accompanied by “a plausible narrative” that is relevant to areas of immediate policy concern [7]. The value of modelling is often located in participatory engagements with the model: participants are able to appreciate complexity of issues, understand the crucial role of data availability, recognise the range of influences at play when contemplating policy and planning decisions [21], or confront sensitive issues with other policy experts [22]. Much of this research investigating the value of participatory modelling for stakeholders tends to be based on case studies of modelling in particular sectors such as public health, or environmental science [23, 24]. As illustrated by the multi-dimensional environmental, socio-psychological, economic and political impacts of the COVID pandemic, however, crises, and potential solutions, are rarely confined to single disciplines or sectors [7].

Recent critical approaches to participatory modelling within science and technology studies have begun to consider how health modelling could be informed by approaches in other sectors which explore how scientists and lay publics can more effectively collaborate [25, 26]. Yet there is generally limited guidance about what learnings from different research traditions have to offer one another, or how issues of transparency and accessibility can travel beyond the core participants involved in modelling projects. In cases where models circulate with lay publics or decision-makers who have not been intimately involved in participatory modelling—as happened with COVID models and continues to happen with climate change modelling—the ramifications of these broader engagements and the value models hold in wider societal contexts remains unclear. To generate a deeper understanding of how models and model findings are made valuable through their circulation and socialisation with diverse publics, we undertook a critical literature review. The review covers applications of simulation modelling across systems science, environmental, biological, health systems, as well as the social sciences where critical perspectives and discussions on public engagements with modelling have been most developed. The review explores (1) How the role and value of models is conceptualised in relation to knowledge sharing, community participation and public engagement (2) The epistemic, theoretical and methodological traditions and assumptions that underpin these approaches to modelling practice. We go on to consider how critical reflections on ‘value’ from different traditions might inform the development of research on modelling practice and the judicious use of models going forward.

Methods

As we were interested in exploring underlying assumptions within the empirical stories of the literature [27], we adopted hermeneutics as the philosophy and methodology for conducting the review [28]. Many reviews of participatory modelling focus on the question of ‘what works?’ or synthesise learnings from a particular field (i.e., health policy or environmental science). In our critical literature review, the emphasis is on induction, interpretation and critique [29], with the goal of deepening understanding of how ‘value’ is understood in different approaches to participatory modelling. Conventional systematic review methodology—testing theories around ‘what works’ through exhaustive searches, or determining averages through quantitative data—would be inappropriate for our aims [29]. Although numerous texts in the simulation modelling literature draw on insights derived from quantitative models, our critical analysis centres on unpacking the implicit assumptions conveyed through the narrative descriptions of the purpose of models and why they are valuable.

Greenhalgh and colleagues’ [30] emphasis on mapping storylines of different research traditions influenced our approach. However, we decided it was more appropriate and informative to pursue a “dynamic, recursive and reflexive” synthesis [27] of how the modelling literature conceptualised the value of models in relation to knowledge sharing and public engagement, rather than a highly systematic, meta-narrative analysis. Drawing on our collective expertise in participatory modelling and social studies of science, we prioritised texts that were relevant to highlight distinctive contributions and major themes, instead of presenting an exhaustive overview of the entire body of literature and creating strict boundaries around inclusion and exclusion criteria.

Through an iterative approach, a search strategy began with a preliminary overview of a broad range of modelling literature across different sectors to help identify key search terms and refine our research questions. After trialling searches in a number of electronic databases including Google Scholar, Web of Science, PubMed, Science Direct, and Proquest we found that Scopus enabled us to locate the most relevant papers in relation to our areas of focus. The review was conducted in two phases between August and December 2021. Sources included in the search were scholarly documents (peer reviewed journals, conference proceedings and PhD dissertations) published in English between 2006 and 2021. Initially review search terms included ‘systems model’ OR ‘systems modeling’ OR ‘systems modelling’ OR ‘systems models’ AND engagement OR engaging OR ‘policy maker’ OR ‘policy makers’ OR ‘policy makes’ yielding 890 documents. Based on themes and terms identified in the initial search, a second search was conducted including the following search terms: ‘group model building’ OR ‘participatory dynamic simulation model’ OR ‘participatory dynamic simulation modeling’ OR ‘participatory model’ OR ‘participatory modeling’ yielding 1235 documents. VL screened and sorted records, initially by reviewing titles and abstracts and then by reviewing full papers. Foundational papers in the literature were supplemented with a focus on more recent publications between 2017 and 2021, to capture current thinking and innovations around participatory methods. This preliminary search was aided by citation tracking and snowballing, as well as recommendations from LF and JO, both with extensive experience in participatory modelling.

Reflecting on these searches, the authors iteratively tailored the search strategy toward literature with a specific emphasis on:

  • Models aimed at forecasting future trajectories and engagement with policy questions and decisions

  • Transparency, socialisation and communication of models

  • Novel methods for integrating data/knowledge and bringing diverse groups of people together around the development and use of models

This search led to a total of 53 texts being included for review, which are summarised below in Table 1.

Table 1 Summary of texts included in the review

The texts were read closely for key concepts describing the purpose of models or engagements with stakeholders. We focused on instances in the texts where the logic of value was invoked and where value was described in relation to the development and communication of simulation models. The epistemologies, methods and focus of the texts are summarised in Table 1. As the table indicates, these papers comprised a mixture of case studies, review articles and theoretical papers. In keeping with a hermeneutic approach, the analysis involved a dialectical tacking back and forth between descriptive detail and the broader themes in the literature [28]. This allowed for new ways of linking concepts and synthesising theories. Themes were identified by VL and presented to LF and JO for feedback where we addressed contradictions and refined the descriptive labels and statements. This approach conforms to Standards for Reporting Qualitative Research [31] (Additional file 1).

Results

Results are presented as four over-arching narrative conceptions of the role and value of models (see Fig. 1). We identified cross-cutting themes around approaches to participation, knowledge sharing and public engagement. These narrative conceptions are not mutually exclusive thematic domains, but are in some cases overlapping and in conversation with one another.

Fig. 1
figure 1

Summary of narrative conceptions

Narrative conception #1: models simulate and help solve complex problems

Models in this research tradition are often referred to as ‘decision-support’ tools that seek to provide predictive planning frameworks to inform and improve policy decisions [32]. In this conceptualisation, models are designed to help policymakers gain better understandings of system behaviour and the multiple challenges and goals in a complex system [33]. The kinds of decisions that models in this tradition are supportive of include service planning, investment strategies, and policy reform. Models act to facilitate sophisticated and intricate understandings of a problem and how it will respond to alternative courses of action, allowing decision-makers to identify more effective intervention strategies [34]. Through forecasting implications of particular investment strategies models have the potential to both optimise resource allocation and minimise unanticipated harmful consequences of decisions.

This conceptualisation of models often draws on the modelling methods of complex systems science (including system dynamics modelling), which examine interrelationships, feedback loops and the presence of equilibria in the system under investigation. The aim of such modelling is to move beyond simple ‘rational’ choice decisions, based on straightforward inputs and outputs, in order to identify opportunities for intervention or action that might have been overlooked by traditional ‘evidence-based’ or cost/benefit approaches. The inclusion of local and practice-based knowledge alongside research data is considered to challenge accepted conventions of evidence ‘hierarchies’ in health [35]. Further, concepts of evidence and data are not understood in terms of isolated individual studies, but rather as a ‘complex puzzle’ which takes account of the additive effects of combining different intervention approaches over time [36]. Modelling can be undertaken at different levels of abstraction (i.e., micro-, meso-, macro-) to arrive at a testable hypothesis and explanation for why a system behaves the way it does [37]. Systems models therefore allow decision-makers to consider how competing priorities and agendas across, for example, different levels of government may undermine the impact of desired policy outcomes [38]. Agent Based Modelling in particular is favoured by those interested in the activities and/or heterogeneity of individual agents (such as people, equipment, services or vehicles) [39]. These models strive toward a depiction of detailed reality through the inclusion of individual behaviour, social interactions and dynamics, and environmental variations [40].

Approaches to participation of stakeholders and public engagement in this tradition are varied. In the participatory development of Agent Based Models, experts are described as being able to easily relate to the depiction of the model at an individual level as they can draw on experiential knowledge of patient case histories and deploy professional judgement to validate and help calibrate the model [41, 42]. This and other approaches are underpinned by an ideal of ‘transparency,’ which eschews the ‘black box’ model in favour of making the model structure, assumptions, and limitations available for outside appraisal to assist in interpreting model results [43]. Others draw on the concept of ‘knowledge mobilisation’ aiming to include ‘end users’ (i.e. decision-makers) of the model in the model development process [32, 44]. Stakeholders are typically policymakers, researchers and practice experts (e.g. clinicians) whose participation is said to increase trust in the model outputs [45], ensure consideration of practical issues with policy implementation or evaluation [46] as well as the translatability of the model findings [35].

Benefits of participation in this tradition include opportunities for capacity building in terms of knowledge and expertise in systems science and modelling methods [37, 47]. This expertise is theorised to improve literacy in interpreting modelling results and empower end-users make better appraisals of future models they encounter in their policy practice [37]. Participatory modelling can be used to aid planning and policy choices in terms of organisational management, to enhance learning and promote shared understanding of complex problems among different actors in the system [33, 48, 49].

Questions surrounding language, accessibility, cultural adaptation, and power dynamics in the modelling process and translation of model results are rarely explored in depth in this approach. Some have pointed out that recruiting a diversity of participants and stakeholders in this tradition tends to favour experts such as health practitioners or policymakers. By contrast, small numbers of community participants are chosen to represent a variety of personal experiences within the system as a whole [50].

Narrative conception #2: models as tools for community engagement

Drawing on principles of community-based participatory research, community development and equity, the emphasis in this conceptualisation is on modelling practice as a ‘powerful learning process’ [23]. Research in this tradition is in keeping with efforts to engage in dialogues and enhance communication between experts and lay publics [51]. Drawing on trends in citizen science, it responds to demands by citizens to be included as stakeholders in planning decisions that affect them [52].

Central to this narrative construct is a focus on harnessing the capacity of multiple publics to engage in coordinated efforts to address the problem/s at hand. Goals of social learning and building ‘social capital’ among participants take priority over collectively producing model results [53]. This orientation is often underscored by a logic that the ‘problems’ tackled through modelling disproportionately affect marginalised groups, therefore meaningful participation of such groups is critical to the success of modelling efforts. Community-based participants are treated as experts of local and historical context that bring crucial knowledge, skills and labour to the model building [52, 53]. Such groups may be identified by virtue of their experience—for example people with personal experience of substance abuse and domestic violence [50], or by virtue of their relationship to place and experience of historic initiatives (e.g. First Nations people). Concepts of evidence and data are construed as contingent in this tradition and there is a recognition that vast diversity of participant experiences exists in relation to modelling issues of interest [50]. Knowledge produced through modelling exercises is therefore fluid and mediated by an “interactive and iterative” learning process [54].

Commitment to engaging a broad range of community participants is not only practical, but also ideological [55], encouraging modelling practice to strive toward engaging participants in ways that empower them to take ownership of the products of the modelling [52]. Power dynamics—and who may be left out of participatory modelling—are of key concern within this research tradition, and authors emphasise how powerful stakeholders can encourage, or conversely prevent, other actors from fully participating [53]. Attending to unequal power dynamics is seen as essential for encouraging engagement of marginalised community members, and methodological approaches carefully consider how to meaningfully engage all participants and ‘amplify’ the voices of previously marginalised stakeholders in the modelling process [54, 56]. Many approaches to modelling in this tradition experiment with forms of knowledge production that go beyond scientific publication, often using arts and performance-based mediums such as storylines, blogs, and alternative forms of visualisation [53, 54]. In addition to experimenting with communication mediums, many authors also draw attention to the issue that some marginalised participant groups may lack the necessary resources (time, financial, psychological, transportation) to take part in projects requiring intensive commitment [50], calling for compensation or remuneration to remedy this. Many authors stress, however, that communication of model results should not be confined to the model builders and decision makers. An enduring question in this tradition is how stakeholders who participated in the model building can take their new understandings and communicate them to others who were not involved in the model construction [57].

Narrative conception #3: models as tools for consensus building

This conceptualisation of models builds on and overlaps with narrative conceptions #1 and #2, but is distinguished by its emphasis on the value of modelling processes for facilitating communication and providing opportunities for negotiating conflicts and building consensus among stakeholder groups. Much of the research on participatory modelling practice acknowledges the value of modelling practice for promoting intellectual exchange and advancing contentious debates. Models are described as platforms for strengthening relationships between different knowledge communities (e.g. academics and policymakers [58] or historically antagonistic institutions [59]). Some texts take this concept further, delving deeply into the role of models and the specific qualities inherent in modelling that lends this practice to consensus building.

Approaches to participation in this tradition are centred on the negotiation of conflict, or reconciling different viewpoints among participants. A major impetus for modelling in this conception is the development of a ‘shared language’ among participants [60]. This is important because modelling is often used to address complex and messy problems and participants bring with them rich, but often partial, prior knowledge of the situation [61]. Participants developing a ‘shared understanding’ of the issue being modelled—be it policy implementation or the overview of a system—is seen as an inherently desirable outcome of the modelling process [46].

Issues of bias and values are also of central concern. Tools and methodologies are deployed to elicit unconscious bias [62] and recognise beliefs and values [23]. Voinov and Gaddis [59] have been particularly influential in arguing that modellers ought to acknowledge and embrace how their work is driven by ‘values’. Others have drawn on psychological theories of persuasion and mental model change to explore how model building processes influence beliefs, attitudes and perceptions of norms [61]. This research identifies “counter-intuitive insights”—where model simulations or views of other participants run counter to originally held positions on an issue—as a key strength in shifting mental models [63]. Mental modelling techniques and tools have been popularised by the work of Gray and colleagues [64] to facilitate engagement and communication between participants [65] and track changes in mental models during model building [66].

‘Boundary object’ theory is often invoked in this research tradition. Many of these texts draw on Star and Greisemer’s [67] theoretical synthesis of Boundary Objects in Science and Technology Studies. Models are depicted as ‘adaptable’ objects whose flexible representation enables people with different backgrounds and expertise to communicate more effectively and engage in co-ordinated activities [65]. Texts emphasise and analyse models’ ability to facilitate communication on contentious issues through developing a shared language and a shared understanding between different groups [68]. From this perspective, what matters is less a model’s precision than its capacity to provide a platform that can mitigate conflict and friction between different groups.

Narrative conception #4: models as volatile technologies that generate social effects

This conceptualisation of models is inspired by efforts from the social sciences to critique dominant approaches to modelling. In contrast to the other three narrative conceptions, this approach is less concerned with developing ‘best practice’ approaches (themselves premised on another kind of model) than with rethinking taken-for-granted assumptions within modelling.

Science and Technology studies, and the work of Bruno Latour [69] has a strong influence on studies in this tradition. In this sense, models are not simply neutral technologies that describe or represent activities and issues of policy interest, they are “generative” [26] and “transformational” [70]. Models—like other products of science in Latour’s theorisation—are considered for their ability to generate social effects. They are dynamic entities or “assemblages,” clusters of actors, practices, discourses and material objects which circulate across social fields and intersect with policy practice. Models and modelled evidence are multiple in their meanings and are perpetually emerging. This conceptualisation of models may also be framed within broader critical approaches to anticipatory governance [71], New Public Management [12] and practices of ‘projection’ or ‘simulation’ [72, 73].

Callon and Law’s [74] conceptual reconfiguration of the concept of calculation to include notions of judgement and passion has spawned a number of texts exploring the ‘affective’ and emotional components of modelling practice [72]. The affective qualities of models, and their ability to generate and be shaped by emotional responses, have led some authors to note the “virality” of models and their virus-like capacity for contagion [12]. Some suggest that the ‘seductive simulations’ tempt modellers to ‘oversell’ their products as ‘truth machines’ [73]. Such studies emphasise the messiness and uncertainties involved in modelling practice.

Knowledge and ‘evidence’ are characterised in this tradition as tentative, evolving, contingent and emergent, highlighting the limits of conceptualisations of models based on ‘evidence-based’ approaches. Some, such as Pawson [1], draw on complexity science and evaluation theory to question the evidence that informs health modelling. This critique further extends to dominant conceptions of ‘interventions’ as the evidence upon which model assumptions are based, stressing “the impossibility, of trying to capture a complex, self-transforming process as a model ‘parameter’” [1]. Texts highlight the dangers of building models based on evidence from interventions which are regarded as ‘fixed’ both in terms of implementation and effect. The focus here invokes questions around the role of “context” in modelling practice. Whilst acknowledging that some models do make generalised assumptions while paying insufficient attention to local conditions, this perspectival view (unlike Narrative conception #2) does not pursue a line of reasoning whereby better models depend on increasingly attuning them to local conditions [2, 75]. Models, after all, will always be “simplifications based on abstraction” [2]. Instead, both interventions and models are thought of as fluid and continually adapting as elements of ‘evidence-making interventions’ [76]. A model may function as an ‘interested amateur,’ adopting the role of an outsider that aids the interaction of policy experts, freeing up participants to openly voice criticisms and confront sensitive issues [22]. In this way, a measure of a good model is not simply how well it incorporates local data within its calculations, but how well models are mobilised as part of an adaptive science in which they can be treated as a pathway to dialogue, and deeper appreciation of interventions and their effects.

Discussion

Other reviews of participatory modelling have explored the value of stakeholder engagement in a particular field (e.g. social and environmental sciences or health). This review, however, examines the narrative conceptualisations of the value of models across a range of fields that engage with participatory modelling approaches. Examining these conceptions side-by-side allows us to consider how learnings from different fields might inform one another. In each of these narrative conceptions, value is constituted through the process of building models with stakeholders. In narrative conception 1, models employ quantitative methods but are embedded in social processes that are pivotal to their success or failure. The process of participatory modelling enhances the quality of model results by integrating new forms of knowledge and building trust with decision-makers. In other conceptions, the process is less about refining model accuracy and more about empowering marginalised stakeholders (narrative conception 2) or negotiating friction and developing a shared language among model participants from different knowledge communities (narrative conception 3). In narrative conception 4, the process is characterised as less goal-oriented, instead describing the unpredictability, multiplicity and self-organisational properties of model building with stakeholders.

An important point of difference in these conceptualisations lies in whether models are considered to be analytic tools to support decision-making, or whether modelling is itself a form of intervention capable of generating social change. Different framings of the role and value of modelling have implications for the evaluation of modelling and understanding the effects of a modelling project. Evaluations of participatory modelling are often underdeveloped and there is a need to develop clearer appreciation of the critical elements entailed in the purpose and associated value of modelling tools [77, 78]. If models are framed as tools for supporting decision-making (narrative conception 1), evaluation would focus on investigating the experience of participant stakeholders including their understanding and perception of the problem, or their trust in the legitimacy of model results [60, 79]. If models are deployed as aides in collaborative problem-solving, or as conflict resolution tools (narrative conception 3) evaluation might focus on how model representations are received among the group, along with their capacity to de-personalise conflicts and allow participants to negotiate less threatening paths to develop shared language [63] and shape consensus [68]. If models are avenues for building social capital or empowering traditionally marginalised communities (narrative conception #2), then evaluation of conflict resolution, or assessments of the development of new knowledge among participants must also attune to attenuating power dynamics and the foregrounding of marginalised or non-scientific forms of knowledge. Importantly, the goal of empowerment ought not to focus on individual participants, but facilitating circumstances so that publics are capable of producing new knowledge [25]. In narrative conception #4, modelling is framed as a starting point or a trigger for generating change processes. This has ramifications for evaluation in that the role of the model can be considered at project initiation, however flexibility is needed as the role may change over time and with different audiences and contexts.

These narrative conceptions may alternatively be used as a package to aid modellers in communicating the role and value of modelling tools to stakeholders including policy experts, funders, media or community representatives. Drawing on all four narrative conceptions of value can help illustrate how models serve multiple purposes and broaden understandings of their role in change processes. If the value of models is too narrowly defined, if their predictive value is overstated, or based on notions of infallible evidence to predict outcomes of policy decisions, there is a risk that the impact of modelling tools may be overlooked.

Conclusions

Crises such as the COVID pandemic and environmental catastrophes offer opportunities to transform the dynamics of knowledge sharing and engagement with models across diverse community and policymaking contexts. Bringing together the different narrative conceptions highlights the multiple ways that simulation modelling can be of value for public policy engagement. Linking ideas in these narrative conceptions—for example, considering how model representations can sit alongside and speak to non-scientific ways of knowing in informing policy decisions—offers new possibilities for harnessing the value of modelling. It may also prompt critical reflection on taken-for-granted ideas such as the value of consensus. The development of consensus or shared language must be weighed against opportunities for building social capital or encouraging participants to better articulate and communicate alternative perspectives and ways of understanding issues. In this sense, modelling may be considered valuable for its potential to generate and legitimate multiplicity.

At the same time, if the value of modelling is also located in the process of model-building or collaborative interaction with models, more work is needed to understand the evolutionary dynamics and systemic transformations that are potentially triggered by modelling beyond the participants directly engaged in model development. Future research tracking the value of models and their ripple effects as they circulate in wider public and policy spheres is needed. These different narrative conceptions may be used as a starting point for understanding the conditions under which simulation modelling can generate proactive pathways for change, or alternatively risk perpetuating the status quo.

Availability of data and materials

Not applicable.

References

  1. Pawson R. The coronavirus response: boxed in by models. Evaluation. 2021;27(2):149–67.

    Article  Google Scholar 

  2. Kaehne A. From abstract to ideal—the limits of models: a reply to Pawson’s ‘boxed in by models.’ Evaluation. 2021. https://doi.org/10.1177/13563890211007505.

    Article  Google Scholar 

  3. Tufekci Z. Don’t believe the COVID-19 models: that’s not what they’re for. Washington: The Atlantic; 2020.

    Google Scholar 

  4. Barton CM, Alberti M, Ames D, Atkinson J-A, Bales J, Burke E, et al. Call for transparency of COVID-19 models. Science. 2020;368(6490):482–3.

    Article  PubMed  Google Scholar 

  5. Jalali MS, DiGennaro C, Sridhar D. Transparency assessment of COVID-19 models. Lancet Glob Health. 2020;8(12):e1459–60.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Saltelli A, Bammer G, Bruno I, Charters E, Di Fiore M, Didier E, et al. Five ways to ensure that models serve society: a manifesto. Nature. 2020;582(7813):482–4.

    Article  CAS  PubMed  Google Scholar 

  7. Squazzoni F, Polhill JG, Edmonds B, Ahrweiler P, Antosz P, Scholz G, et al. Computational models that matter during a global pandemic outbreak: a call to action. J Artif Soc Soc Simul. 2020. https://doi.org/10.18564/jasss.4298.

    Article  Google Scholar 

  8. Amsen E. How ‘Flatten the Curve’ is on its way to become a historic disease image. Jersey City: Forbes; 2020.

    Google Scholar 

  9. Rhodes T, Lancaster K, Rosengarten M. A model society: maths, models and expertise in viral outbreaks. Crit Public Health. 2020;30(3):253–6.

    Article  Google Scholar 

  10. Karvelas P. Policy makers, public able to compare COVID-19 modelling on new site. 2021.

  11. Anderson W. The model crisis, or how to have critical promiscuity in the time of Covid-19. Soc Stud Sci. 2021;51(2):167–88.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Østebø MT. Village gone viral: understanding the spread of policy models in a digital age. Redwood City: Stanford University Press; 2021.

    Book  Google Scholar 

  13. Sterman J. Business dynamics. New York: McGraw-Hill, Inc.; 2000.

    Google Scholar 

  14. Sterman JD. Learning from evidence in a complex world. Am J Public Health. 2006;96(3):505–14.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Mobus GE. Principles of systems science. In: Mobus GE, editor. Systems science: theory, analysis, modeling, and design. Berlin: Springer; 2022. p. 41–87.

    Chapter  Google Scholar 

  16. Homer JB, Hirsch GB. System dynamics modeling for public health: background and opportunities. Am J Public Health. 2006;96(3):452–8.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Marshall DA, Burgos-Liz L, IJzerman MJ, Crown W, Padula WV, Wong PK, et al. Selecting a dynamic simulation modeling method for health care delivery research—Part 2: report of the ISPOR Dynamic Simulation Modeling Emerging Good Practices Task Force. Val Health. 2015;18(2):147–60.

    Article  Google Scholar 

  18. Marshall DA, Burgos-Liz L, IJzerman MJ, Osgood ND, Padula WV, Higashi MK, et al. Applying dynamic simulation modeling methods in health care delivery research—the SIMULATE checklist: report of the ISPOR simulation modeling emerging good practices task force. Val Health. 2015;18(1):5–16.

    Article  Google Scholar 

  19. Basco-Carrera L, Warren A, van Beek E, Jonoski A, Giardino A. Collaborative modelling or participatory modelling? A framework for water resources management. Environ Model Softw. 2017;91:95–110.

    Article  Google Scholar 

  20. Gilbert N, Ahrweiler P, Barbrook-Johnson P, Narasimhan KP, Wilkinson H. Computational modelling of public policy: reflections on practice. J Artif Soc Soc Simul. 2018;21(1):14.

    Article  Google Scholar 

  21. Barbrook-Johnson P, Badham J, Gilbert N. Uses of agent-based modeling for health communication: the TELL ME case study. Health Commun. 2017;32(8):939–44.

    Article  PubMed  Google Scholar 

  22. Johnson P. Agent-based models as “interested amateurs.” Land. 2015;4(2):281–99.

    Article  Google Scholar 

  23. Jordan R, Gray S, Zellner M, Glynn PD, Voinov A, Hedelin B, et al. Twelve questions for the participatory modeling community. Earth’s Future. 2018;6(8):1046–57.

    Article  Google Scholar 

  24. Elsawah S, Filatova T, Jakeman AJ, Kettner AJ, Zellner ML, Athanasiadis IN, et al. Eight grand challenges in socio-environmental systems modeling. Soc Environ Syst Model. 2020;2:16226.

    Google Scholar 

  25. Adams S, Rhodes T, Lancaster K. New directions for participatory modelling in health: redistributing expertise in relation to localised matters of concern. Glob Public Health. 2021. https://doi.org/10.1080/17441692.2021.1998575.

    Article  PubMed  Google Scholar 

  26. Landström C, Whatmore SJ, Lane SN, Odoni NA, Ward N, Bradley S. Coproducing flood risk knowledge: redistributing expertise in critical ‘participatory modelling.’ Environ Plan A. 2011;43(7):1617–33.

    Article  Google Scholar 

  27. Dixon-Woods M, Cavers D, Agarwal S, Annandale E, Arthur A, Harvey J, et al. Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol. 2006;6(1):1–13.

    Article  Google Scholar 

  28. Boell SK, Cecez-Kecmanovic D. A hermeneutic approach for conducting literature reviews and literature searches. Commun Assoc Inf Syst. 2014;34(1):12.

    Google Scholar 

  29. Greenhalgh T, Thorne S, Malterud K. Time to challenge the spurious hierarchy of systematic over narrative reviews? Eur J Clin Invest. 2018. https://doi.org/10.1111/eci.12931.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O, Peacock R. Storylines of research in diffusion of innovation: a meta-narrative approach to systematic review. Soc Sci Med. 2005;61(2):417–30.

    Article  PubMed  Google Scholar 

  31. O’Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Acad Med. 2014;89(9):1245–51.

    Article  PubMed  Google Scholar 

  32. Atkinson J-A, Skinner A, Lawson K, Rosenberg S, Hickie IB. Bringing new tools, a regional focus, resource-sensitivity, local engagement and necessary discipline to mental health policy and planning. BMC Public Health. 2020;20:1–9.

    Article  Google Scholar 

  33. Ibrahim Shire M, Jun GT, Robinson S. Healthcare workers’ perspectives on participatory system dynamics modelling and simulation: designing safe and efficient hospital pharmacy dispensing systems together. Ergonomics. 2020;63(8):1044–56.

    Article  PubMed  Google Scholar 

  34. Gilbert N, Terna P. How to build and use agent-based models in social science. Mind Soc. 2000;1(1):57–72.

    Article  Google Scholar 

  35. O’Donnell E, Atkinson J-A, Freebairn L, Rychetnik L. Participatory simulation modelling to inform public health policy and practice: rethinking the evidence hierarchies. J Public Health Policy. 2017;38(2):203–15.

    Article  PubMed  Google Scholar 

  36. Atkinson J-A, Skinner A, Hackney S, Mason L, Heffernan M, Currier D, et al. Systems modelling and simulation to inform strategic decision making for suicide prevention in rural New South Wales (Australia). Aust N Z J Psychiatry. 2020;54(9):892–901.

    Article  PubMed  Google Scholar 

  37. Langellier BA, Kuhlberg JA, Ballard EA, Slesinski SC, Stankov I, Gouveia N, et al. Using community-based system dynamics modeling to understand the complex systems that influence health in cities: the SALURBAL study. Health Place. 2019;60:102215.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Occhipinti J-A, Skinner A, Carter S, Heath J, Lawson K, McGill K, et al. Federal and state cooperation necessary but not sufficient for effective regional mental health systems: insights from systems modelling and simulation. Sci Rep. 2021;11(1):1–12.

    Article  Google Scholar 

  39. Freebairn L. “Turning mirrors into windows”: a study of participatory dynamic simulation modelling to inform health policy decisions. Sydney: The University of Notre Dame Australia; 2019.

    Google Scholar 

  40. Matthews RB, Gilbert NG, Roach A, Polhill JG, Gotts NM. Agent-based land-use models: a review of applications. Landscape Ecol. 2007;22(10):1447–59.

    Article  Google Scholar 

  41. Osgood N. Frontiers in health modelling. In: El-Sayed AM, Galea S, editors. Systems science and population health. Oxford: Oxford University Press; 2017.

    Google Scholar 

  42. Freebairn L, Atkinson J-A, Osgood ND, Kelly PM, McDonnell G, Rychetnik L. Turning conceptual systems maps into dynamic simulation models: an Australian case study for diabetes in pregnancy. PLoS ONE. 2019;14(6):e0218875.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  43. Brugnach M, Tagg A, Keil F, de Lange WJ. Uncertainty matters: computer models at the science—policy interface. Water Resour Manage. 2007;21(7):1075–90.

    Article  Google Scholar 

  44. Freebairn L, Rychetnik L, Atkinson J-A, Kelly P, McDonnell G, Roberts N, et al. Knowledge mobilisation for policy development: implementing systems approaches through participatory dynamic simulation modelling. Health Res Policy Syst. 2017;15(1):1–12.

    Article  Google Scholar 

  45. Freebairn L, Atkinson J-A, Kelly PM, McDonnell G, Rychetnik L. Decision makers’ experience of participatory dynamic simulation modelling: methods for public health policy. BMC Med Inform Decis Mak. 2018;18(1):1–14.

    Article  Google Scholar 

  46. de Oliveira Morais LM, Kuhlberg J, Ballard E, Indvik K, Rocha SC, Sales DM, et al. Promoting knowledge to policy translation for urban health using community-based system dynamics in Brazil. Health Res Policy Syst. 2021;19(1):1–15.

    Google Scholar 

  47. Siokou C, Morgan R, Shiell A. Group model building: a participatory approach to understanding and acting on systems. Public Health Res Pract. 2014;25(1):e2511404.

    Article  PubMed  Google Scholar 

  48. Hosseinichimeh N, Kim H, Ebrahimvandi A, Iams J, Andersen D. Using a stakeholder analysis to improve systems modelling of health issues: the impact of progesterone therapy on infant mortality in Ohio. Syst Res Behav Sci. 2019;36(4):476–93.

    Article  Google Scholar 

  49. Muttalib F, Ballard E, Langton J, Malone S, Fonseca Y, Hansmann A, et al. Application of systems dynamics and group model building to identify barriers and facilitators to acute care delivery in a resource limited setting. BMC Health Serv Res. 2021;21(1):1–10.

    Article  Google Scholar 

  50. Deutsch AR, Lustfield R, Jalali MS. Community-based system dynamics modelling of stigmatized public health issues: increasing diverse representation of individuals with personal experiences. Syst Res Behav Sci. 2021. https://doi.org/10.2139/ssrn.3573207.

    Article  PubMed  PubMed Central  Google Scholar 

  51. Popay J, Williams G, Thomas C, Gatrell T. Theorising inequalities in health: the place of lay knowledge. Sociol Health Illn. 1998;20(5):619–44.

    Article  Google Scholar 

  52. Voinov A, Kolagani N, McCall MK, Glynn PD, Kragt ME, Ostermann FO, et al. Modelling with stakeholders—next generation. Environ Model Softw. 2016;77:196–220.

    Article  Google Scholar 

  53. Sterling EJ, Zellner M, Jenni KE, Leong K, Glynn PD, BenDor TK, et al. Try, try again: lessons learned from success and failure in participatory modeling. Elementa. 2019;7:9.

    Google Scholar 

  54. Williams DS. Enhancing autonomy for climate change adaptation using participatory modeling. Weather Clim Soc. 2020;12(4):667–78.

    Article  Google Scholar 

  55. Van Bruggen A, Nikolic I, Kwakkel J. Modeling with stakeholders for transformative change. Sustainability. 2019;11(3):825.

    Article  Google Scholar 

  56. Bou Nassar JA, Malard JJ, Adamowski JF, Ramírez Ramírez M, Medema W, Tuy H. Multi-level storylines for participatory modeling–involving marginalized communities in Tz’olöj Ya’, Mayan Guatemala. Hydrol Earth Syst Sci. 2021;25(3):1283–306.

    Article  Google Scholar 

  57. Smetschka B, Gaube V. Co-creating formalized models: participatory modelling as method and process in transdisciplinary research and its impact potentials. Environ Sci Policy. 2020;103:41–9.

    Article  Google Scholar 

  58. Atkinson J-A, Knowles D, Wiggers J, Livingston M, Room R, Prodan A, et al. Harnessing advances in computer simulation to inform policy and planning to reduce alcohol-related harms. Int J Public Health. 2018;63(4):537–46.

    Article  PubMed  Google Scholar 

  59. Voinov A, Gaddis EB. Values in participatory modeling: theory and practice. In: Gray S, Paolisso M, Jordan R, Gray S, editors. Environmental modeling with stakeholders. Berlin: Springer; 2017. p. 47–63.

    Chapter  Google Scholar 

  60. Falconi SM, Palmer RN. An interdisciplinary framework for participatory modeling design and evaluation—what makes models effective participatory decision tools? Water Resour Res. 2017;53(2):1625–45.

    Article  Google Scholar 

  61. Rouwette EA, Korzilius H, Vennix JA, Jacobs E. Modeling as persuasion: the impact of group model building on attitudes and behavior. Syst Dyn Rev. 2011;27(1):1–21.

    Article  Google Scholar 

  62. Frerichs L, Lich KH, Funchess M, Burrell M, Cerulli C, Bedell P, et al. Applying critical race theory to group model building methods to address community violence. Prog Commun Health Partnersh. 2016;10(3):443.

    Article  Google Scholar 

  63. Rouwette E, Bleijenbergh I, Vennix J. Group model-building to support public policy: addressing a conflicted situation in a problem neighbourhood. Syst Res Behav Sci. 2016;33(1):64–78.

    Article  Google Scholar 

  64. Gray SA, Gray S, Cox LJ, Henly-Shepard S. Mental modeler: a fuzzy-logic cognitive mapping modeling tool for adaptive environmental management. In: 2013 46th Hawaii International Conference on System Sciences; IEEE. 2013.

  65. Huang J, Hmelo-Silver CE, Jordan R, Gray S, Frensley T, Newman G, et al. Scientific discourse of citizen scientists: Models as a boundary object for collaborative problem solving. Comput Hum Behav. 2018;87:480–92.

    Article  Google Scholar 

  66. Singer A, Gray S, Sadler A, Olabisi LS, Metta K, Wallace R, et al. Translating community narratives into semi-quantitative models to understand the dynamics of socio-environmental crises. Environ Model Softw. 2017;97:46–55.

    Article  Google Scholar 

  67. Star SL, Griesemer JR. Institutional ecology, translations’ and boundary objects: amateurs and professionals in Berkeley’s Museum of Vertebrate Zoology, 1907–39. Soc Stud Sci. 1989;19(3):387–420.

    Article  Google Scholar 

  68. Black LJ, Andersen DF. Using visual representations as boundary objects to resolve conflict in collaborative model-building approaches. Syst Res Behav Sci. 2012;29(2):194–208.

    Article  Google Scholar 

  69. Latour B. Science in action: how to follow scientists and engineers through society. Cambridge: Harvard University Press; 1987.

    Google Scholar 

  70. Schubert C. Situating technological and societal futures: pragmatist engagements with computer simulations and social dynamics. Technol Soc. 2015;40:4–13.

    Article  Google Scholar 

  71. Aykut SC, Demortain D, Benbouzid B. The politics of anticipatory expertise: plurality and contestation of futures knowledge in governance â introduction to the special issue. Sci Technol Stud. 2019;32(4):2–12.

    Article  Google Scholar 

  72. Rhodes T, Lancaster K. Excitable models: Projections, targets, and the making of futures without disease. Sociol Health Illn. 2021. https://doi.org/10.1111/1467-9566.13263.

    Article  PubMed  PubMed Central  Google Scholar 

  73. Lahsen M. Seductive simulations? Uncertainty distribution around climate models. Soc Stud Sci. 2005;35(6):895–922.

    Article  Google Scholar 

  74. Callon M, Law J. On qualculation, agency, and otherness. Environ Plan D. 2005;23(5):717–33.

    Article  Google Scholar 

  75. Rhodes T, Lancaster K, Lees S, Parker M. Modelling the pandemic: attuning models to their contexts. BMJ Glob Health. 2020;5(6):e002914.

    Article  PubMed  PubMed Central  Google Scholar 

  76. Rhodes T, Lancaster K. Evidence-making interventions in health: a conceptual framing. Soc Sci Med. 2019;238:112488.

    Article  PubMed  Google Scholar 

  77. Lee GY, Hickie IB, Occhipinti J-A, Song YJC, Camacho S, Skinner A, et al. Participatory systems modelling for youth mental health: an evaluation study applying a comprehensive multi-scale framework. Int J Environ Res Public Health. 2022;19(7):4015.

    Article  PubMed  PubMed Central  Google Scholar 

  78. Lee GY, Hickie IB, Occhipinti J-A, Song YJC, Skinner A, Camacho S, et al. Presenting a comprehensive multi-scale evaluation framework for participatory modelling programs: a scoping review. PLoS ONE. 2022;17(4):e0266125.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  79. Atkinson J-A, Wells R, Page A, Dominello A, Haines M, Wilson A. Applications of system dynamics modelling to support health policy. Public Health Res Pract. 2015;25(3):e2531531.

    Article  PubMed  Google Scholar 

  80. Rhodes T, Lancaster K, Rosengarten M. A model society: maths, models and expertise in viral outbreaks. Crit Public Health. 2020;30:253–6.

    Article  Google Scholar 

  81. Rhodes T, Lancaster K. Mathematical models as public troubles in COVID-19 infection control: following the numbers. Health Sociol Rev. 2020;29(2):177–94.

    Article  PubMed  Google Scholar 

  82. Saltelli A, Bammer G, Bruno I, Charters E, Di Fiore M, Didier E, et al. Five ways to ensure that models serve society: a manifesto. Nature. 2020. https://doi.org/10.1038/d41586-020-01812-9.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

This research has been supported by The Australian Prevention Partnership Centre, funded by the NHMRC, the Australian Government Department of Health, ACT Health, Cancer Council Australia, NSW Ministry of Health, Wellbeing SA, Tasmanian Department of Health, and VicHealth. It is administered by the Sax Institute.

Author information

Authors and Affiliations

Authors

Contributions

Conceptualization, VL, LF and JO; methodology, VL; formal analysis, VL, LF and JO; writing—original draft preparation, VL; writing—review and editing, LF and JO; All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Victoria Loblay.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1

: Standards for reporting qualitative research.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Loblay, V., Freebairn, L. & Occhipinti, JA. Conceptualising the value of simulation modelling for public engagement with policy: a critical literature review. Health Res Policy Sys 21, 123 (2023). https://doi.org/10.1186/s12961-023-01069-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12961-023-01069-4

Keywords