Unpacking the Black Box: Demystifying Ecological Models Through Interactive Workshops and Hands-On Learning

Introduction

Numerical models are powerful tools for informing decision-makers of the consequences of alternative management actions (Sierhuis and Selvin, 1996; Gregory and Keeney, 2002; Hurrell et al., 2013). Yet, models are commonly viewed as mysterious black boxes, which leads to mistrust, misinterpretation and/or misapplication of model results. When used in the context of complex management decisions with many partners, environmental and ecological modeling often benefits from approaches that emphasize transparency, increase user input during development, and clearly communicate model assumptions and limitations (van den Belt, 2004; Voinov and Bousquet, 2010). There are many types of stakeholder-based model development processes that overlap in approach and emphasis in collaboration: Model Prototyping, Participatory Modeling, Group Model Building, Companion Modeling and Mediated Model Development (See reviews by Voinov and Bousquet, 2010; Gray et al., 2017; Hall et al., 2019). These processes are generally well-suited to environmental management problems that are politically and publicly-sensitive, complex, and engage diverse audiences (van den Belt et al., 2006). We use the phrase mediated modeling processes to describe this family of techniques for building consensus among multiple partners and producing credible and defensible ecological models in a transparent way (van den Belt, 2004).

Mediated modeling processes can build a strong foundation of knowledge and trust from which multiple partners, disciplines, and stakeholders can co-develop tools tailored to their management challenge. However, many failures of systems modeling often arise from a lack of communication between a model development team, stakeholders, and decision-makers (Hall et al., 2014). At best, these cases can lead to confusion and miscommunication; at worst, they can lead to misinterpretation, mistrust, increased risk of ineffectual decisions, and/or litigation over the eventual decision (Augusiak et al., 2014), which exacerbates frustration associated with the use of models (Langsdale et al., 2013). Some examples of barriers that hinder effective communication associated with model development, application, and documentation include:

1. Participants may not self-identify as “modelers” due to a lack of familiarity with formal model development or a belief that conceptualization is separate from “modeling” (Hannon and Westervelt, 2012);

2. Participants often assume that modeling contributions are constrained by mathematical knowledge, which ostracize them due to underlying “mathematics anxiety” (Hembree, 1990);

3. Existing models may not align with management objectives (e.g., a fish model of a river and management focused on mussels; Carpenter and Turner, 2017), which can lead to misunderstanding between technical and non-technical audiences regarding scope and purpose of models;

4. Lack of familiarity with modeling processes or opaque development can lead to feelings of mistrust by stakeholders and collaborators or potential misapplication of models and results (Jakeman et al., 2006; Schmolke et al., 2010);

5. Ecosystem management problems tend to be interdisciplinary in nature, and stakeholders may be limited by discipline-specific expectations, biases, and epistemologies when collaborating outside of their background (Eigenbrode et al., 2007; Hall et al., 2019);

6. Participants often assume that ecological models must be validated, “accurate,” or match the complexity of ecosystems to be useful in environmental management. However, the model development community has long distinguished different uses of ecological models such as for “explanation” or “for prediction” (Hall et al., 2014); and

7. Models often suffer from the “black box” syndrome, where technical details or insufficient documentation limit the ability to communicate the theory, application, and relevance of the tool. In these cases, stakeholders can be justifiably wary of model results (Glaser and Bridges, 2007; Schmolke et al., 2010).

Repeatable mediated modeling techniques are needed to overcome these communication gaps, embed modelers and stakeholders in a co-development process, and illuminate the (perceived) opaqueness of model development. This paper presents a workshop-based mediated model development process that can overcome communication challenges by demystifying the modeling process for diverse audiences representing multiple disciplines, actors, and experiences. Many other misconceptions and fallacies have been identified with respect to the role of ecological models, such as the need for validation, logistical constrains of development, and best practices for mediation (Starfield, 1987; van den Belt, 2004; Voinov and Bousquet, 2010; Addison et al., 2013; Langsdale et al., 2013), but we focus on communication issues directly dealing with ecological model development in group settings with diverse technical and non-technical audiences.

This paper briefly describes the design and structure of a workshop-based model development process along with best practices for workshop planning and execution. The workshop approach was developed over the last decade through a variety of large-scale ecosystem restoration projects across the United States, and a case study is presented to demonstrate the application of this process. Lastly, we reframe the seven communication barriers above as “model mysteries” because workshop participants often found the topics initially puzzling, and we directly address how the workshop approach demystified these issues and empowered participants to embrace the utility of ecological model development.

Breaking Down Model Communication Barriers Through Interactive Workshops

Modern environmental issues are inherently complex and encompass multiple disciplines. Distilling this complexity into a refined model that stakeholders agree upon requires that transdisciplinary teams understand why a model was developed, the process for development, the overarching conceptual logic of a model, assumptions implicit in its construction, and key limitations and weaknesses. Clear communication, transparency, and stakeholder engagement are integral to achieving these goals across all stages of model development. Here, we propose a generic, repeatable structure for mediated model development, particularly for complex environmental management issues. This framework has been tested and refined in more than a dozen workshops over the course of a decade (Table 1) with varying purpose (e.g., instructional vs. project-driven needs), scope of model development (e.g., generic regional model vs. site-specific conceptual model), time investment (e.g., 1-h, 1-day, 3-days, multiple workshops), breadth of participants (e.g., agency team vs. interagency stakeholders), and ecosystem type (e.g., urban stream, large river, coastal marsh). The following sections present the workshop structure, best practices for workshop planning and execution, and a case study workshop focusing on large-scale ecosystem restoration in southern California.

Table 1. Case studies applying the overarching mediated modeling workshop approach described in this paper.

Workshop Structure and Content

Like other mediated modeling processes, our workshop approach focuses on interactive model building with diverse participants and follows common stages of ecological model development such as conceptualization, quantification, evaluation, and application (Grant and Swannack, 2008), but we have added content explicitly covering communication strategies for each phase of model development. We define our use of the term model, participant, and system in Box 1.

Box 1. Mediated modeling terminology as applied in this paper.

• Integrated model: Integrated modeling is the method that is developing to bring together diverse types of information, theories, and data originating from scientific areas that are different not just because they study different objects and systems, but because they are doing that in very different ways, using different languages, assumptions, scales, and techniques (Voinov and Fishwick, 2019).

• Model: Simplified representation of a complex system, containing both conceptual and mathematical components, which are composed of the critical characteristics necessary to address specific questions.

• Mediated modeling (used synonymously here with participatory modeling): “A practice of coordinated communication of actors' working understandings of how a particular system of interest functions and the organization of the communicated conceptualizations into a shared representation of “model.” The process combines collaborative learning, public participation, and systems dynamics modeling. The model (a product of the process) may take many forms and is at the discretion of the group driven by the purpose, the system, and the participants (Hall et al., 2019).

• Participant: A modeling stakeholder attending and contributing to a mediated modeling workshop. Herein, typically an actor with either technical expertise, organizational representation, or site-specific (i.e., local) knowledge to contribute.

• System: A defined set of objects that interact in space and time. Systems are organized collections of interrelated physical components characterized by a boundary and functional unity. Systems are subjective entities created by humans for specific purposes, generally to do work, answer questions, illustrate theory, or explain the natural world. The complexity of any system increases as the number of system components and connections among components increases. An important property of a system is that the interactions among components create emergent properties, specific to that system (Swannack, 2019).

Four elements, however, are prominent in this framework, which are critical to overcoming communication barriers in effective use of ecological models. First, a series of modules are used to step through model development, each of which includes presentation of material and facilitated group exercises. Second, we have found that an interactive workshop-based approach that creates a model from start-to-finish facilitates model understanding. Third, inherent in the mediated modeling process is the empowerment of participants' perspectives into model conceptualization, which increases participant's awareness of the problem, understanding of the system of interest and ownership of the model itself. Fourth, our approach focuses on integrating real-time, interactive quantification into the process, which further increases model transparency and facilitates model communication.

Over the course of the last decade, we have refined our workshops from a straight lecture-and-learn approach to an interactive suite of modules that incorporates real-time modeling, engaging participants in each stage of model development (Table 2). Modules are dedicated to each stage of traditional model development processes (Conceptualization, Quantification, Evaluation and Application) with additional modules focusing on the system of interest and model communication. Each module contains an informational session on a specific topic, then an interactive exercise where participants apply the information directly to model development. Rather than focus on describing each phase of model development, which have been extensively covered in other sources (e.g., Starfield, 1987; Grimm and Railsback, 2005, Grant and Swannack, 2008; Ford, 2010), the following sections focus on what we view as best practices for designing workshops to achieve consensus on model development and application in large multidisciplinary stakeholder groups.

Table 2. Mediated model development workshop content, learning objectives, and facilitated exercises.

The most important thing to consider when designing workshop modules is that all modules should be explicitly designed to describe the key aspects of how to develop a model and open the door to broad discussion in the context of the focal system. This encourages questions and discussion among the participants and facilitators, which increases the efficacy of communication of modeling technique. Presentations by the facilitators should be geared toward a diverse audience, usually with minimal technical jargon and include real world examples relevant to the system of interest. The basic format of the modules follows the principles of system dynamics (Forrester, 1961) and ecological model development (Starfield, 1987; Grant and Swannack, 2008; Ford, 2010). However, the first module presented should be an introduction to the system of interest presented from a participant and should not contain much, if any, modeling information. We have found that requesting presentations from local experts to kick off the workshop, rather than start the workshop with modules on ecological modeling, not only eases the participants into the modeling portion, but also empowers them to be part of the process. The content of each module should be developed to provide a common lexicon among attendees, describe concrete examples from other systems, and educate attendees on key aspects of model development. Each module has an interactive exercise that participants work through in breakout groups. The model development process is enriched with the introduction to the study system, group development of model objectives, and basics of model communication. Modules should seek to be as short as possible to accomplish these goals (certainly not exceeding 45 min per informational session). Figure 1 presents the efforts of a group working through model development from understanding the important components of a system of interest (e.g., system drivers) through quantification.

Figure 1. Typical outcomes from multiple mediated modeling workshops include preliminary conceptual models, which serve as a basis for intellectual organization and provide a foundation for numerical model development. This figure shows conceptual models iteratively developed at a series of mediated modeling workshops for East San Pedro Bay. (A) A preliminary conceptual model was developed by the technical team prior to the workshop, but the complexity overwhelmed project partners. (B) The workshop team constructed a model using scraps of paper. (C) The scrap-paper model was translated into a preliminary “lumped” model for rapid communication. (D) Qualitative response curves were developed on a whiteboard to guide numerical model development [example using seagrass habitat]. (E) Subset of variable quantification seagrass habitat using a rapid development, index-based modeling platform.

Break-out exercises should be as interactive as possible to engage participants who might not engage with the larger group. As such, group size is crucial to inclusion and engagement. Prior workshops (Table 1) indicate that break-out groups of five to seven participants are ideal to encourage communication in this setting. Break-out exercises also provide an opportunity to minimize the voice of dominant personalities that may be pushing the group toward a particular outcome. Our exercises are designed so that breakout groups develop their own model (this interactive content is described in more detail in the following section). In each exercise, the participants are assigned a task (e.g., develop a conceptual model, quantify a relationship) to complete as a group. Facilitators should serve as guides and assist when the group reaches trouble spots, but they should not offer solutions or develop the model themselves. Separate groups working on the same topic can lead to synergistic outcomes drawing from each group's best ideas.

Interactive modeling is necessarily iterative in nature. At the beginning of the first exercise it is important to explain that iteration is part of the process—that model evaluation is not a separate, distinct step, but that we evaluate models throughout their development. A group will reach initial decisions at each model development step, but they may want to change that decision and revise the model as necessary. This is a natural step in the modeling process and important to emphasize. Often participants want proceed in a linear fashion during model development, but model development does not occur in a clean, straight line. The overall workshop goal is to have a quantified, working model at the end of the workshop, but it is important to establish expectations with the participants that the model will need to be further revised through technical analyses, literature review, and follow up meetings and conference calls. At the end of the model development process, participants develop a deeper understanding of how to build models, and model-users trust that the model will produce an output they can be confident will assist them in their decision-making process.

Best Practices for Workshop Planning and Execution

While workshop structure and content are crucial aspects of model mediation, workshop execution can also lead to a successful or failed meeting. The following topics review a few (hard-won) lessons learned in workshop planning, logistics, and facilitation.

Planning

Mediated modeling workshops are commitments of time, resources, and energy by organizers and participants. Pre-workshop planning ensures efficient and effective use of resources, clarifies modeling objectives, and provides early and ongoing communication between facilitators and stakeholders, all of which ultimately are critical factors in the success of a model development project (Schmolke et al., 2010).

Organization of the workshop requires the identification of appropriate participants, both facilitators and stakeholders—getting the right people to the table is critical for success (Crawford et al., 2017). A local project champion familiar with key stakeholders can be crucial to building this participant list. The project champion need not be familiar with modeling, but should be able to assist with analysis of potential stakeholders to engage (Langsdale et al., 2013). The objective of the analysis should be to identify important stakeholders, not a comprehensive listing of all stakeholders. Participants should be chosen to represent the variety of interests invested in the system (e.g., federal, state, non-profit, local organizations, etc.) with a variety of backgrounds or expertise (e.g., engineering, ecology, hydrology, planning & policy, etc.), and should include local residents living in or adjacent to the system of interest. Participant diversity ensures that relevant knowledge and understanding of the system of interest is available during model development and will facilitate a deeper understanding.

A pre-workshop meeting or conference call is typically needed between facilitators and other organizers to manage expectations, review workshop objectives, discuss the system of interest and identify the participants. Workshop agendas should be developed with workshop champions, organizers, and facilitators to efficiently accomplish goals of each agenda item (Table 2 provides a scalable template). The facilitators need to engage with the workshop champion to also determine where potential controversies could occur, so the agenda can be tailored for the specific group. This champion can also serve as a key conduit to send invitations, assist with scoping, and avoid conflicts during the workshop (e.g., two conflicting personalities should be placed in different breakout groups).

Invitations should be sent at least 6 weeks in advance and should clearly articulate objectives for the workshop along with a basic agenda. Follow-up communication from workshop facilitators should provide brief details of the modeling process as well as any reading material specific to the project. Invitations should be written so that the participants can clearly see their individual role in generating the workshop's outcome(s).

During this first phase, it is important to provide participants with an outline of what to expect from the modeling workshop. Facilitators' roles are to provide instruction and guidance on model development, but workshop participants will actually develop the model. Facilitators should be modelers and not be the subject matter experts. Participants will provide background on the system of interest, the specific environmental issue being tackled. The role of the facilitators is to provide direction to the participants that guides them through important decision points in the model development. Stakeholders should provide the facilitators with a basic understanding of the targeted ecosystem. This ensures that the group discussions are focused and productive without extraneous or unimportant issues taking up valuable time.

Logistics

While not the purpose of the meeting, the logistics of a meeting can dramatically impact not only who is present and engagement of participants, but also their overall enjoyment of the event. Workshop length should scale with complexity of the problem being investigated. We have conducted week-long modeling courses, 3–4 h informational sessions and 2 day workshops. Two days seems to balance the difference between longer timelines, which lose key participants, belabor old grudges, and deplete resources, and shorter timelines, which can rush important discussions and truncate informational sessions. The 2 day time period increases the amount of time spent as a group in the workshop setting, but the trade-off to this increased amount of time is that participants are able carry forward a much greater level of understanding of each model development step. As they become more familiar with model development, they become more comfortable in their role as a modeler and the model development process becomes less mysterious.

The workshop location and setting are critical for success. In order to for the participants to be fully engaged, workshops should be held off-site from any one organization to break-down “turf” and keep attendees engaged (i.e., avoid people returning to their desk). Locating near the focal ecosystem can facilitate field trips and individual exploration of the system on breaks. Another reason to hold workshops off-site is to facilitate networking. Multi-stakeholder groups often do not network outside of the few hours their respective offices dedicate to the project. Finding time to discuss the project during breaks can facilitate relationship building. Similarly, meals together before, during, or after workshops can be crucial mechanisms for understanding competing interests, resolving disagreements over technical issues, and expanding relationships beyond technical and organizational boundaries.

Similarly, breaks serve as important networking opportunities to discuss both the task at hand, while also build longer lasting relationships and trust. Organizers can provide ample opportunity for these interactions and can also facilitate the interactions by providing snacks or drinks on-site.

While not required, field trips to study systems can be invaluable learning opportunities and mechanisms for testing key hypotheses. For instance, a workshop group may develop a simple conceptual model on Day-1, followed by a field trip on Day-2 and a refinement of the conceptual model on Day-3. The field trips can also provide opportunities for the group to hear from local or non-traditional expertise not represented at the workshop (e.g., a boat driver or park ranger).

Facilitation

Many workshops benefit from a facilitator external to the focal system, or a long-term stakeholder trusted by all participants. The workshop leader should be perceived as an unbiased, “honest broker” capable of navigating differences of opinion among attendees. Additionally, facilitators can be selected strategically to represent multiple disciplines, and should have experience in ecological model development. For example, our facilitating team typically consists of ecologists, engineers, geographers, and planners, all of whom have developed ecological models for ecosystem restoration projects. Facilitators must also balance the needs to hear multiple voices, resolve conflicts, and drive the team to produce tangible outcomes. We also find that facilitators with different stylistic approaches can reach participants in unique ways, and we explicitly encourage presenters to tailor content and presentation to their preferences (e.g., no standard slide templates).

The number of facilitators should scale to the breadth and scope of the workshop, and attendance should be limited by the participant to facilitator ratio; ideally, the number does not exceed seven to one for break-out groups. These require the facilitators' full attention and larger groups can be difficult to manage. The seven to one ratio provides the group with enough diversity among the participants while also allowing the facilitator to dedicate time with any participant who finds the content challenging.

Facilitators must manage workshop time to produce the team's desired outcomes. For instance, time keeping may include strict enforcement during some sessions, while allowing flexibility for key discussions to proceed to completion during others. The most successful workshops have detailed notes that provide clear documentation of the discussion and after-action steps to move forward. Notes can be taken by the facilitator (e.g., on a flip chart) or assigned to others while the facilitator manages the agenda. Notes that are displayed in real-time are particularly effective to avoid misinterpretation or miscommunication (e.g., flip charts, projected computer screen). For instance, an attendee may correct the intent of a misinterpreted remark. Notes should be summarized and sent to the attendees within a week of the workshop to keep it fresh in the minds of the participants, maintain credibility with attendees, and demonstrate the importance of their input and time.

Case Study: East San Pedro Bay Ecosystem Restoration Feasibility Study

The East San Pedro Bay Ecosystem Restoration Feasibility Study is a case study that serves as an example of how the mediated model development process can help to overcome challenges associated with the “black box” syndrome. The study required the use of an ecological model that could differentiate among various proposed ecosystem restoration plans with indicators of habitat quality and quantity. The system of interest was a highly-altered coastal bay located along the Southern California coastline within the City of Long Beach, CA (Figure 2). The bay originally contained various habitat types that were currently lacking (e.g., rocky reefs) or severely impacted from past human activities (e.g., seagrass beds, salt marshes, etc.). The groups represented during the model development process included federal (National Oceanic and Atmospheric Administration, U.S. Fish and Wildlife Service, U.S. Army Corps of Engineers) and state natural resources agencies (California Department of Fish and Wildlife, California State Water Resources Control Board, California Coastal Commission, Southern California Coastal Water Research Project), academia (Occidental College, California State University Long Beach), local government (City of Long Beach) and private consultants. Pertinent expertise represented by the workshop members included engineers, hydrologists, biologists, economists, program managers, and natural resource regulators. The model was developed over 3 years from two different workshops and multiple webinars and conference calls.

Figure 2. Map of East San Pedro Bay study area.

Early on, the workshop organizers, the U.S. Army Corps of Engineers, Los Angeles District, identified model objectives:

1. To be able to distinguish between proposed restoration alternatives, including the future without project alternative (no action scenario);

2. To be able to include input from other agencies and stakeholders once a conceptual model or model framework has been developed;

3. To be relevant to habitats of interest and at the ecosystem level, not just the species; and

4. To communicate benefits derived from a recommended restoration plan.

The first workshop resulted in an initial conceptual model of the entire bay that identified its critical drivers and stressors (Figure 1A). The initial workshop resulted in a “deep dive” into the drivers and stressors of the entire bay, asking what is needed to support a healthy, more natural ecosystem and what factors cause ecosystem degradation. Answering these questions revealed the level of understanding of what environmental problems could be addressed through ecosystem restoration and what things could not (e.g., operations of the Port of Long Beach), but the resulting model was too complex and overly complicated (Figure 1A). Intensive discussions designed to cull the extraneous details resulted in an intermediate conceptual model (Figure 1B). The first workshop set the stage for further model development and a follow-up workshop resulted in the identification of important habitat types, a refined conceptual model of the bay with specific habitats and their critical system level components (Figure 1C). Specific habitat types included rocky reef, kelp forest, seagrass, oyster reef, tidal salt marsh, soft bottom, and sandy islands. The second workshop also resulted in the initial quantification of the individual parameters for each habitat type (example of parameters Figure 1C). Initial response curves were developed in the first workshop (Figure 1D) and quantified in the second (Figure 1E). Successive webinars and conference calls followed the second workshop where the model was further refined based on filling in data gaps and soliciting input from other stakeholders not previously involved with the model (e.g., fresh eyes). The final model included six different habitat types that were identified as significant resources that should be the focus of restoration within the bay. Habitat types were described quantitatively by 102 difference equations, all created with direct input from workshop participants. Overall, the model was successful at satisfying its original objectives of helping to show differences between proposed ecosystem restoration alternatives, cultivated agency buy-in, relevant to habitats of interest and was able to communicate the potential environmental benefits from a proposed recommended restoration plan.

Model Mysteries Resolved Through Workshopping

The prior section presents a reproducible workshop approach developed from the authors' applied ecological modeling experiences (e.g., Swanack et al., 2009; McKay et al., 2013; Hall et al., 2014) and structured around basic tenets of ecological model development (Grant and Swannack, 2008). This framework has been tested and refined in more than a dozen workshops (Table 1). Across these workshops, barriers consistently emerged that hindered effective communication associated with model development, application, and documentation. Here, we frame these challenges as “model mysteries” because participants often found the topics initially puzzling, but the workshop demystified and empowered them to embrace the utility of ecological model development. The following section details each of the seven challenges and how interactive model development workshops helped overcome these issues. For each mystery, we embed observations from East San Pedro Bay, other workshops, and peer-reviewed literature.

1. Everyone is a modeler.

2. Mathematical talent does not constrain modeling ability.

3. Workshops benefit modelers and participants equally.

4. Trust derives from transparency.

5. Trust and truth are not synonyms in modeling.

6. Model and ecosystem complexity do not intend to match.

7. Model documentation is a form of storytelling.

Mystery 1: Everyone Is a Modeler

The central thesis of our modeling workshops is everyone is a modeler. Communicating this at the beginning of the workshop is crucial because it allows stakeholders to place themselves into the modeling process and take ownership of the model being created. Initially, participants often do not perceive themselves as modelers because of an assumed need for programming skills or deep mathematical training. However, people innately observe, categorize, organize, and predict their environment (Westervelt and Cohen, 2012). Workshops provide an opportunity for participants to integrate their innate modeling skills into the formal model development process. Lecture modules establish a common lexicon between technical and non-technical participants, which overcomes a common linguistic divide in transdisciplinary endeavors (Eigenbrode et al., 2007). Key aspects of model literacy are reiterated for all attendees (Seidl, 2017), and the lecture-based format instills a culture of learning in the model development exercise (Voinov and Bousquet, 2010). Together, these approaches place participants at the center of model development, rather than as observers, such that they are not just providing information to the modeling team.

Mystery 2: Mathematical Talent Does Not Constrain Modeling Ability

By far, the biggest challenge is clearly communicating that participants' perspectives can be turned into a model. Less experienced participants (i.e., non-modelers) have been understandably wary of this—they view environmental models as specialized tools that require years of advanced study in mathematics to develop. Many workshop participants expressed a fear and/or hatred of mathematics in any form, which has been observed empirically by reduced citation rates of biology papers presenting many equations (Fawcett and Higginston, 2012).

This issue manifests in the assumption that models can only be developed by modelers, but the linchpin for workshop success is clearly articulating that a person's mental models can be transformed into a set of equations without the use of sorcery. The workshop approach overcomes this obstacle by deconstructing the development of equations to its basal components. Model development is broken down into a series of iterative steps, following the spiral approach to learning (Harden, 1999), by first asking participants to create a verbal model without empirical data or numbers, then introducing increasingly numerical modeling concepts in a stepwise manner to demystify the mathematics (Figure 3). Starfield (1990) recommends an analogous qualitative to quantitative method for developing models based on participant specified rules.

Figure 3. Example of iterative, interactive progression of transforming mental models into equations for the relationship of salinity and seagrass habitat suitability. (A) Represents a general verbal model of the relationship between salinity and seagrass. (B) Introduces a simple linear equation representing these relationships (C) introduces the concept of an index function where 1 represents pristine habitat and 0 represents completely unsuitable, and (D) introduces data points from empirical studies and the relevant equations that form the basis of the model.

We have found for the purposes of our workshops that developing index-based models is the cleanest mechanism for facilitating model development with a large audience with diverse backgrounds. Briefly, index-based approaches attempt to distill the complexity of species-habitat relationships into a 0 to 1 scale, with 1 representing a favorable or “ideal” relationship and zero representing a completely unsuitable relationship (i.e., uninhabitable). These suitability indices (SI) are generally quantified as a step functions with the breakpoint values of the steps representing species response thresholds (e.g., what is the lowest salinity seagrass can tolerate before it dies? What is the minimum amount of inundation a wetland requires?). Equations are then generated by linearly interpolating between the steps. A given SI can have multiple steps and therefore multiple equations. These models can be quickly parameterized with information from the scientific literature, empirical data, and/or expert opinion. Suitable habitat can be assessed by using a single SI, but more commonly multiple indices are combined into a composite score that estimates suitable habitat quality across a range of environmental parameters of the species' niche (Hirzel et al., 2002; Tittensor et al., 2009). While index-based models are not predictive in nature (Swannack et al., 2012), they are useful for rapid assessments and scenario-based comparisons common in environmental decision making [e.g., the Habitat Evaluation Procedure (HEP) used by US Fish and Wildlife or the Hydrogeomorphic Method of wetland assessment used by the US Army Corps of Engineers (Brinson, 1993)].

In East San Pedro Bay, seagrass was identified as an ecologically important habitat type and focus of restoration. Turbidity, salinity and temperature were identified as critical parameters. Using salinity as an example to depict the process, we asked stakeholders to describe the seagrass-salinity relationship in terms of “good” or “bad” under the following conditions “too salty, too fresh, in the middle” (Figure 3A). We then demonstrated how qualitative data can be transformed into mathematical equations using a simple linear equation (Figure 3B). Most participants are familiar with the simple linear equation Y = MX+B, so we use this as the general functional form throughout the workshops. Before we move forward, we query the group to make sure this concept is understood. We then introduce the concept of index modeling where we transform good/bad into a 0 to 1 scale with 1 representing pristine habitat and 0 is completely unsuitable habitat (Figure 3C). We then introduce actual data values and generate the respective equations (Figure 3D), a step often occurring after a workshop when data may be compiled. This simplicity might be necessary to empower participants whose backgrounds are not in science or engineering. This transformative example seems to alleviate some of the reluctance to engage with mathematical representations of system components. We further developed this concept by creating an Excel spreadsheet that auto-generates graphics and equations based on participant input (Figure 1E; Carrillo et al., in review).

Facilitators and others more comfortable with mathematical modeling may be frustrated by these simple approaches and patience may be tested. However, buy-in gained from less technical team members is generally worth the time invested. An audience analysis (Hall et al., 2014) should be conducted before the breakout group for model quantification to avoid any potential conflicts within the groups, and mini-breakout sessions during lunches or after the formal sessions are recommended to assist those who are having difficulties.

Mystery 3: Workshops Benefit Participants Equally

Highly technical participants and experience modelers may initially view workshops as slow, redundant, or unnecessary. These participants sometimes assert that the problem is already solved (often by their own models), and there may be an inclination to only engage during certain sessions (e.g., quantification). However, model scoping and conceptualization activities often enrich more technical participants' understanding of the decision context, user needs, data availability, and system function. Mediated modeling approaches often emphasize the importance of workshops as an opportunity for stakeholders and less technical audiences to gain insight and buy-in (e.g., Langsdale et al., 2013), but our experiences indicate that workshops are crucial for contextualizing modeling and instilling a sense of humility and growth in more technical team members. A successful workshop environment seeks to provide an environment for co-development, establish participants as co-equals, and facilitate two-way communication.

Mystery 4: Trust Derives From Transparency

Models are often perceived as black boxes, and users want to know what is in the model to be able to trust the outputs. Early in the workshop series, facilitators would transform the conceptual model into a quantitative model after the first day's sessions without the participants present. While this approach is easier for the modelers, it was a total failure from the stakeholder perspective. Participants still viewed the model as a black box and did not embrace the model even though it was based on their conceptualization. We developed an interactive modeling toolkit (Figure 1E) that allows stakeholders to rapidly conceptualize and quantify the environmental relationships (Carrillo et al., in review). Briefly, the toolkit, written for Excel because most if not all the participants are familiar with the software, allows users to enter breakpoint values for index-based models based on expert opinion and/or empirical data and then equations are autogenerated. Environmental relationships are treated as step-functions with linear interpolations between the steps. Autogenerating the equations demystifies the modeling process because it allows stakeholders to see quantitative relationships that they conceptualized (Figures 3D, 1C). The model is no longer a black box because it is participant conceptualized and quantified. Building their own model invests stakeholders in the process and the outcome. Taking ownership over the model becomes part of the outcome of model development process and facilitates trust in the “black box.” Workshops indicate that participants in the model development process are generally more prone to buying into the model than those reviewing workshop products.

Mystery 5: Trust and Truth Are Not Synonyms in Modeling

Participants often assume that only validated or “accurate” ecological models are useful and should be trusted in environmental management. However, the model development community has long distinguished different uses of ecological models such as explanatory or predictive (Hall et al., 2014; Rastetter, 2017). Modules should provide participants with examples of model applications including structuring thinking, hypothesis testing, generating understanding, predicting outcomes, and relative comparison of alternatives. In particular, materials on model evaluation emphasize issues beyond “validation” and “truth” such as defensibility, sensitivity, parameter vs. structural uncertainty, and usability. Participants walk away with an understanding of the diversity of model uses along with an appreciation for trusting insights from tools without “believing” in them (Milner-Gulland and Shea, 2017). Workshops help communicate the model's limitations and provide all participants with the appropriate context for interpreting results and outcomes.

Mystery 6: Model and Ecosystem Complexity Do Not Intend to Match

Non-modelers view the complexity of their socio-environmental systems as an insurmountable, overwhelming obstacle that cannot be captured mathematically (Starfield, 1987; Nicolson et al., 2002; Addison et al., 2013). This perceived complexity stems, in part, from a deep understanding of their system which has evolved over time from first-hand experiences, localized cultural factors, and countless qualitative and quantitative data streams. Inexperienced participants sometimes assume that the entirety of the system should be represented because everything is connected to everything. However, too much detail can obscure important cause-effect relationships, which can ultimately lead to ineffectual decisions.

Undoubtedly stakeholders will want to represent components of their system that are not critical to decision-making or are not directly related to the modeling objective. This issue is tricky for modeling facilitators to address because the participants should be allowed to represent their perspective of the system without destructive criticism (van den Belt, 2004). In general, this issue is addressed during model conceptualization (but is often revisited multiple, multiple times throughout the other sections of the workshop). We have found that compiling the conceptual models from the conceptual modeling breakout groups into a master model for the class actually helps reduce model complexity. In general, each conceptual model contains similar components, which are often the critical pieces of the model (e.g., for East San Pedro Bay, turbidity was considered an important driver for all of the conceptual models). Each component depicted in the master conceptual model is discussed with special emphasis on why these components appeared in common in multiple models.

The balance between complexity and simplicity should be emphasized throughout the workshop using different mechanisms. For instance, lectures reiterate the importance of parsimony by highlighting Occam's razor and an aphorism often attributed to Einstein (i.e., “Everything should be made as simple as possible, but not simpler”). In providing example models, we identify processes well-beyond the scope of the model as counterpoints (e.g., extinction probability due to meteor strike). A running list of “what's not included” can help participant make notes of assumptions in conceptual modeling. These mechanisms empower participants to add complexity to models where needed, while not getting mired in the natural complexity of open, constantly varying ecosystems.

Mystery 7: Model Documentation Is a Form of Storytelling

Models used for decision-making require clear, transparent documentation from the model developers so models can be easily recreated, applied and understood by pertinent audiences. Elements of good technical modeling documentation typically recommend: (1) methods or process used in development, (2) the ability of a model to be recreated, (3) a detailed list of assumptions, and (4) ranges of appropriate usage. Efforts have been made to improve model documentation (Rykiel, 1996; Kettenring et al., 2006; Schmolke et al., 2010; Hall et al., 2014), but they are often ignored. A detailed discussion of the reasons for poor model documentation is outside the scope of this paper, but briefly, model documentation is often overly technical and without a clearly defined, logical process that describes how the model was developed.

Mediated modeling workshops provide an opportunity to link model documentation to different audiences and users. Specifically, workshop participants often represent key technical reviewers, the end-user community, or an audience for non-technical documentation. Workshops facilitate an understanding of different audiences' needs and the “story” of the model that needs to be told, thus providing an implicit form of audience analysis (Hall et al., 2014). For instance, a technical reviewer may raise issues about documentation of functional forms, data sources, or programming techniques. Conversely, non-technical users may be more interested in the general outcomes of the model, visualization of results, or implications of model simulations. The technical participants assembling model documents are able to listen to all of these needs and understand what is expected of the model documentation they will ultimately produce.

Likewise, for a model to be useful, stakeholders need to have a general impression of the system components and interactions (Hall et al., 2014). Interactive engagement in model development ensures that a common understanding is reached among participants. The model itself is no longer viewed as overly technical because the participants observe exactly where the equations come from and how they are generated.

As such, workshops add relatability to the documentation of a model. That is, they help safeguard that the model is being communicated in such a way that the technical developers understand the audience and the audience understands the broad-strokes of development. The clear connection between technical and non-technical participants provides bi-directional transparency for overcoming a well-acknowledged model communication shortcoming (i.e., documentation).

Concluding Remarks

The need for numerical ecological models continues growing as scientists are asked to forecast the effects of management actions, anticipate future conditions and understand ecosystems without historical analogs (Carpenter and Turner, 2017). Ecological models are also increasingly integrated across disciplines, modeling traditions, and epistemologies (Voinov and Fishwick, 2019), particularly for multi-objective problems like ecosystem restoration, habitat conservation, and integrated water resource management (Addison et al., 2013; Langsdale et al., 2013; Badham et al., 2019). As models become more ubiquitous, the need for clear, transparent communication strategies are essential for distilling model complexity and achieving consensus from diverse audiences and stakeholders that are involved with the decision-making process. Stakeholder groups must understand what the model represents, why it was structured the way it was, and its intended use.

In this paper, we have presented a repeatable and reproducible workshop approach for mediated modeling that engages technical and non-technical audiences in model development, application, and communication. Our approach uses brief concept-based informational session coupled with intensive break-out group modeling exercises that target specific phases of model development. Based on more than a dozen workshops, attended by participants from over 50 different organizations (federal, state and local agencies, universities, NGOs), we have identified a common set of model mysteries that workshop participants typically encounter with respect to modeling, and we summarize how each issue is demystified and clarified by the workshop process. It is difficult to quantify specific metrics regarding how well our methods distilled model complexity. However, we can point to large stakeholder groups using models developed in these workshops to inform decision making for ecosystem restoration projects throughout the United States. Feedback from participants indicates that workshop design and real-time model building decrease the amount of time and resources normally dedicated to reach consensus among their stakeholders. Model communication remains an underemphasized and challenging issue, but co-development workshops provide a useful mechanism for shedding light on what is traditionally viewed as the “black box” problem of ecological model development.

Author Contributions

BH, SM, SA, NR, MR, CP, and TS contributed the conceptualization, development of the workshop series, wrote sections of the manuscript, contributed to manuscript revision, read, and approved the submitted version. BH wrote the first draft of the manuscript.

Funding

The U.S. Army Corps of Engineers funded this research through the Ecosystem Management and Restoration Research Program, a variety of USACE District studies, and two different internal training programs.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

The authors kindly acknowledge all of the participants who have attended these model development workshops. T. W. Clumpkin provided an excellent review of an earlier draft. The use of products or trade names does not represent an endorsement of these products by either the authors or the Department of the Army. Opinions expressed here are those of the authors and not necessarily those of the agencies they represent.

References