Making the most of mental models: Advancing the methodology for mental model elicitation and documentation with expert stakeholders

Eliciting stakeholders ’ mental models is an important participatory modeling (PM) tool for building systems knowledge, a frequent challenge in natural resource management. Therefore, mental models constitute a valuable source of information, making it imperative to document them in detail, while preserving the integrity of stakeholders ’ beliefs. We propose a methodology, the Rich Elicitation Approach (REA), which combines direct and indirect elicitation techniques to meet these goals. We describe the approach in the context of the effects of climate change on Baltic salmon. The REA produced holistic depictions of mental models, with more variables and causal relationships per diagram than direct elicitation alone, thus providing a solid knowledge base on which to begin PM studies. The REA was well received by stakeholders and fulfilled the substantive, normative, instrumental, and educational functions of PM. However, motivating stakeholders to confirm the accuracy of their models during the verification stage of the REA was challenging.


Introduction
Mental model, or cognitive map, elicitation is an essential tool within the increasingly popular field of participatory modeling (PM) (Voinov et al., 2016;Voinov and Bousquet, 2010), owing to its utility in formalizing knowledge and facilitating problem-solving ( € Ozesmi and € Ozesmi, 2004). This technique extracts a person's "internal representation of an external reality (Jones et al., 2011)," or put another way, their conceptualization of a system's causal dynamics (Moray, 1998), built on personal experience, knowledge, and values (Johnson-Laird, 1983;Jones et al., 2011) to cognitively aid in reasoning (Johnson-Laird, 2010;Nersessian, 2002). Here we develop a methodology, the Rich Elicitation Approach (REA), to improve the elicitation and documentation of stakeholders' mental models. Advancing these processes is crucial to ensure they adequately contribute to the functions of PM (Fiorino, 1990;Jones et al., 2009), which are: 1) The normative function, which suggests, incorporating stakeholder knowledge into a model increases its legitimacy in the decisionmaking context. 2) The substantive function, which is the capacity to synthesize available knowledge from a variety of sources to enhance problem-solving. 3) The instrumental function, which describes the process of relationshipbuilding between stakeholder groups, 1 may reduce conflict and ease the implementation of decisions made using model outputs.
PM also serves a fourth function.
4) The educational function, which describes the act of engaging with the PM process as an educational experience for stakeholders Voinov and Bousquet, 2010).
The REA primarily develops the substantive function of mental model elicitation and documentation, while preserving the educational experience elicitation provides Voinov and Bousquet, 2010). However, we briefly address our participants' perceptions of the process's normative and instrumental value as well.
Substantive gain is an important output of many PM techniques, including mental model elicitation, which collect, inventory, and build knowledge relevant to a given issue, particularly in complex and datapoor contexts ( € Ozesmi and € Ozesmi, 2004). Under such circumstances, stakeholder knowledge is often the best available source of information (Krueger et al., 2012;Kuhnert et al., 2010;Sutherland, 2006), hence its prior application to a variety of natural resource issues, including, fisheries stock assessment (Chrysafi et al., 2017;Haapasaari et al., 2013Haapasaari et al., , 2012M€ antyniemi et al., 2013) and Arctic oil spills (Nevalainen et al., 2018). Mental model elicitation specifically, provides substantive gain because individual stakeholders' mental models offer alternative hypotheses about a system's causal dynamics (Krueger et al., 2012) and when aggregated, allow for the co-production of systems knowledge (Olazabal et al., 2018). Stakeholder input is particularly relevant for solving high-stakes natural resource-related problems when time is limited and action cannot be delayed while more formal scientific knowledge is generated (Kangas and Leskinen, 2005;Knol et al., 2010). Moreover, mental model elicitation serves an educational function, allowing stakeholders to engage with their cognitive structures, exposing personal knowledge gaps (Kaplan and Kaplan, 1982;Zellner, 2008) and aiding the development of an integrated understanding of complex socio-ecological systems (Fortuin et al., 2011;Novak and Cañas, 2008;Smajgl and Ward, 2013). Hence, the use of mental models is commonplace in a variety of PM methods, including the development of Bayesian belief networks (Haapasaari et al., 2012;Meynecke et al., 2017;Smith et al., 2018), fuzzy cognitive maps (Olazabal et al., 2018;€ Ozesmi and € Ozesmi, 2004;Solana-Guti� errez et al., 2017), conceptual content cognitive maps (Kearney and Kaplan, 1997) and Actors, Resources, Dynamics, and Interactions models Mathevet et al., 2011). Guidelines for discerning whether mental modeling (cognitive mapping) is an appropriate methodological choice for a given study are given by Voinov et al. (2018).
Presently, stakeholders' mental models are obtained via either direct or indirect elicitation (Jones et al., 2011). During direct elicitation, participants, individually or in groups, actively create and define the structure of their models themselves, typically assisted by a facilitator and visualization tools depicting system variables and the connections between them (Dray et al., 2006;Haapasaari et al., 2012;€ Ozesmi and € Ozesmi, 2004). This deliberate articulation and visualization of knowledge constitute a learning experience for participants (Marcot et al., 2001;Uusitalo, 2007). The hallmark of direct elicitation, however, is an immediate means of verifying participants' representations of their mental models (Jones et al., 2011), reducing reliance on the skill and interpretation of an analyst (Abel et al., 1998). This immediate feedback is likely why direct elicitation is more common than indirect elicitation in natural resource-related PM. Alternatively, during indirect elicitation, an analyst determines the structure of the participant's mental model based on textual information, like interview transcripts or written questionnaire responses (Carley and Palmquist, 1992;Masinde et al., 2018;Verkerk et al., 2017). However, an important and largely unexplored question is, do these techniques accurately transfer knowledge from brain to paper?
This question became important when we noticed that the mental models we documented with stakeholders during direct elicitation were simplified in comparison with their verbal descriptions, equating to a loss of potentially critical information. We suspect this was the unintentional result of time constraints and stakeholder fatigue (Burgman, 2005) coupled with the difficulty in articulating and visualizing variables and complex model structures, both on the part of the stakeholder and the PM facilitator. Hence, direct elicitation did not represent stakeholders' mental models as accurately as we would have hoped. We believed taking an indirect elicitation approach instead may have reduced simplification and subsequent information loss, as this technique allows an analyst time to carefully consider and define model variables and structure. However, this technique is also susceptible to inaccuracies, since an indirectly elicited representation of a stakeholder's mental model reflects only the analyst's interpretation of it, which may be influenced by their own biases, beliefs, and values. Besides, we were not keen to give up the educational benefits direct elicitation provides. Therefore, we propose combining direct and indirect elicitation methods to retain the strengths of each, while compensating for their shortcomings.
Radonic (2018) adopted a related approach in his cultural anthropological study, using audio recordings of stakeholder interviews to ensure they had succeeded in drawing the concepts vocalized during direct elicitation (Radonic, 2018). We take Radonic's approach one step further, employing indirect elicitation as a verification method for directly elicited models and as a measure to ensure they are represented in the same level of detail as described verbally while maintaining the stakeholder's control over their accuracy. We demonstrate that the REA reduces information loss compared to direct elicitation alone, resulting in "rich models," which are holistic depictions of stakeholder knowledge created per their understanding of the world around them.
In addition to improving the process of mental model elicitation, we also believe it is imperative to improve the documentation of PM methodologies, which is often poor or bypassed altogether in the literature (Voinov et al., 2016). Some resources provide general guidelines relevant to the PM process (e.g., Durham et al., 2014), but peer-reviewed articles describing the lessons learned from completed PM studies are few Nevalainen et al., 2018) and the methodological mistakes others could learn from are rarely published (Krueger et al., 2012). Since PM methods are included in virtually all environmental modeling efforts today Voinov and Bousquet, 2010), it is important to describe the techniques used to produce and analyze the results of these processes. To these ends, Gray et al. (2018) assert that despite increasing interest, PM has not been able to establish itself as a cohesive field of study owing to poor reporting and a lack of reproducibility. As a solution, they suggest the "4P framework" for reporting PM studies, which encourages documentation of (1) the purpose for choosing the PM approach (why); (2) the process used to involve participants (how); (3) the partnerships formed as a result of the PM process (who); and (4) the products resulting from the PM study (what). We are dedicated to the improvement of PM methodologies, reproducibility, and wish to promote the effective and transparent use of stakeholder knowledge in natural resource management, and as such, report our study using the 4P framework.
Therefore, the purpose of this article is to present the methodology (the REA) we used to enhance mental model richness, without sacrificing the integrity of stakeholders' ideas or their learning experience, through the combination of direct and indirect elicitation techniques. Further, we hope this article will contribute to the development of guidelines for best practices in PM, thus deepening the rigor and expanding the utility of these techniques in the scientific process.

The why: The effects of climate change on the salmon system
We tested the REA in the context of a problem-framing study, a process for exploring the dimensions of the problem from multiple perspectives and re-defining it to allow for the development of clearer, more mutually beneficial, and creative solutions (Bardwell, 1991). The goal of our problem-framing study was to determine the effects climate change may have on Atlantic salmon (Salmo salar L.) in the Baltic Sea, hereafter referred to as "Baltic salmon" and to begin to adapt salmon management accordingly. We explored this problem with expert stakeholders individually, asking them targeted questions and documenting the mental models underlying their answers. This text specifically addresses the elicitation of these expert stakeholders' mental models, not their analysis or aggregation. These steps were conducted later to produce a synthesis of these individual results, which is intended to be the first step in a multi-step modeling process to incorporate climate change effects into the pre-existing Baltic salmon stock assessment model (Michielsens et al., 2008). The results of this model form the basis of the International Council for the Exploration of the Seas' (ICES) fishery management advice to the European Union (Kuikka et al., 2014). Here we provide the context, which prompted us to begin our study.
Baltic salmon management has been a priority since the formation of the internationally adopted "Salmon Action Plan" in 1997, implemented in response to severe population declines, attributed largely to decades of overfishing (Romakkaniemi et al., 2003) and reduced access to spawning rivers (Romakkaniemi et al., 2003). In recent years, however, salmon populations have rebounded and fish have been observed returning in increasing numbers to many of their traditional spawning grounds (HELCOM, 2011;ICES, 2018;LUKE, 2016). Nevertheless, Baltic salmon are still considered vulnerable (HELCOM, 2013). Therefore, continued management efforts to support the long-lasting sustainability and recovery of salmon stocks in the Baltic Sea are imperative, including prompt action to address existent and emerging threats.
While many factors, like overfishing (Romakkaniemi et al., 2003), reduced access to spawning rivers (Romakkaniemi et al., 2003), changing food web dynamics, and nutrient deficiencies (Ejsmond et al., 2019;ICES, 2018) are already understood to affect salmon, climate change presents a new challenge for these fish, since this phenomenon is expected to bring substantial change to the Baltic Sea environment (Graham, 2004;HELCOM, 2013;Reusch et al., 2018). Though articles discussing the effects of these changes on Baltic salmon are limited, a tentative link between these fish and climate change has been established (Huusko and Hyvarinen, 2012;Jokikokko et al., 2016;Jutila et al., 2005;Kallio-Nyberg et al., 2004;Russell et al., 2012). Nevertheless, the causal mechanisms by which climate change is likely to impact Baltic salmon are under-researched at present. Therefore, we believe it is critical to develop a better understanding of the ways in which climate change may affect or may already be affecting Baltic salmon and their fishery, to better direct further research, explore management goals and strategies, and to begin the process of incorporating these effects into ICES' existing stock assessment model as expediently as possible. Although climate change is by no means the only factor influencing Baltic salmon populations, incorporating its effects into ICES' model is a step towards ensuring stock estimates are realistic and that they thereby assist in producing reasonable management recommendations. The results of this problem-framing can also help kick-start a conversation about the impacts of climate change on salmon, how management goals and strategies may need to be reassessed considering these changes, and the importance of including this issue in a new long-term management plan for Baltic salmon stocks. Additionally, since the research linking climate change and Baltic salmon is limited, it is worthwhile to develop a holistic understanding of the intertwined social and ecological systems linking climate change, salmon, and their fishery, as changes in salmon populations are likely to cascade, impacting other species, the environment, and human society. This task is time-sensitive and research about the topic is lacking. Therefore, we identified PM, specifically mental modeling, as an appropriate strategy to develop a knowledge base about the effects of climate change on the "salmon system."

Who & how: Eliciting & documenting mental models via the Rich Elicitation Approach
To build our knowledge base about the salmon-climate change problem, we elicited stakeholders' mental models and documented them as influence diagrams using the REA approach, illustrated in Fig. 1. Here, we describe our experience conducting this process, although we expect that the REA will be adapted to suit the specific contexts where it may be applied in the future (Voinov et al., 2016). Those interested in implementing the REA should consult both this section of the text and the discussion section for guidance.

Stakeholder selection
We included only expert stakeholders, hereafter referred to as "stakeholders," in our study, since domain-specific knowledge enhances mental model richness (Nersessian, 2002). The distinction between experts, stakeholders, and expert stakeholders is presently fuzzy within the PM literature (Krueger et al., 2012). However, we consider expert stakeholders to be individuals encompassing both the concept of expert, defined by extent and depth of their experience (Fazey et al., 2006), and stakeholder, broadly considered to be those who influence or are influenced by the research in question (Durham et al., 2014). We considered stakeholders working with salmon and salmon issues, either professionally or as part of a registered leisure organization, like angling clubs, to be sufficiently experienced to be considered experts. Although we determined expert stakeholders to be the correct group to engage for this PM process, we recognize that salmon are a common-pool resource, managed for the benefit of society and the environment. As such, engaging non-expert stakeholders should also be considered at appropriate points in the salmon management process.
We identified stakeholders to participate in our study via snowball sampling (Browne, 2005). First, we reached out to our contacts whom we considered to be Baltic salmon experts based on their contributory and interactional expertise . Then, we asked them to pass on our request for participation to others who might be interested and fit our criteria.
The 11 stakeholders who chose to participate described their salmon expertise in a variety of contexts and came from a diversity of organizations, including a transnational management agency, a government ministry, a university, three county management agencies, and five nongovernment organizations (see Table A1). Only stakeholders working in Finland and Sweden were asked to take part, as the majority of Baltic salmon production occurs in these two countries (ICES, 2018). To protect our participants' privacy, they were randomly assigned a letter pseudonym from A-K, e.g. "stakeholder K."

Facilitator learning
Before the direct mental model elicitation sessions, hereafter referred to as "elicitation sessions," mock sessions were conducted to help the facilitator develop the skills needed to elicit and document mental models. Afterward, we revised the elicitation session protocol and questionnaire per the mock participants' comments and critiques.

Direct mental model elicitation sessions & administering the questionnaire
Elicitation sessions were semi-structured and one-on-one between a facilitator and stakeholder, lasting approximately 2 h each. We chose to conduct elicitation sessions individually because we were interested in aggregating the thoughts of individuals, rather than producing a single collective response. The rationale behind this decision was to avoid the influence of perceived normative pressures (Heeren Alexander et al., 2016) on the stakeholders' depictions of their mental models, or over-representing the views of the most influential stakeholders (Burgman, 2005;Martin et al., 2012). Stakeholders were asked to describe their own beliefs rather than attempting to represent those of their organizations and audio was recorded throughout the duration of each session.

Stakeholder learning
At the beginning of their elicitation session, the stakeholder was asked to describe their career background and interests related to salmon. Next, the facilitator prepared the stakeholder for the task of representing their mental model as an influence diagram (Haapasaari et al., 2012), by explaining how to interpret and visualize them in this format, using simple examples (Fig. 2).

Fig. 1.
Schematic diagram representing the Rich Elicitation Approach (REA) from beginning to end. The bolded text represents the steps we carried out during our implementation of the REA and italicized text represents steps we recommend adding for future use (see discussion).
Influence diagrams include three variable types: uncertain variables, actions, and personal valuation of the outcomes (goals) (Haapasaari et al., 2012). These variables are then connected with arrows representing the causal relationships between them and the direction of the effect (Haapasaari et al., 2012). Influence diagrams include personal uncertainty about these relationships, expressed as degrees of belief, which are elicited either qualitatively (Haapasaari et al., 2012;Varis and Fraboulet-Jussila, 2002;Varis and Lahtela, 2002) or quantitatively as joint probability distributions (e.g. M€ antyniemi et al., 2013). By eliciting quantitative degrees of belief, influence diagrams are easily transformed into risk assessment models (Haapasaari et al., 2012), however, for problem-framing, we chose a qualitative approach, representing uncertainty by the thickness of the arrows drawn between variables (with thicker arrows representing more certain relationships). Whether quantitative or qualitative, model building, as influence diagrams or otherwise, is a useful tool to encourage participants to think deeply and clearly articulate their thoughts (Lynam et al., 2007;Marcot et al., 2001;Uusitalo, 2007).

Direct elicitation
During the elicitation session, the facilitator asked the stakeholder to consider three primary problem-framing questions, adapted from those used by Haapasaari et al. (2012): 1) What variables and causal relationships do you think should be considered when determining the impacts of climate change on Baltic salmon and their associated fishery? 2) What goals do you have for salmon and their fishery in the future considering climate change? 3) What management strategies or actions can be undertaken to achieve those goals?
The purpose of the first question was to elicit the stakeholders' mental models of both the direct and indirect cause and effect relationships between salmon, the environment, society, and climate change. The second explored these mental models further, eliciting the future outcomes the stakeholders hoped to see for salmon, highlighting the perceived value of this species, it's utility, and the importance of addressing the issue of climate change. We asked the final question to identify human actions that stakeholders believed affect or could affect the achievement of these goals. These questions were reiterated as needed over the course of the elicitation session. Other questions asked by the facilitator were meant to provide clarification, keep the session on task, or prompt discussion while influencing the stakeholder as As a starting point for their influence diagrams, we gave the stakeholders one variable, climate change. We also provided them with a core biological model (Fig. A1) originally developed to depict the annual stock dynamics of Central Baltic herring by Haapasaari et al. (2012). Its inclusion in their diagrams was optional. The rationale behind providing this model was to give the stakeholders some sense of the types of connections that could be drawn between different aspects of the salmon life history and the fishery system, and to provide a starting point for idea generation. At the time, we believed the herring model was general enough to be repurposed for this use and vague enough to avoid the anchoring effect (Oppenheimer et al., 2008;Tversky and Kahneman, 1974).
The influence diagrams were hand-drawn on large sheets of paper and stakeholders chose to either draw the diagrams themselves or allow the facilitator to draw. When the facilitator was responsible for drawing, she encouraged the stakeholders to explicitly describe the relationships between variables, frequently reminded them to interject if the influence diagram was not drawn according to their beliefs, and regularly confirmed the model's accuracy. After defining the system variables and the structure of their influence diagrams, stakeholders were asked to express the certainty or "strength" of the relationships between variables as either (1) weak, (2) medium, or (3) strong. Recording these "effect strengths" was the last step in the elicitation process, to avoid disrupting the stakeholders' flow of thoughts as they constructed their models.

Administering the questionnaire
Following each elicitation session, we sent the stakeholder a link to an online questionnaire created on the SurveyPlanet© (https://surveypl anet.com/) website. The questionnaire included questions intended to (1) contextualize thestakeholders' perceptions of the salmon-climate change problem, (2) determine the utility of problem-framing and the elicitation process, and (3) assist in improving future elicitations. The questions were asked in multiple-choice, scoring, and short answer formats. Most multiple-choice and scoring questions were followed by space for the stakeholders to elaborate if desired. Unexpectedly, Sur-veyPlanet recorded multiple values when a stakeholder chose to change their answer. In these cases, it was impossible to distinguish their final choice and therefore, we report the average of the two answers. Stakeholders could skip questions they did not wish to answer. All responses were anonymous, except for one stakeholder who completed the questionnaire in paper form and therefore, that stakeholder's responses were not anonymous to us.

Digitizing influence diagrams
Once the elicitation sessions were completed, the facilitator digitized each hand-drawn influence diagram using GeNIe 2.0 Academic software (BayesFusion, LLC).

Transcription, note taking, & coding
To begin the indirect elicitation phase of our study, the audio recording from each elicitation session was transcribed. Then, the analyst read and coded each transcription, categorizing passages as variables within the salmon system, causal linkages between them, effect strengths, or remarks about the elicitation session itself. Additionally, the analyst recorded two sets of notes on each transcription, hereafter referred to as "elicitation notes (A) and (B)." (A) documents the stakeholders' comments during the elicitation session about the activity itself, major themes described during the session, and novel ideas about either the elicitation process or the effects of climate change on salmon. (B) documents the stakeholders' predictions about how climate change will affect Baltic salmon and their associated fishery, which are not necessarily reflected by their influence diagrams. These notes were used for two purposes: (1) to analyze and improve the elicitation process and (2) to supplement problem-framing analysis later.

Enhancing influence diagrams
Following coding, each directly elicited influence diagram was "enhanced" by restoring all the variables and causal relationships that had been described verbally by the stakeholders during the elicitation session, but which were later identified by the analyst as missing or oversimplified upon inspection of the coded transcripts and elicitation notes (A).

Sending enhanced diagrams for stakeholder approval & first revision
Once the enhanced versions of the influence diagrams were completed, they were sent back to the stakeholders via email for their approval and to ensure the analyst had represented their beliefs accurately. The stakeholders were given a deadline to contact the analyst to express their approval or disapproval of the diagrams and describe any changes they wished to make. They were explicitly instructed that no response would be interpreted as their approval. Additionally, stakeholders were asked to provide effect strengths for any new causal linkages included in their influence diagram or original linkages that had not been assigned a strength during the elicitation session. After the allotted time had passed, any corrections the stakeholders indicated were addressed by the analyst and the influence diagrams were revised.

Variable harmonization & final revision
Next, variables included in the influence diagrams were harmonized at the analyst's discretion, as was done in Martinez et al. (2018) and Olazabal et al. (2018), as stakeholders often articulated the same concepts in slightly different terms. For example, "how warm the river is" and "the temperature of the river" were both changed to "temperature: river." After harmonization, the influence diagrams were revised a final time to reflect these changes.

Methodological evaluation
An evaluation of our methodology's performance was conducted in two parts. First, we analyzed the REA's ability to produce richer depictions of stakeholders' mental models. We compared the number of variables and causal relationships included in the influence diagrams produced via direct elicitation alone with their final versions, completed after indirect elicitation, verification, and standardization. In addition to interest in producing richer, more accurate mental models, we also believe that by taking note of stakeholders' experiences and suggestions, we can improve the process of mental model elicitation, leading to better results in the future. Therefore, for the second stage of our methodological evaluation, we consulted select questionnaire questions and elicitation notes (A) to assess how well the stakeholders felt the approach addressed the four functions of PM. Additionally, we used the stakeholders' comments and recommendations extracted from the coded transcriptions and the facilitator's observations to identify areas for methodological improvement and suggest solutions. The results of the methodological evaluation of the REA will be the focus of the remainder of this article.

Comparison of influence diagrams post-direct elicitation versus post-REA
The REA produced richer representations of stakeholders' mental models than direct elicitation alone. Following the REA, the stakeholders' diagrams contained more variables and causal relationships. See Table 1 for a numerical comparison of the influence diagrams post-direct elicitation versus post-REA and Fig. 3 for a visual comparison. Table 2 summarizes the stakeholders' responses to the questionnaire questions concerning their perceptions of the mental model elicitation process and its ability to fulfill the four functions of PM. Although their responses varied, they generally found the process useful for fulfilling these functions. They were most convinced that mental model elicitation offers substantive and educational gains, with the clear majority recording the highest or second-highest score in favor of these capabilities. The stakeholders were less certain, however, about mental model elicitation's normative and instrumental value. Their opinions spanned from negative to positive for most questions in these categories (see Table 2). However, the stakeholders would, in general, feel more satisfied with management decisions if their influence diagrams were considered during the decision-making process (normative), exemplified by majority reporting scores of �4. When asked whether mental model elicitation could be useful in reaching consensus about how to manage the fishery (instrumental), the majority of stakeholders reported scores of �4. The second instrumental function question received the highest number of negative responses; when asked if they would feel more invested in the scientific process by attending problem-framing events with researchers (instrumental), three stakeholders responded with a score of �2, two with non-committal 3s, and six with scores of �4.

Evaluation of the elicitation process
The general positive attitude toward mental model elicitation was reflected in the elicitation notes (A) as well, with several stakeholders remarking that they found the process interesting. Two stakeholders asked to take pictures of their influence diagrams to show colleagues and one later enquired about facilitating a group mental model elicitation for problem-framing about another topic. One stakeholder mentioned that the elicitation process helped him think deeply about his ideas, specifically in response to the facilitator's question about why biodiversity is important, intended to encourage him to clarify his thoughts for the influence diagram.
During the elicitation sessions, the majority of stakeholders quickly understood the process of creating influence diagrams. Only three expressed some uncertainty or difficulty beginning the task. This challenge was largely overcome, however, since the majority of stakeholders preferred to allow the facilitator to draw. Three suggested both they and the facilitator should contribute. While the facilitator was drawing the diagram, some stakeholders were highly engaged in ensuring their thoughts were represented accurately, providing detailed instructions about how the diagram should be drawn. Others were more interested in verbalizing their thoughts than documenting them on paper.
The core biological model (see appendix) seemed to be of little interest to the stakeholders. Three stakeholders mentioned they felt it might be useful and wished to keep it on hand as reference material.
However, only one stakeholder incorporated a portion of the model into his influence diagram.
Understanding how to document the strengths of the effects between variables proved to be a more conceptually challenging task than understanding how to draw influence diagrams. Two stakeholders wondered whether the effect strengths should be considered in isolation, concentrating on only the strength of the effect between one variable and another, or whether an entire causal pathway with several connected variables should be given the same effect strength. In such cases, the stakeholders were advised to concentrate on individual linkages between two variables. There was also some concern over the relative proportions of the three types of effect strengths. One stakeholder wondered whether there should be 33% of each strength category included in the diagram. Some also questioned whether it was appropriate to include a greater proportion of one strength than the others. Based on the stakeholders' comments, it is also apparent that the effect strengths represented their level of certainty about the relationship between two variables, as intended, but also perhaps their perceived importance of the relationship. When questioned about his reasoning behind his assignment of effect strengths, one stakeholder stated the weak effect strengths represented relationships he was uncertain of. Another mentioned, however, that a strong effect represented established knowledge and also signified that the relationship was important. Two stakeholders chose not to include effect strengths in their diagrams at all. While one did not give a reason for this, the other was too fatigued after creating the influence diagram to continue with the effect strengths as well.
Many stakeholders felt uncomfortable with the uncertainty inherent in the complex interactions between climate change, salmon, and society. Some also found the scope of the problem difficult to grapple with. These issues were reflected in one stakeholder's comment that "there are factors (related to the effects of climate change on the salmon system) that we cannot even begin to speculate about" and another stated that "there are so many dependencies and feedbacks that I find it difficult to write down." Phrases like "I guess" and "I don't know" were very common throughout the elicitation sessions. This uncertainty promptedthe facilitator reassure the stakeholders and remind them that they were not expected to produce a correct diagram, but rather one reflecting their own best judgments.
Several stakeholders acknowledged the complexity of the influence diagrams as the elicitation sessions progressed, most commonly by indicating that they felt the diagrams were "messy." Two stated their unease with the complexity of the maps more explicitly. One jokingly commented, "there might be too much information (in the diagram)" and another indicated that portions of his diagram had become confusing due to the number of causal linkages.
The facilitator also remarked that the stakeholders may have altered their description of their mental models in response to her background credentials. For example, some stakeholders meticulously described the salmon lifecycle when they assumed the facilitator to be a non-expert, but became less explicit about perceived areas of common knowledge after asking about her educational background and work experience in fisheries science. After the enhanced versions influence diagrams were sent back to the stakeholders (Fig. 1), only two responded. One stakeholder remarked that the new diagram looked "complicated" and he trusted our judgment to represent his thoughts accurately. A second completed the task in its entirety, returned the diagram with most of the effect strengths filled in and confirmed his approval of the enhanced diagram. The remaining nine stakeholders chose not to respond to our inquiry, thereby tacitly giving their approval. One stakeholder contacted us about completing the task after the allotted time but never submitted any requested changes.

Rich representations of stakeholders' mental models
Demonstrated by the clear increase in the number of variables and causal linkages included in the stakeholders' influence diagrams after implementing the REA, coupled with our process to ensure all their articulated thoughts were documented clearly, leads us to conclude that the REA was successful in producing richer models. We believe this increase in detail equates to more accurate depictions of stakeholders' thoughts, allowing us to take fuller advantage of the valuable knowledge they possess. In short, the REA improves the transfer accuracy of mental models from brain to paper, making a more complete depiction of stakeholders' knowledge accessible. As such, we suggest rich models are useful in the development of a holistic knowledge base from which to begin comprehensive PM studies on complex knowledge-limited socioecological systems.
Although the REA provides methodological advancements intended to reduce unintentional model simplification, we argue a paradigm shift is also necessary to reduce premature intentional simplification. Typically, studies using either direct or indirect elicitation methods contain relatively few variables and causal relationships, indicating that needless simplification and subsequent information loss may be widespread in mental model elicitation of both types.
Purposeful simplification may be driven by adherence to a "simpler is better" mindset, analytical and technological constraints, or a lack of resources needed to conduct in-depth and time-consuming studies (Kuparinen et al., 2012). The merit of the simpler is better mindset is often attributed to the principle known as Occam's razor. However, the principle states, "entities should not be multiplied without necessity" (Schaffer, 2015). As such, the definition of necessity dictates when and where Occam's razor should be applied. Here, the objective is to gather a holistic picture of potentially important social and environmental factors according to a set of stakeholders. The diversity and depth of thought they provide useful for informing the models and the problem-framing results we use to better understand and develop solutions to complex and poorly understood socio-environmental issues. In this sense, the acquisition of holistic mental models is necessary to develop the broad and comprehensive knowledgebase this task requires. From this perspective, everything a stakeholder says is necessary and should not be discarded on grounds of artificial preference for simpler explanation or practicality. Therefore, we urge PM practitioners to alter their perception that simpler is always better and to do their best to meet the practical demands of studying complex systems. After all, "To keep every cog and wheel is the first precaution of intelligent tinkering" -Aldo Leopold (1972).

Rich mental models & their applications in environmental modelling & problem framing
Environmental modeling processes begin with perceptual models, which are systematic and qualitative representations of reality, analogous to the mental models discussed here (Beven, 2009). These perceptual models are later simplified and abstracted to accommodate the terms of mathematics and coding (Beven, 2009). Since necessary simplification is built into the modeling process, which should be deliberate and iterative by default (Jakeman et al., 2006), a rich perceptual model should not hinder later stages of the model development process. Additionally, it is more practical to reduce a rich model than to return to stakeholders later to develop a wider knowledge base if new ideas, hypotheses, and potential solutions are required. We posit that if care is taken to ensure mental models are elicited from truly expert stakeholders dedicated to the success of the PM project, richer models may provide a more realistic understanding of complex natural resource problems. Improving realism is frequently considered a goal of environmental modeling (Aben et al., 2016;Kuparinen et al., 2012), since improved realism equates to a more accurate representation of the world and therefore, improved model performance. Kuparinen et al. (2012), illustrate the necessity of model realism in the context of fisheries stock assessment models, notorious for their lack of biological realism and tendency to neglect relevant information. As a result, these models have frequently failed to provide adequate population dynamics information, contributing to stock collapse.
Problem-framing, which may be used to develop perceptual models at the outset of environmental modeling processes (Haapasaari et al., , 2012, is intended to develop a holistic understanding of a problem to best direct problem-solving (Bardwell, 1991) in the research, policy, management, and risk assessment arenas. As € Ozesmi and € Ozesmi (2004) suggest, the more comprehensive the understanding of a problem is, the more interventions that can be identified and explored. Since mental model elicitation is often included in problem-framing, it is imperative to elicit rich mental models to ensure a more detailed Table 2 Stakeholder responses to relevant questionnaire questions. The questions are grouped into four categories: substantive function, normative function, instrumental function, and educational function, based on the four functions of PM. Stakeholders provided a ranking between 1 and 5 for each question according to a scale where 1 ¼ No, definitely not and 5 ¼ Yes, definitely. NR ¼ No response. X represents one stakeholder's response. The term "mind mapping" is analogous to "mental model elicitation." understanding of the issue at hand, reducing the chance that important aspects are overlooked.
In addition to the sources of simplification described, simple mental models may also indicate poorly informed stakeholders, since learning domain-specific information is linked with increased model richness and reasoning capabilities (Kinchin et al., 2000;Nersessian, 2002). However, this is a question of stakeholder selection and beyond the scope of this article.

Fulfilling the functions of PM
Based on the feedback the stakeholders provided, they were interested in the elicitations sessions and building their influence diagrams, found the topic relevant, and felt that the incorporation of their influence diagrams (i.e. knowledge) into the fishery management process could substantively aid in problem-solving and decision-making. Although their perceptions of elicitation and more generally, problemframings' normative and instrumental value were more mixed, their generally positive responses do suggest these methods hold value in these domains.
The stakeholders' overall positive response to the questionnaire's learning-related question and their comments about how building influence diagrams made them think deeply about their beliefs suggest direct elicitation was a valuable learning experience. This finding is consistent with existing literature documenting the value of mental models in scientific and environmental education (Fortuin et al., 2011;Kinchin et al., 2000). We believe this provides a clear rationale for the direct elicitation phase of the REA, further excluding the option of performing indirect elicitation alone.

Improving stakeholder response rate to enhanced influence diagrams
Despite the REA's successes, post hoc analysis of its implementation indicated potential areas for improvement. One of the most concerning issues was the low response rate we received when we requested that stakeholders review, revise, approve, and include missing effect strengths in the enhanced versions of their influence diagrams (verification stage in Fig. 1). By providing stakeholders with this opportunity for revision, we intended to reduce misinterpretations of their thoughts made by the analyst following indirect elicitation (Abel et al., 1998). We suspect the low response rate resulted in part from our choice to communicate remotely, via email, which often leads to poorer response rates than face-to-face communication (Kuhnert et al., 2010;Nevalainen et al., 2018). The complexity of the enhanced influence diagrams may have also made the task of assessing them and providing missing information seem daunting and time-consuming. A general lack of commitment to the project may have also been at play if the stakeholders felt burdened by the collaboration process or uncertain about its real-world value and impact (Bracken et al., 2015). The long gap in time between the elicitations sessions and the delivery of the enhanced models to the stakeholders may have also been to blame for their disengagement in the final stages of the REA.
To rectify this issue, we suggest discussing expectations for the elicitation processes thoroughly with prospective participants before beginning the study. We also recommend clearly explaining the importance of the stakeholders' role in the study, its implications for the natural resource they and PM practitioners are mutually invested in, and adhering to a pre-determined schedule for interactions with the stakeholders (Fig. 1). This way they will clearly understand what the study will require of them before committing and be fully aware of the realworld impacts of their involvement. Additionally, we propose a second face-to-face meeting between the stakeholder and facilitator after indirect elicitation to assist with the process of correcting and completing the enhanced diagrams to reduce confusion and frustration with the task Fig. 1).

Coping with complexity, uncertainty, & "messiness"
Another important consideration for future use of the REA is coping with the messiness and uncertainty inherent in describing the cause and effect structure of complex socio-ecological systems. The influence diagrams' messiness, which troubled some stakeholders, could be reduced by creating them digitally during direct elicitation. However, this tactic may reduce the ease and accessibility that drawing by hand provides. Additionally, it is logical that the complexity of a well-informed expert stakeholder's mental model would reflect the complexity of the socioecological system in question. As such, when the goal of elicitation is to preserve the details of stakeholders' complex mental models, as it is in the REA, some degree of messiness is unavoidable.
The discomfort with messiness, however, may have been less to do with the actual organization of the figures and more to do with complexity itself. The complexity of the salmon system adn the effects climate change has on it seemed to be the source of the uncertainty, which caused nearly all the stakeholders to express concern about the validity of their ideas. When stakeholders question their ability to provide accurate information, they should be encouraged to speak freely and contribute their ideas to the best of their ability despite their uncertainty. Since expert knowledge is likely the best source of information when access to more traditional forms of data is unavailable, their opinions are valuable (Kuhnert et al., 2010;Sutherland, 2006).
During his elicitation session, one stakeholder suggested that asking participants only about issues directly related to their fields of expertise may go a step beyond encouragement to help them feel more competent and confident in expressing their ideas. Firstly, he suggested that since climate change is highly complex and predicting its effects on the abiotic environment is generally not directly within the knowledge domain of salmon experts, it may help to provide a specific climate change scenario and subsequent abiotic effects at the outset of the elicitation process. This would allow stakeholders to focus on what they know well: how the abiotic environment affects fish, the biological community salmon are embedded within, and the downstream impacts on the socio-economic system. Taking this idea a step further, he suggested the creation of an aggregated model of the salmon-climate change system developed by combining the knowledge of a variety of experts, each specializing in one portion of the system. Although this may be the best possible way to produce the most thorough depiction of the system, as Burgman (2005) writes, "(modeling is a) balancing act between keeping experts within their domain of knowledge and putting aside sufficient time for the elicitation process." In short, such an effort may require more time and effort than is feasible for practical reasons, like budget restrictions or the timeline of the political decision-making process.

Representing mental models as influence diagrams
In addition to the issues mentioned above, several small improvements should be made to the REA's direct elicitation phase (Fig. 1) based on our experience. Firstly, the mild confusion expressed by some stakeholders about how to draw influence diagrams could be corrected by spending more time teaching them to express their thoughts in this format. Perhaps the facilitator could coach stakeholders to develop this skill through practice with a simpler and more straightforward subject. Facilitators should consider this tactic when stakeholders directly express confusion or ask the facilitator to draw their map without explaining why they prefer this arrangement. The high percentage of stakeholders who preferred to allow the facilitator to draw could be linked with discomfort in visualizing mental models as influence diagrams. Stakeholders must develop enough competence with this process to interject if they feel their views have been misrepresented.
Since there was some confusion surrounding the concept of effects strengths and how to assign them, we suggest describing this more thoroughly during the stakeholder-learning portion of the elicitation sessions (Fig. 1). Additionally, many stakeholders did not assign effect strengths to their models or did not complete the task, presumably since the demanding modeling process exhausted them (Burgman, 2005). This issue could be corrected by incorporating more breaks into the elicitations sessions, breaking it into multiple sessions, or recording effects strengths throughout the diagramming process, rather than leaving the task until the end. Assigning effects strengths at the end of the elicitation sessions may be preferable, however, because stakeholders may fatigue before completing their model's structure, which is likely more detrimental than missing effect strengths.

Coping with biases, beliefs, heuristics, & values
Other considerations for future users of the REA are the biases, beliefs, heuristics, and values (BBHVs) of both participants and modelers, which are inherent in every PM process and shape their outcomes H€ am€ al€ ainen, 2015). Left unchecked, BBHVs can diminish the rigor and credibility of the scientific process . Therefore, PM practitioners must recognize the influence BBHVs exert on their studies and their origins, which are not only the study's participants but also themselves H€ am€ al€ ainen, 2015). To these ends, PM practitioners should introspectively evaluate how their own preferences, values, and motivations may affect a project from its inception to the publication of its results, in addition to how the participants' do. Further, a PM practitioner's BBHVs may affect the participants' behavior (Slotte and H€ am€ al€ ainen, 2015). Therefore, all conscience PM practitioners must acknowledge the problems BBHVs pose and strive to reduce their influence where appropriate (H€ am€ al€ ainen, 2015).
To aid in this pursuit, all PM practitioners should familiarize themselves with the BBHV literature when planning a PM study. Seminal works regarding BBHVs include Kahneman (2011); Kahneman andTversky (2012), 1979;Tversky and Kahneman (1981), 1974. Glynn (2014), 2017Glynn et al. (2017); H€ am€ al€ ainen (2015); Voinov et al. (2016) describe BBHVs in the context of natural resource-related PM. While a robust understanding of this field will assist PM practitioners in anticipating and therefore, mitigating the effects of BBHVs, their consideration should not end with the preparation phase of the REA but should continue throughout the project (Fig. 1).
While conducting elicitations using the REA approach, we addressed the potential effects of BBHVs during all its steps, from Preparation to Verification & Standardization (Fig. 1). During Preparation, we carefully considered the wording of the three problem-framing questions (see section 3.2) and the questionnaire, and later discussed and revised them per the suggestions made by our test participant. We did this to reduce framing biases (Tversky and Kahneman, 1981), which may have caused us to unintentionally lead stakeholders to respond in a manner supporting our own beliefs about the effects of climate change on salmon. We were also careful not to include leading information in our communications with the stakeholders before the study, discussing only the necessary logistics information, our justification for contacting them, and minimally stating the topic we planned to discuss.
Nevertheless, our study's focus on the effects of climate change may have itself introduced a framing bias, implying that climate change does indeed impact these fish. To control for this potential bias, however, we included questions 6-11 in the questionnaire (see appendix), which were intended to gauge whether the stakeholder believed climate change will affect salmon, by how much, and during what time frame. In general, the stakeholders tended to believe that it would affect these fish, although some indicated that climate change was not the most pressing threat facing Baltic salmon.
As described in section 3.2, we conducted direct elicitations sessions one-on-one with each stakeholder to avoid the influences of social norms (Heeren Alexander et al., 2016) and group dynamics . These issues may have otherwise altered the stakeholders' depictions of their mental models, via, for example, groupthink (Janis, 1982;McCauley, 1989), or lead to issues like the overrepresentation of the most influential stakeholder's ideas (Burgman, 2005;Martin et al., 2012). Although conducting elicitations individually may limit problem-solving potential or diminish the benefits a group setting offers, like strengthening relationships between stakeholders, we believe our strategy was justifiable given our aims. Our problem-framing study and hence, the REA, were intended to build as holistic a view of a complex system as possible. As such, we were uncomfortable with the limits group interactions may have imposed. Nevertheless, the REA, as with all PM methodologies, should be adaptable depending on the circumstances and aim of individual PM processes (Voinov et al., 2016). Therefore, we acknowledge that the REA could be conducted in a group setting if desired, following the same procedure we have described here.
During Direct Elicitation, the facilitator's BBHVs may also affect stakeholders' influence diagrams. When drawing, a facilitator may depict a skewed interpretation of the stakeholder's statements, which instead reflect their own BBHVs. To reduce this effect in our study, the facilitator ensured the stakeholder could see what she was drawing, consistently asked for confirmation of her interpretation's accuracy, and encouraged interjection at the first sign of departure from the stakeholder's ideas (see section 3.2.2). However, if a stakeholder is uncertain or uncomfortable with the process of drawing causal diagrams, their ability and confidence to interject may be jeopardized. For this reason, the facilitator was as thorough as possible when explaining causal diagrams, how to read them, and how to produce them during the Stakeholder Learning portion of Direct Elicitation. For future implementations of the REA, we would add that if a stakeholder seems unsure about how to proceed or asks the facilitator to draw, the facilitator should ask the stakeholder to briefly draw a sample causal diagram and explain its contents before moving on. Additionally, when confirming the accuracy of the diagram with a stakeholder, the facilitator should verbally describe what each portion of the diagram depicts, indicating her position within the diagram as she describes it.
The facilitator's behavior or speech may also subtly influence the visualization of a participant's mental model. To reduce this influence in our study, the facilitator made an effort to remain encouraging of the participant's progress drawing or describing their mental model, while remaining neutral about its contents. Instead, she made comments intended to request clarification, further explanation, confirm the model's accuracy.
Before conducting our elicitations, we were keenly aware of the problem inherent in indirect elicitation: without the stakeholder's immediate guidance, a PM practitioner may inject their own BBHVs as they attempt to reproduce the stakeholder's mental model from textual information (see introduction). For this reason, we conducted direct elicitation first. It was imperative to ensure, in real-time, that each stakeholder's influence diagram was drawn according to their internal mental model. During Indirect Elicitation, careful coding and notetaking before enhancing the diagrams served to limit the analyst's interpretation and provide a trail of justification for each change made. Lastly, we incorporated the Verification Stage, during which that stakeholder must approve of the changes made by the PM practitioner. During this phase, the stakeholder also has the chance to make changes to the model or revoke permission for its use, providing one further check on the PM practitioner's interpretation. However, we discussed the difficulties we encountered during this stage in section 5.3.1.
Despite these safeguards, we suspect some of our actions during direct elicitation may have unintentionally biased the stakeholders' influence diagrams. Although we initially thought the core biological model for herring (see Fig. A1) was general enough not to trigger anchoring bias (Oppenheimer et al., 2008;Tversky and Kahneman, 1974) when used as an example during the Stakeholder Learning (Fig. 1), we cannot rule out the possibility that it did. Additionally, based on the stakeholders' disinterest in it, we believe it was unnecessary as a tool for idea generation. Explaining the core biological model also took valuable time, which could have been better spent practicing the process of building influence diagrams or on the elicitation process itself. Due to its lack of utility and its potential to induce anchoring bias, we recommend only exposing participants to example diagrams entirely unrelated to the subject at hand during the Stakeholder Learning.
We are also concerned that the stakeholders' knowledge about the facilitator's professional background may have influenced the study. Stakeholders often asked about the facilitator's career background and educational history while building rapport at the beginning of their elicitations sessions. Later, some seemed to tailor their descriptions of their mental models accordingly. After learning of the facilitator's experience with fisheries science, stakeholders tended to leave out the causal relationships within their mental model they assumed were common knowledge between them. For example, the stages of the salmon life cycle. To adapt to this situation in real-time, during our study, the facilitator frequently requested that the stakeholder be explicit about the causal relationships within their mental model, explain as if to a non-expert, and clarify portions of their model where they appeared to assume something about the facilitator's prior knowledge. We recommend that other PM facilitators follow this practice as well. Further, the facilitator should not discuss their background with stakeholders, although any interested stakeholder could undoubtedly find this information online before meeting the facilitator.
Despite the potential for a PM practitioner's BBHVs to affect the visualization of stakeholders' mental models, the REA coupled with our advice for its implementation and perhaps, facilitation training (Voinov et al., 2016), can address many of these issues. Further, we reiterate that by coupling direct and indirect elicitation, the REA reduces the potential introduction of an analyst's BBHVs, while maintaining the thoroughness indirect elicitation provides. Additionally, questionnaires implemented after both direct and indirect elicitation (Fig. 1) should include questions to gauge the participants' understanding of and satisfaction with these processes. This information should be used to improve future facilitation of the REA and to develop the methodology over time. As such, the REA provides a cohesive methodology, which acknowledges and reduces the effects of BBHVs. We believe this is a step forward for mental model elicitation in PM. Nevertheless, PM practitioners cannot control for their influence completely and therefore, in addition to learning from past facilitations and endeavoring to reduce their influence in each subsequent facilitation, PM practitioners should thoughtfully and honestly report how BBHVs may have influenced their study.

Formalizing core principles & process
In addition to the methodological recommendations we have provided here, we also recommend that all PM processes including the REA, be governed by a set of thoughtfully developed core principles, agreed upon by all the actors involved (Voinov et al., 2016). Although by necessity these principles will differ between PM processes, owing to the vastly different circumstances under which they are conducted, their aim should be to establish norms, or "rules of engagement" for each PM process and to ensure it is conducted in an effective manner and that results contribute meaningfully to knowledge co-production and decision-making (Voinov et al., 2016). These principles should be based on ensuring transparency, accountability, and follow-through between PM practitioners, participants, and the end-users of the PM process's products. Additionally, they should act as a code of conduct between these groups, ensuring fairness, civility, and fostering trust between them, while reducing the influence of BBHVs Voinov et al., 2016). An example of such core principles is available in Voinov et al., (2016). To ensure the PM process adheres to these principles, a process for appropriately documenting and reporting the PM process is essential. Although no standard process has yet been identified, Gray et al. (2018)'s 4P framework, the records of engagement suggested by , or both could help fulfill this need.
Although we discussed the general aims of our study with the stakeholders, how their contributions would be used and made a commitment to inform them about how the study results were ultimately employed, we believe a more formalized process for discussing and agreeing on the study process and its core principles would have been warranted (Fig. 1, Principles & Process Discussion). A more thorough discussion about what to expect from the PM process, about the aims of the study, and more contact with the stakeholders throughout may have improved their commitment to it, potentially encouraging more engagement during the model verification stage of the REA. In the future, discussing the REA process itself with stakeholders may prepare them better for elicitation and emphasize the importance of their continued involvement in the study after direct elicitation. Additionally, agreement on the core principles of the project may have given the stakeholders a stronger voice in shaping the study and its outcomes into more relevant and mutually beneficial, real-world advice for salmon management in the face of climate change. A greater degree of connection, transparency, and shared control over the project may improve the fulfillment of the instrumental and normative PM functions as well.
It is also worth noting that PM is typically a sub-process within a larger scientific and decision-making efforts to govern socio-ecological systems. While we have discussed the importance of defining core principles and processes in the context of PM specifically, these ideas also apply to the overarching processes they are embedded within. See Glynn et al. (2018) and Glynn et al. (2017) for discussion of these ideas in the broader scientific and decision-making context.

Analyzing rich mental models & problem-framing
Although the purpose of this article is to describe the REA, not discuss the results of our problem-framing study, we find it pertinent to exemplify how one can use rich depictions of mental models. As stated previously, the purpose of collecting rich influence diagrams was to frame the problem of the impacts of climate change on Baltic salmon. This study's results are intended to guide the incorporation of climate change effects into ICES' existing Baltic salmon stock assessment model by indicating areas of special concern that the model should take into account and prepare for future management challenges. We suggest the number of times the stakeholders included a particular variable, category of variables, life stage, or habitat in their causal diagrams can serve as a proxy for their importance within the salmon-climate change system. Therefore, more frequently identified variables should be prioritized for inclusion in the model.
We recognize there are disadvantages to prioritization based on frequency alone, as this technique is not well suited for capturing important causal pathways or "themes." For example, we may be able to detect that river temperature, age at smoltification, and the number of salmon occur frequently in stakeholders' influence diagrams. However, by considering their frequency alone, we may miss the concept they represent together. A stakeholder may have used these variables to indicate that changes in river temperature alter the time it takes for young salmon to leave the river, altering the age at maturity, resulting in changes in population size. Stakeholders may describe the same process in vastly different terms and different levels of specificity, further complicating the task of identifying themes. Therefore, we suggest it is imperative to couple this analysis strategy with analysis of elicitation notes (A) and (B) (Fig. 1), which describe the stakeholder's thoughts, predictions, and common themes.
Alternatively, visualization techniques, like fuzzy cognitive mapping (FCM) (Olazabal et al., 2018;€ Ozesmi and € Ozesmi, 2004;Solana-Guti� errez et al., 2017), may help to more smoothly aggregate stakeholders' depictions of their mental models, making identification of frequently described causal pathways (themes) easier. FCMs can also be deconstructed to explore causal relationships pertaining to particular variables of interest (Olazabal et al., 2018). Creating FMCs also requires mental model elicitation and therefore, the REA approach could be applied to this methodology as well. Additionally, we recognize the number of times a variable is mentioned may also be indicative of the limits of stakeholders' knowledge, with well-understood variables and causal pathways occurring more frequently, than potentially poorly understood, but highly instrumental ones. Nevertheless, the inclusion of frequently cited variables in ICES' stock assessment model should reflect the best available knowledge, providing a starting point for further work, and promoting the model's legitimacy in the eyes of the stakeholders and hopefully, in their peers. ' If desired, after the decisions about which aspects of the salmonclimate change system to include in the stock assessment model have been made, a second PM process, similar to "task 1" described in Haapasaari et al. (2013), can begin. As in their study, we propose that stakeholders build directed acyclic graphs (DAG), explaining how the variables identified during the elicitation sessions connect with the current stock assessment model and with climate change. Unlike the qualitative influence diagrams we produced for problem-framing, the strengths of the effects between variables in DAGs should be quantified, expressed as joint probability distributions (Jensen and Nielsen, 2007). Whether a causal link is positive or negative should also be noted. In their study (M€ antyniemi et al., 2013) 2 , instructed stakeholders to choose variables to include in their DAGs from a previously collected list provided by herring experts. In our case, these variables would come from the influence diagrams developed during the elicitation sessions described here. Once completed, the DAGs can be pooled via Bayesian model averaging (M€ antyniemi et al., 2013) and subsequently incorporated into the stock assessment model. Fully quantified influence diagrams also enable Value of Information (VoI) analysis (M€ antyniemi et al., 2009), which measures the maximum amount a decision-maker should be willing to pay to obtain more precise information before making a decision. As such, VoI can help determine what data to collect to assist decision-making.
In addition to the utility they provide in incorporating new information into environmental models, rich depictions of mental models provide context and justification for doing so. Their development also serves as a brainstorming tool, identifying the goals stakeholders have for a particular natural resource, strategies to reach those goals, areas where knowledge is lacking, and providing new, testable hypotheses about how a system works. Beyond the results of the study themselves, the process of conducting mental model elicitation promotes stakeholder learning and causal thinking (Fortuin et al., 2011;Kinchin et al., 2000;Marcot et al., 2001;Uusitalo, 2007) and begins to build stakeholder support for management recommendations (Fiorino, 1990;Jones et al., 2009).

Future directions
We suggest future users of the REA implement the suggestions for methodological improvement discussed in section 5.3 and that targeted questionnaires administered following Direct Elicitation and the completion of the full REA process could develop the methodology further ( Fig. 1, Final Questionnaire). Additionally, we encourage further advancements to ensure mental model elicitation fulfills all functions of PM effectively.
Beyond the practical advice and suggestions we provide here, we also believe PM practitioners should strive to reduce the loss of valuable stakeholder knowledge even further. The most pressing barrier preventing the acquisition of holistic stakeholder knowledge in PM is limited participation, since typical studies include 30 or fewer participants (Voinov and Bousquet, 2010). Eliciting the beliefs of such a small segment of the stakeholder population constitutes its own form of knowledge simplification and potentially reduces PM's substantive, normative, instrumental, and educational value. These low numbers may be the result of limited time, financial, and analytical resources, or arise from the difficulty in securing stakeholder participation (Nevalainen et al., 2018). These obstacles may be circumvented by making the PM process easier, more enjoyable, or by connecting it more clearly with real-world impact. Improving communication styles and channels between stakeholders and researchers may also help. Traditionally, stakeholder participation has been increased by conducting group PM activities, like group problem-framings, but eliciting stakeholder knowledge in groups presents drawbacks as well (see sections 3.2 and 5.3.4). Still, group participation may only marginally increase the number of stakeholders involved in PM processes. Less traditionally, text analysis techniques, like topic modelling (Blei, 2012) from the computational social sciences literature, could provide analytical tools for extracting important insights from text (interview or elicitation session transcripts, questionnaires, tweets, blogs, etc.) provided by a number of stakeholders orders of magnitude greater than more traditional means allow. However, employing these techniques could mean less face-to-face time between researchers and stakeholders, limited verifiability of their ideas, and less opportunity for learning. Additionally, the time-consuming steps, like coding and note-taking, involved in traditional PM studies force the analyst to engage deeply with the data, improving their depth of knowledge and ability to draw conclusions from the data. As such, alternative means for incorporating more stakeholders into the PM process should be developed.

Conclusions
In conclusion, the Rich Elicitation Approach (REA) presented here is a strategy for eliciting rich mental models from expert stakeholders. The approach's novelty comes from the deliberate combination of direct and indirect elicitation strategies to ensure stakeholders' mental models are represented as holistically as possible while preserving the integrity of their knowledge and the learning process inherent in direct elicitation. We believe this approach can and should be adapted and applied to any PM process involving the elicitation of mental models.
Though the rich mental models the REA produces are timeconsuming to create and complex, making their analysis more challenging, we believe they are necessary for forming a strong knowledge base on which to begin any PM project in the face of data limitation and incomplete scientific knowledge. Greater detail allows researchers to take advantage of the full depth of stakeholder knowledge, reducing the propensity to overlook potentially instrumental causal pathways, promoting the production of new testable hypotheses, improving realism, and generating more thorough solutions and management strategies. Developing a comprehensive knowledgebase may also improve the resilience of PM projects, by providing a reservoir of new ideas when researchers find themselves going "back to the drawing board." Despite the challenges, the best available expert knowledge should be used to its fullest potential to solve natural resource-related challenges. We also encourage researchers and PM practitioners to transparently and thoughtfully share their experiences to allow for the development of best practices and to bring PM into the light as a legitimate and valuable field of its own.

Funding
This study received funding from the University of Helsinki's Doctoral Program in Interdisciplinary Environmental Sciences (DENVI) and the Finnish National Agency for Education.