Collaborative modelling or participatory modelling? A framework for water resources management

Decision Support Systems, and, more recently, participatory and collaborative modelling have emerged as a response to increased focus on stakeholder participation in modelling activities for certain ﬁ elds like water resources management. Researchers and practitioners frequently use ‘ buzzwords ’ such as ‘ participatory modelling ’ and ‘ collaborative modelling ’ . In some cases, both terms are used inter-changeably, largely due to unclear distinction between them in literature. This article draws the line between participatory and collaborative modelling by using levels of participation and cooperation as conditioning dimensions. Based on this methodology, a new generic framework is presented. This framework can help identify determinant features of both modelling approaches currently used in water resources management. It permits analysis of these approaches in terms of context, speci ﬁ c use, infor- mation handling, stakeholder involvement, modelling team and means. The article concludes with an application of the framework to a collaborative modelling approach carried out for a groundwater study in the Netherlands.


Introduction
Over recent decades Water Resources Management (WRM) has experienced a significant transformation. The top-down, monodisciplinary and single sector managerial and planning approach was reformulated into Integrated Water Resources Management (IWRM) (GWP, 2000). IWRM is a bottom-up, demand-oriented approach based on multi-disciplinary activities. It has paved the way for stakeholder participation in planning and decision making processes (Rees, 1998). In particular, IWRM principles (known as Dublin Principles) have served as a turning point for public participation in WRM decision making processes (GWP, 2000). Ever since their declaration in 1992, stakeholder participation has become increasingly institutionalized in legislation like the EU Water Framework Directive (Directive, 2000/60/EC) and in global WRM frameworks and guidelines (GWP-ToolBox; Pegram et al., 2013;UNESCO, 2009). As the integrated approach to water management is widely accepted, the terms IWRM and WRM are often used interchangeably, also in this article.
A wide variety of participatory approaches and methods for participatory planning and decision making in WRM have been developed in response to the prominence of public participation in IWRM. Focus groups (Dürrenberger et al., 1997;Gearin and Kahle, 2001), the Delphi method (Linstone H. and Turoff M. (Ed), 2002), citizen panels (Armour, 1995), World Caf e (Brown, 2002), and Participatory Rural Appraisal (PRA) (Chambers, 1994;Mukherjee, 1993) among other forms are being used to increase stakeholder participation in decision making (Bousset et al., 2005). Much research has been oriented towards engaging stakeholders in planning and decision making processes. Much less scientific research has been undertaken for exploring the use of conventional computer-based models within these participatory planning and decision making processes. The development of Decision Support Systems (DSSs) emerged as a means to address this gap. However, in many cases DSSs were not used by stakeholders and decision makers after their development. This was due to a variety of reasons, primarily associated with the different knowledge and expertise of the developers of such systems and the diverse stakeholders as intended users. Participatory modelling approaches then started to be conceived to strengthen stakeholder ownership of DSSs and modelling tools by increasing stakeholder involvement in the actual modelling process. Although stakeholder participation cannot be considered as the unique pre-requisite for guaranteeing long term use of computer-based models, it can be a critical factor. Consequently, today there are various participatory modelling approaches being used worldwide. Some refer to these approaches as participatory modelling, whilst others employ the term collaborative modelling. Although certain differences between the two terms may be identified, their inherent similarities can result in them being used interchangeably. This is in large part due to unclear distinction having been made between them in the literature. This makes it difficult for researchers, practitioners and policy makers to identify which participatory or collaborative modelling approach is best suited to each type of decision making and related processes (Bots and van Daalen, 2008;Hare, 2011;Serrat-Capdevila et al., 2011). To support such identification, a new framework for evaluating participatory and collaborative modelling approaches in WRM has been developed and is presented in this article.
Some may question the value of yet another 'framework' given that others have previously been developed. For example, several evaluation frameworks have been developed for assessing participatory processes (Abelson et al., 2003;Rowe and Frewer, 2004). Similarly, evaluation frameworks and protocols for participatory and collaborative modelling approaches have been variously developed to assess the value of these approaches and their outcomes. For instance, Smajgl and Ward (2015) present an evaluation protocol based on the Challenge and Reconstruct Learning (ChaRL) Framework to assess the learning process of decision makers. Jones et al. (2009) developed the Protocol of Canberra to evaluate the influence of tools on the sharing of information among participants, their relations between each other and the outcomes of the participatory process. This was also even based on an earlier participatory modelling evaluation initiative (HarmoniCOP) developed by Mostert et al. (2007a). Plus, other scholars have developed frameworks to compare case-specific participatory modelling processes, such as the Comparison of Participatory Processes (COPP) framework (Hassenforder et al., 2015). The generic framework presented in this article differs from all of these other existing frameworks as it distinguishes between the key characteristics and features of both "participatory modelling" and "collaborative modelling" approaches based on 20 relevant parameters for WRM. This helps to categorize existing approaches and corresponding tools into one of the two generalized terms via a consideration of their generic characteristics and features (trade-offs).
The focus of this research is to both distinguish between and highlight the importance of participatory and collaborative modelling approaches in the field of WRM. For this, we first describe background information necessary to understand this research, including definitions and typologies. In Section 3, we propose four pillars of both modelling approaches. Based on this, and taking levels of participation and cooperation as the critical conditioning dimensions, we make a classification of participatory and collaborative modelling approaches in Section 4. Finally we describe the new generic framework to help categorize existing approaches into "participatory modelling" or "collaborative modelling" based on their key characteristics and features (Section 5). This framework is then applied to evaluate interactive modelling though a collaborative groundwater modelling study in the Netherlands (Section 6). The article concludes with a general discussion on the suitability of the framework and future research directions.

Challenges of decision support systems for IWRM
The IWRM process aims to strike a balance between using currently available water and land resources for socio-economic purposes and protecting them in such a way that they can also be used in the future (GWP, 2000). Within this framework, the development of DSSs has served as a major initiative targeted towards bridging the gap between the development and use of computer-based models with stakeholders and how the planning and decision making processes are actually carried out (Alter, 1980;Georgakakos, 2007;Giupponi and Sgobbi, 2008;Jolk et al., 2010;Keen, 1987;Loucks and da Costa, 2013;Serrat-Capdevila et al., 2011;Sharda et al., 1988;Soncini-Sessa et al., 1991;Thiessen and Loucks, 1992;Walsh, 1993;Zindler et al., 2012). However, in many instances these initiatives have not been sufficient, with the DSSs not actually used by stakeholders and decision makers. Extensive research has been carried out to identify the main challenges of the use of DSSs in WRM planning and decision making. These are: 1) The key points of a planning and decision making process are the objectives and criteria. DSSs need to focus on the goals the decision maker and stakeholders wish to achieve, which might differ depending on the decision making process and might evolve over time (Bousset et al., 2005;Medema et al., 2008;Mintzberg, 1978); 2) Most DSSs focus on the tool to be developed rather than on their participatory use by or with stakeholders and decision makers.
The main focus is often on the software structure, the user interface and the visualization capacities. Less emphasis is placed on stakeholder-model interactions or the specific conditions that makes the use of models being more effective (Refsgaard et al., 2005;Serrat-Capdevila et al., 2011); 3) The use of DSSs in decision making processes often demands that the modeller remains a central part of the process. Consequently, these models are commonly perceived as 'black boxes'. They are often developed and implemented in the back-room, even in those instances when there is interactive work done during data collection and results are shown and discussed with stakeholders (Bourget L. (Ed.), 2011; Loucks et al., 2005).

Definitions and typologies of participatory and collaborative modelling
'Participatory modelling' and later 'collaborative modelling' emerged as possible solutions to address certain challenges encountered with traditional DSSs. At the core level, both generic sets of approaches emphasize the importance of involving stakeholders in a modelling process (Voinov and Bousquet, 2010) Stakeholders should be exposed to the same information and problems encountered during the modelling process (Castelletti and Soncini-Sessa, 2007). Various scholars have built upon this basic definition; for instance, by distinguishing stakeholder involvement in various modelling stages (Hare, 2011), by specifying the stakeholder groups to be involved (Voinov and Gaddis, 2008), or by emphasizing the importance of communication activities and visualization tools (Evers et al., 2012).
Specific types of participatory and collaborative modelling have emerged in the last few decades. Some are extensively used for WRM, whilst others are emerging approaches. The most frequently applied approaches in WRM are listed in Table 1. As Voinov and Bousquet (2010) highlight, these approaches share several similarities; however, a number of subtle differences also exist. These mainly refer to their applicable contexts, specific uses, information handling, stakeholder involvement, modelling/organizing teams and/or means. Table 1 groups these approaches under a number of overarching main lines or umbrella terms identified from the literature.

Key pillars of participatory and collaborative modelling
In this article we propose that participatory and collaborative modelling for policy analysis in WRM rests upon the integration of four key pillars: (i) water resources planning, (ii) informed decision making by means of computer-based models, (iii) stakeholder participation, and (iv) negotiation (Fig. 1). Stakeholder cooperation in collaborative modelling will generally be greater than in participatory modelling, leading to the increased importance of negotiation within the process. These inter-linked aspects are considered the basis for effective and sustainable WRM.

Water resources planning
The planning and management of water resources has always been an important vehicle for development. A central challenge for sustainable development is how to balance the many competing uses and users of water, to ensure the needs of all are met while maintaining healthy and diverse ecosystems; in other words, to achieve water security. IWRM has been identified as the vehicle by which to achieve water security (Van Beek and Arriens, 2014). Taken together, they represent both the ultimate objective and the process by which it is attained. Hence, the water resources planning pillar encompasses these two components (Fig. 1).
IWRM demands that solutions are found to complex problems that incorporate various environmental, economic and social  Richardson and Andersen (1995); Vennix (1996Vennix ( , 1999 dimensions (GWP, 2000). Commonly there is no single optimal solution to these complex, messy problems (Vennix, 1999). Participatory and collaborative modelling help characterize the relationship between the process of planning and decision making and the resultant environmental, economic and social impacts of concern to stakeholders. Problem complexity is one of the factors that can determine whether to include participatory and/or collaborative modelling in a planning approach. The structure of policy problems in general (Simon, 1977) is determined by the degree of cooperation and conflict among stakeholders (Douglas and Wildavsky, 1983;Zeitoun and Mirumachi, 2008) and the level of knowledge uncertainty (Hommes, 2008;Van de Graaf and Hoppe, 1996). On this basis, three types of problems can be distinguished (Table 2): (i) structured problems, for which a high level of scientific certainty exists and there is a high degree of consensus among stakeholders; (ii) semi-structured problems, which can be the result of either (i) low degree of consensus (regarding values, norms and standards, beliefs and ambitions) in combination with some certainty about the scientific knowledge, or (ii) the knowledge of the system is limited in combination with consensus among stakeholders; (iii) unstructured problems, for which a low degree of consensus exists and there is a lack of scientific certainty.
Many problems faced in water resources planning can be classified as being either semi-structured or unstructured. This is due to the complexity inherent to both natural and built water systems, as well as the fact that water is a shared resource for many different socio-economic and subsistence functions (leading to many diverse stakeholders). Water resources planning and its implementation occur at different scales and time horizons to the majority of functions they support. The IWRM planning cycle is the common mechanism with which to structure the planning process towards achieving water security. It includes a logical sequence of phases driven and supported by continuous management and promotion (Fig. 2). Participatory and collaborative modelling approach(es) must be flexible to facilitate stakeholder engagement during all the planning phases of the cycle and to allow the complexity associated with IWRM to be adequately addressed.

Computer-based models for informed decision making.
Modelling tools are central to collaborative modelling processes. Modellers and technical analysts develop, enhance, and validate these tools via a collaborative process for the purpose of informed decision making. Models must be both understood and trusted by the stakeholders and decision makers involved.
In recent decades, there has been a trend to develop computerbased models to improve understanding of water resource systems, to provide more integrated assessments and to better account for uncertainties (Haasnoot et al., 2014;Jakeman and Letcher, 2003;Loucks et al., 2005;Refsgaard et al., 2005). These models support evidence-based stakeholder dialogues and help focus and enhance the scientific basis of informed decision making (Loucks et al., 2005). DSSs are intended to communicate the necessary information and render modelling outputs understandable, transparent, acceptable and time appropriate for stakeholders (Bourget L. (Ed.), 2011; Jonoski and Evers, 2013). Different types of DSSs are depicted in Table 3. Depending on the type of problem to be addressed, as well as the stakeholders involved, DSSs can range from minimal if any computer-based model use (case 1 in Table 3) to DSSs that are fully automated (case 6 in Table 3). A clear example of automated decision making is the automatic closing of the flood gates in Rotterdam harbour, where no human involvement is present (Loucks et al., 2005). In many DSS, Geographical Information Systems (GIS) and databases (DB) are used for data provision. Computer-based models can then support analysis of this data, generation of possible options as well as support decision makers and stakeholders in evidence-based strategy making. Computerbased models can also be useful tools to assist stakeholders reach a common understanding and consensus regarding any conflicting interests, values, or norms. This is because they generally provide neutral information about the functioning of the system.
Many different types of software platforms can be used in participatory and collaborative modelling approaches. For instance, one could develop conceptual diagrams using system dynamics software packages to help understand system relationships.
Alternatively, one could develop narratives using fuzzy cognitive mapping approaches. Or, one could use Open Street Maps (OSM) together with local communities and technical analysts to provide feedback on the available or necessary data, models and possibilities. In many cases, several models may be coupled dynamically or using generalized functional relationships through a simple interface (e.g. as for the development of Fast Integrated Systems Models; Haasnoot et al., 2014).
Stakeholder involvement during one or more stages of the modelling process is critical in participatory and collaborative modelling approaches. Wherever possible, stakeholders should be directly involved in the construction of the models and tools, the formulation of scenarios and policy options to be modelled, as well as during assessments of the efficacy of these options against the key performance criteria (which they will have also defined). To enable such involvement, any developed models and tools should be constructed, where possible, using open source or freeware software that can facilitate their distribution to and use by the stakeholder population. Furthermore, sufficient attention should be placed on the visualization and communication capabilities of these tools to facilitate the transmission of information to lesstechnically minded stakeholders. As such, participatory and collaborative modelling can encompass the development and use of various computer-based models and analytical tools, communication and visualization tools, in addition to mental and cultural models (Jones et al., 2011;Paolisso, 2002).

Stakeholder participation
It is generally accepted that stakeholder participation in WRM can serve as a tool for achieving sustainable WRM (Abbott and Jonoski, 2001;Edelenbos and Klijn, 2006). Stakeholder participation is both a means and an end, insofar as it can lead to increased stakeholder empowerment and make the planning and decision making process more transparent and democratic (Hare et al., 2003). Participation is also a process that enhances the capacity of individuals to improve their own lives and that facilitates social Table 3 Types of decision support systems (adapted from: Loucks et al., 2005). change (Cleaver, 1999). Through building trust, ownership, and consensus the legitimacy and stakeholder support of the planning process and its outputs are increased. Local knowledge and expertise can be a valuable tool for understanding local situations and contexts, planning objectives and policy measures, as well as improving and/or creating innovative and alternative strategies. As a result, the sustainability of the adopted policy strategy will generally be higher.
Stakeholder participation can also promote collaborative learning. Two variants of collaborative learning are distinguished: social learning and shared learning. Social learning is the process where stakeholders acquire knowledge and collective skills through better understanding the system and its complexity; the perceptions, concerns and interests of other stakeholders; and on this basis the inter-connection between physical processes and social dynamics (Evers et al., 2012;Hare, 2011;Voinov and Bousquet, 2010). In shared learning, also referred to as colearning, information flows occur in all directions. This means, information and knowledge flows from the organizing team, including researchers and modellers, to stakeholders, and vice versa (Voinov and Bousquet, 2010). In collaborative learning individual knowledge is increased within the social context, further assisting the acquisition of collective skills (Hare, 2011;Mostert et al., 2007b;Pahl-Wostl et al., 2007;Voinov and Bousquet, 2010).

Stakeholder engagement process: from stakeholder analysis to levels of participation
The effectiveness of a participatory process is heavily influenced by the specific characteristics, interests, concerns and needs of the stakeholder groups involved. As Voinov et al. (2016) stress, we need to consider not only the willingness of stakeholders to participate, but also how other powerful stakeholders might allow, facilitate or encourage the involvement of other stakeholders. Alternatively, they could prevent their participation. At the beginning of a participatory and/or collaborative modelling process it is always recommended to perform a stakeholder analysis. This is a useful tool to analyse stakeholder roles, responsibilities, interests, perceptions, concerns and dependencies (Grimble and Chan, 1995). The stakeholder community can then be later divided into various stakeholder groups to improve process efficiency if required. Common approaches include the Circles of Influence, the Nested approach or bull's eye approach used in the Water Framework Directive (Bourget L. (Ed.), 2011;European Communities, 2003;Lamers et al., 2010;Werick, 1997). This then leads to the next challenge: the definition of stakeholder roles to systematize planning and decision making processes.
In participatory and collaborative modelling it is important to find ways in which each stakeholder group can participate effectively. There exist many different roles that stakeholders may take in a planning and decision support process. Defining these roles according to the IWRM planning cycle and related modelling phases may be beneficial (Fig. 2). Naturally, these choices will be based upon the goals of the specific water resources planning process. It may also be necessary to involve different stakeholders at different levels of participation. Arnstein (1969) provides useful insight into stakeholder participation by describing a ladder of participation related to power sharing. This varies from nonparticipation to citizen power processes such as partnership, delegated power and citizen control. Based on this, Bruns (2003) proposes an extended ladder of participation, ranging from low levels of participation such as informing, consulting and involving to higher levels such as establishing autonomy, advising and enabling. Similarly, Mostert (2003) identifies six main levels of stakeholder participation in water policy. These are information, consultation, discussion, co-designing, co-decision making and independent decision-making. These three ladders of participation have oriented the development of a simple typology of participation levels for planning and management of water resources. As illustrated in Fig. 3, the revised ladder of participation includes one level of non-participation (i.e. ignorance), three levels of low participation (awareness, information and consultation) and three levels of high participation (discussion, co-design and co-decision making).
The organization of stakeholder engagement according to varying levels of participation can extend involvement to those stakeholders affected by decisions, but who may not be able to actively collaborate in planning and decision making processes due to their characteristics, interests and/or capabilities. The use of participatory and decentralized tools such as social media can be an effective mechanism in this regard as they allow for the collection and provision of data that is both geographically and temporally traceable (Wendling et al., 2013).
In structuring stakeholder engagement in this way, we address a major challenge for stakeholder participation: launching and Fig. 3. Ladder of participation for water resources planning and management (adapted from: Arnstein, 1969;Bruns, 2003;Mostert, 2003). maintaining the participatory decision making process (Almoradie et al., 2015).
When combined with the use of modelling and analytical tools, effective stakeholder participation can foster consensus among competing organizations. It opens channels of communication via evidence-based stakeholder dialogues that generate mutual understanding and negotiated solutions (Hare, 2011;Loucks et al., 2005). In doing so, it leads us to the final pillar of participatory modelling: negotiation.

Negotiation
A decision making process concerning water resources typically involves complex problems that incorporate disputes among the stakeholders involved. Depending on the context and the structure of the problem, the willingness of the involved stakeholders to cooperate in joint decision making might differ (i.e. competitive or cooperative interaction context). Stakeholder participation in WRM inevitably involves cooperation and conflict management that is achieved through negotiation. Different types of cooperation can be used to assist stakeholders' transition from dispute to integration. Sadoff and Grey's (2005) cooperation continuum, illustrated in Fig. 4, is a useful tool for differentiating four principal types of cooperation: unilateral action, coordination, collaboration and joint action. Sadoff and Grey use this continuum to focus on transboundary cooperation in international rivers. In this article this typology is adapted and applied to the concepts and contexts of participatory and collaborative modelling.
Unilateral action occurs when stakeholders work in an independent and non-transparent way. There is no cooperation as there is little or no communication or information sharing between the organizing or modelling team and interested stakeholders.
Coordination is reached when there is regular communication and information exchange between the organizing or modelling team and interested stakeholders. The exchange of information (e.g. collection of data) helps the organizing or modelling team in the planning process. The coordination between sectors and governance levels helps to avoid conflicting ideas or initiatives, as the team can assess the possible benefits and impacts.
Collaboration is achieved when collective learning occurs and when the ideas and initiatives of stakeholders are adapted to achieve mutual benefits. This implies they are adapted to either secure mutual gains or to mitigate harm being caused to other stakeholders.
Joint action results when the organizing and modelling team act as partners with other key organizations in the planning and decision making process. This level of cooperation is generally formalized by legal agreements. Joint ownership, institutions and/ or investments are some of the greatest cooperative efforts that can be achieved.
Negotiation processes within participatory and collaborative modelling can enhance capacity development for the stakeholders involved via individual et al., 2012 Negotiation processes within participatory and collaborative modelling can enhance capacity development for the stakeholders involved via individual and collaborative learning (Evers et al., 2012;Hare, 2011;Voinov and Bousquet, 2010).

Participation and cooperation as critical dimensions for stakeholder involvement
The involvement of stakeholders in WRM planning processes is not a simple and straightforward process. Rather, it is a complex, interactive and iterative process to achieve certain specific objectives. Fig. 5 organises the possible involvement of different stakeholder groups in a planning and decision making process according to the four types of cooperation. Potential stakeholders have been labelled as either key stakeholders, other interested stakeholders and disinterested stakeholders. These are each distinguished by different grey tones (see legend). Commonly, the organizing team would be responsible for grouping stakeholders according to the local context and conditions via stakeholder analysis (Grimble and Chan, 1995).
Four main cases of stakeholder involvement in participatory and collaborative modelling have been identified according to two critical dimensions: participation and cooperation. That is, the four cases are defined in relation to the seven levels of participation ( Fig. 3; vertical axis) and the four types of cooperation ( Fig. 4; horizontal axis). It is important to highlight that each of the cases can relate to the timing of participation in the modelling process (Table 4), and can change over the course of the participatory or collaborative modelling process.
Case 1: Unilateral action implies low levels of participation. Key stakeholders might be informed about the planning and/or decision making process, however, they are not able to actively participate. Other stakeholders that may be interested in the process are either aware due to other information channels or else are completely unaware.
Case 2: Following the IWRM approach, decision makers and the organizing and modelling team agree to coordinate with key stakeholders in the planning process. These stakeholders might participate in stakeholder consultation meetings and discussions. In some instances, they can even be partly involved in the co-design of the modelling process and modelling tools. Other interested stakeholders can attend public meetings where they are informed about the planning process and the decisions taken. Social media can be used for engaging any disinterested stakeholders.
Case 3: Here, collaboration is considered crucial for the sustainability of WRM, and there is a willingness to actively involve key and other interested stakeholders in the planning process. The design of the planning and decision making process is carried out jointly with key stakeholders, as is the construction of the computer-based model. They may also be involved in discussions depending on the timing of participation in the modelling process. Other interested stakeholders can participate in discussions (although their concerns and ideas may not end up determining outcomes), be consulted (e.g. attend public consultation meetings, provide information and data, etc.) or be informed. The use of social media is encouraged for the engagement of any disinterested stakeholders.
Case 4: This mainly differs from case 3 in terms of the high levels of participation of the key stakeholders. They are not only encouraged to co-design the modelling process and co-construct the computer-based model, but also jointly make decisions within the planning process. As in case 3, key stakeholders may also be involved in discussions depending on the timing of participation in the modelling process. There will be strong cooperation among stakeholders as well as high stakeholder capacity and a good governance setting. In more competitive contexts which may exhibit lower levels of trust and capacity, joint action that incorporates co-designing and co-decision making can be an effective mechanism for providing transparency and comfort, and thereby, building trust.

Participatory modelling versus collaborative modelling
In this article, a distinction is made that delineates collaborative modelling as a subset and more intensive form of participatory modelling. As depicted in Fig. 6, collaborative modelling approaches are more suited to decision making processes in highly cooperative contexts (collaboration and/or joint action) with high levels of participation for key stakeholders (i.e. co-design and codecision making). In some cases when key stakeholders are involved in regular discussions, the approach may also be considered to be collaborative. By contrast, participatory modelling occurs across a wider spectrum and can involve lower levels of participation. It can include stakeholder involvement ranging from discussion to consultation to information sharing. Types of cooperation between the organizing and modelling team, and stakeholders can range from coordination to joint action.
It is important to note that many participatory and collaborative modelling approaches consider one unique level of participation and type of cooperation for the relatively limited number of stakeholders involved. However, other approaches are used for large scale planning and decision making, where large numbers of stakeholder groups preclude the common involvement of all stakeholders. Such approaches frequently divide the stakeholder community into various groups (Section 3), in which the level of participation and type of cooperation for each of the groups might differ.

New framework for participatory and collaborative modelling in water resources management
The preceding section distinguished between collaborative and participatory modelling according to their two determining dimensions: levels of participation and cooperation. In this section, we expand on these two dimensions by taking into consideration other factors that can influence the selection of a particular participatory or collaborative modelling approach. Where relevant, distinction is drawn between participatory and collaborative modelling in relation to these factors. In spite of this, the reader should keep in mind that whether an approach can be considered to be participatory or collaborative will in the first instance be determined by its comparative levels of participation and cooperation. Any differences between the remaining factors serve as a guide to refine the design of the stakeholder modelling approach.

Purpose and structure
Decision makers, stakeholders and practitioners must be able to identify when to use participatory or collaborative modelling approaches, or a combination of both. They need to be able to determine which tool or combination of tools, and which existing approach(es) ( Table 1) is most suited to the given context, considering the trade-offs (Gray et al., 2015). This demands a systematic analysis of the conditions related to the problem being addressed as well as the enabling environment. The critical aspects that need to be considered can be summarized with the following question: Who (which group of stakeholders) needs to be involved in which steps of the planning process (timing), to what extent (level of involvement) and how (participatory approach, communication techniques and visualization tools)? All these aspects lead to the design of a participatory modelling or collaborative modelling approach. This analysis will help the design process of the participatory and/or collaborative modelling approach.
The generic framework presented in this article helps to: (i) define the generic characteristics and features (trade-offs) of existing participatory and collaborative modelling approaches (Table 1) and tools; (ii) generalize case-specific participatory and collaborative modelling approaches, and corresponding tools; and finally, (iii) categorize the previous approaches (i) and (ii) into participatory or collaborative modelling approaches.
The new framework for WRM is presented in Table 4. It combines the definitions and typologies described in the previous sections with other features identified in earlier work by other scholars. The generic framework comprises 20 parameters categorized into six main factors: context and application, specific use, information handling, stakeholder involvement structure, modelling and organizing team and means. These are all important factors to consider during the selection of a particular participatory or collaborative modelling approach. Their selection was based upon their relevance to planning and decision making processes for sustainable WRM. In this regard, Table 4 also concretizes the peculiarities of participatory and collaborative modelling (Fig. 6) by stressing their differences in the design process according to selected parameters.

Development and validation process
The design of the generic framework was firstly based on literature review. The majority of relevant parameters for WRM were then determined. The framework was then tested in three different participatory and collaborative modelling cases, one of which is illustrated in Section 6. Validation and refinement followed, during which additional parameters were identified and included in the framework such as decision making context, time horizon, planning or management cycle phase, and means, among others. The final version of the generic framework was then tested and validated in two more approaches.

Context and application
5.3.1. Problem type 5.3.1.1. Problem structure. Problems can be distinguished based upon their degree of complexity (Section 3). This relates to the degree of structure involved. Two factors are considered for the evaluation of problem structure: uncertainty and consensus. Problems can be classified as being either: structured, semistructured (dominated by either uncertainty or disagreement), or unstructured (Table 3).

Scale of action.
The scale of action for addressing a problem and the size of the potential stakeholder community can affect stakeholder participation in the various modelling stages. The problem scale (i.e. local, regional, national, transboundary) can determine the influence and interest of different stakeholders (Hare et al., 2003).

Time horizon.
The planning time horizon can influence levels of stakeholder interest and involvement. The considered time horizons are: short (0e10 years), medium (15e30 years) and long (50e100 years).

Domain
Problem contexts can be categorized according to their dominant management domain (Hare et al., 2003). Certain participatory modelling approaches may be more suitable for particular WRM domains, for instance, Integrated River Basin Management, Integrated Coastal Zone Management, urban/rural water management, environment, groundwater management, spatial planning, land use management, etc.

Interaction context
Two interaction contexts can be distinguished when considering the willingness of the involved stakeholders to cooperate in joint decision making. This relates to problem structure (above) and therefore will impact the selection of the participatory modelling approach (selection of a participatory or collaborative modelling approach).
In cooperative interaction contexts two or more stakeholders agree to engage each other and work jointly towards a resolution of a particular decision making problem. Also, information is commonly shared. In competitive interaction contexts, two or more stakeholders face a decision making issue in which each stakeholder is less willing to give ground. Stakeholders typically generate preferred solutions independently without considering the concerns and ideas of others. Commonly, these contexts generate confrontation, discourage information sharing, and demand that agreed solutions are established through mediation and negotiation.

Participatory modelling purpose
Different approaches better serve different purposes (Hare, 2011). Those purposes considered in the framework include: Decision making: where the outcome of the participatory modelling process is a management or planning decision. As stressed by Borowski and Hare (2007), not every recommendation from the participatory or collaborative modelling process need to be adopted, but rather serve as an input to the decision making process.
Capacity development through collaborative learning: where stakeholder education and learning is the principal purpose for the participatory or collaborative modelling approach. Learning is a social act; communication between individuals fosters both individual and collective learning (CL, 2009;Voinov and Gaddis, 2008). Stakeholders can share their concerns and perspectives, develop skills on joint problem solving and generate collective ideas and measures (Hare, 2011).
Mediation: where the intended outcome of the participatory modelling process is to help mitigate or resolve stakeholder disagreements and conflicts (i.e. in semi-structured or unstructured problems) ( Van den Belt, 2004).
Model improvement: where the objective of the process is to improve the model in terms of quality, acceptance or integration (Hare, 2011).
Many participatory modelling approaches do not have a single purpose, but rather a combination of rationales (Voinov and Bousquet, 2010). In certain cases these can be complementary whilst in others they may act in opposition. Nevertheless, a dominant purpose should be identified to help better define the participatory or collaborative modelling approach.

Planning or management cycle phase
When defining the participatory modelling or collaborative modelling approach, particularly when designing the stakeholder engagement process, it is important to consider each of the different phases of the planning cycle (Fig. 2) and make clear that participation can never be all-inclusive. The involvement of stakeholders needs to be a balance between "breadth" and "depth" (Voinov et al., 2016). The level and structure of involving stakeholders as well as when to use the selected methods and tools needs to be defined based on the different stages of the modelling and planning processes (Loucks et al., 2005).
5.5. Information handling 5.5.1. Model characterization 5.5.1.1. Model system focus. The framework adopts Bots and van Daalen's (2008) five model system types. The three main components of a WRM system are the physical system, social system and actors. Models can therefore be classified according to their focus as either: physical system models (PSM), single actor decision models (SADM), individual actor impact models (IAIM), social system models (SSM) and socio-physical system models (SPSM). 5.5.1.2. Model type. The type of computer based model can vary according to the modelling techniques used. Three modelling techniques are considered: (i) analytical models (including conceptual, (numerical) simulation models), (ii) data driven models (e.g. statistical models), and (iii) optimization models (Kelly et al., 2013;Loucks et al., 2005).

Modelling tool/software platform
Certain participatory and collaborative modelling approaches use specific modelling tool(s) or software platform(s), whilst others permit the use of a variety of tools. These can vary from Excel sheet models, agent based models, Bayesian network models, system dynamics models, spatial GIS based models, hydraulic and hydrological models, to raster-based visualization models that include both temporal and spatial dynamics (Gray et al., 2015;Kelly et al., 2013;Loucks et al., 2005;Voinov and Bousquet, 2010;Voinov and Gaddis, 2008).

Information type
The information being handled in the modelling process can relate most to either complex processes or system interactions. This can affect the model type and any visualization of results. For complex processes, the main modelling focus is on the representation of a particular individual system and its sub-system processes at specific scales. For system interactions, models focus mainly on general interactions between various sub-systems elements, and not on any specific sub-system processes. This leads to the use of qualitative and/or quantitative information. The availability of one or another, or the co-existence of both will also determine the modelling approach.

Information delivery medium
The medium in which the information is delivered can constrain the involvement of some stakeholder groups, particularly in remote areas. It can also affect the decision making process. Two delivery media are included in the framework: face-to-face delivery and delivery via a virtual platform (Almoradie et al., 2015;Heller, 2010;Jonoski, 2002). 5.6. Stakeholder involvement structure 5.6.1. Participatory method Two dominant generic approaches for stakeholder involvement are considered: participatory or collaborative. These vary according to the prevailing levels of participation and types of cooperation in each approach (Fig. 6 and Section 4).

Stakeholders involved
A variety of stakeholders can be involved in participatory and collaborative modelling approaches. Certain modelling activities may demand specific skills and knowledge to develop and/or use the model or tool. In this framework, stakeholders are identified by organization, type of stake in relation to the problem(s) addressed, their backgrounds and any minimum skills and knowledge requirements for participation in the approach (e.g. general local knowledge and technical skills/specific knowledge).

Model users
A distinction is made for any stakeholders targeted as users of the model. Direct users are those who will directly manipulate and provide inputs to the models. Indirect users are those who will indirectly manipulate the model via an intermediary, for instance, an expert modeller. The technical skills and knowledge required to enable such model use are also defined (e.g. general, specific or no computer skills).

Participation mode
Four participation modes of stakeholder involvement are considered in the framework: no participation (only modellers), individual (individual stakeholders are involved separately), and either homogeneous or heterogeneous groups (participation occurs collectively with multiple stakeholders). Homogeneous groups are groups where stakeholders with similar interests and perceptions participate together. Heterogeneous groups are those where participation occurs amongst stakeholder with divergent interests and perceptions of the problem (Andersen and Richardson, 1997;Bots and van Daalen, 2008). The participation modes may vary in time based on the specific planning step and modelling phase.

Level of participation
The level of stakeholder involvement can vary between approaches and for different activities within each approach. The ladder of participation introduced in Section 3 is used for defining the levels of participation.

Timing of participation (modelling phase)
Modelling phases in which stakeholder may be involved are distinguished as: data collection, model definition, model construction (e.g. initial model building or model refinement), model validation and verification, model use (e.g. providing model inputs, actual use of the model or acting in gaming simulations), and formulation of measures and design of alternative strategies. Depending on this timing and phase(s) of involvement the approaches can then be classified in four generalized participatory modelling forms: Front-and Back-End (FABE), Co-construction, Front-End (FE) or Back-End (BE) (Bots and van Daalen, 2008;Hare, 2011).

Type of cooperation
Four types of cooperation are considered for this framework: unilateral action, coordination, collaboration and joint action. These are based on the cooperative continuum introduced in Section 3 (Sadoff and Grey, 2005).

Team
The organizing team is responsible for the design and guidance of the participatory and collaborative modelling process. This includes model construction and use. A good organizing team will typically include at least one modelling expert with comprehensive knowledge of the modelling tool used (Hare, 2011).

Skills
Additional information included refers to the skills required by the team to effectively guide the process (e.g. modelling skills, facilitation skills, knowledge acquisition skills and process management skills).

Means
Different participatory and collaborative modelling approaches demand different time and financial commitments.

Application example of the generic framework to a collaborative groundwater modelling study in the Netherlands
The proposed framework was applied to a case study of an interactive modelling approach (Table 1). This section provides practical insight on the use of the generic framework to evaluate this approach. The methodology of the evaluation will be based on Table 4.

Context and application
The case study is based on a regional groundwater model development project in the Netherlands, abbreviated as MIPWA (Development of a Methodology for Interactive Planning for WAter management; Berendrecht et al., 2007).

Problem type: scale of action
The case study consists of predominantly agricultural and natural areas with some urban developments. It covers much of northern Netherlands, including the provinces of Friesland, Groningen, Drenthe and parts of Overijssel (Fig. 7).

Domain
Prior to this project, numerous groundwater models had been built by the various stakeholders for different purposes and according to inconsistent sets of assumptions.

Interaction context
The fact that there were many different groundwater models inevitably resulted in conflicting model results and decisions from these stakeholders on water-management related issues. To eliminate or reduce the instance of such knowledge-based conflicts in the future, the development of a single, consensus model that all stakeholders accepted and viewed as credible was broadly desired. The decision making context was therefore cooperative.

Problem type: problem structure
The problem context in which the case study occurred can best be considered as being semi-structured and dominated by uncertainty.

Collaborative modelling purpose
The key objective of the interactive modelling approach was to reduce these uncertainties by establishing technical consensus. Focus was placed on achieving the highest quality model with the available budget, whilst also integrating the technical perspectives and interests of each of the stakeholders involved. Shared learning was an additional concern, as each of the scientists, engineers and policy professionals involved were able to discover the possibilities of the intended model whilst collaborating in its construction.
6.3. Information handling 6.3.1. Model type and software platform The computer-based model consisted of an interactive, scalable, high-resolution iMOD physical groundwater simulation model (Berendrecht et al., 2007). It included a rapid assessment tool consisting of a pre-calculated database of policy action effects.

Information delivery medium
This tool was capable of supporting collaborative and interactive policy analysis for surface-and ground water regime planning. The visualization and discussion of modelling results occurred during a series of interactive face-to-face workshops.

Type of cooperation
The seventeen water management organizations active in this region joined together in a consortium (i.e. in joint action).

Stakeholders involved
The stakeholders involved included representatives from the four provincial authorities, three drinking water companies, six water boards and three large urban municipalities. Given the complex, detailed and sophisticated nature of the developed simulation model, involvement in the consortium was restricted to members with sufficient hydrological knowledge to effectively contribute to the discussions.

Participatory method and model users
With assistance from expert knowledge institutions, stakeholders collaborated intensively in the development of the iMOD model.

Timing of participation
Stakeholders were involved in all model development activities, ranging from data collection, model schematization and definition, through to construction, model verification and validation.

Participation mode
Model construction, verification and validation was achieved through a series of twenty interactive workshops, in which participants were often divided into smaller heterogeneous groups to enhance opportunities for individual contribution.

Level of participation
Stakeholders were therefore able to exert a high degree of influence and control over the entire model development process, and took joint decisions at all key milestones.  The team consisted of scientific experts and process managers from two Dutch water research institutes: TNO (now Deltares) and Alterra.

Skills
The modellers and process managers who formed this team needed to demonstrate the requisite facilitation, knowledge acquisition, modelling, and process management skills to bring about the broad success of the approach.
6.6. Means 6.6.1. Financial resources This case study was a V2 million regional groundwater model development project in the Netherlands.

Timing
Stakeholders cooperated effectively in this joint action over a period of more than two years.

Discussion
There is an increased attention to stakeholder involvement in modelling processes (Voinov et al., 2016). Therefore, a growing number of participatory planning and modelling approaches are now available in the field of WRM. Examples include Group Model Building, Shared Vision Planning, Interactive Modelling, Collaborative Modelling using Networked Environments, among others (Table 1). Although these approaches have common similarities, there are trade-offs to consider (Gray et al., 2015). The terms "participatory modelling" and "collaborative modelling" are also used interchangeably. This leads to one of the main challenges faced by researchers, practitioners and policy makers: to identify which participatory and collaborative modelling approach is best suited to each type of decision and related process. In this article we have detailed the common features of "participatory modelling" and "collaborative modelling" (Sections 2 and 3), but also made a distinction between both umbrella terms (Sections 4 and 5). We have also developed a generic framework that helps categorize existing approaches and their corresponding tools into "participatory modelling" or "collaborative modelling" (Table 4).
The use of the generic framework might seem a straightforward process; however, it frequently requires several iterations. The process begins with filling in known information about the given approach for each of the parameters. Information can be obtained through literature review or practical experience. For well-defined approaches (Table 1) this first step will be quite straight-forward. In other cases where the approach is more case-specific, the generalization will require a comparative analysis of a number of similar case-specific approaches. This may demand a more iterative process. This process will allow decision makers, practitioners and researchers to define their generic characteristics and features. The last step is to categorize the given approach into "participatory modelling" or "collaborative modelling" or a combination of both by comparing the obtained results with the general characteristics and features of "participatory modelling" and "collaborative modelling" provided in Table 4.
An example of the application of the generic framework to an existing approach is illustrated in Section 6. This approach is a good example of collaborative modelling. Based on Fig. 4, the interactive modelling approach adopted comprised high levels of stakeholder participation (i.e. co-decision making) and joint action as the type of cooperation. The contextual situation was a complex, semistructured problem where uncertainty in scientific knowledge was the dominant factor. The ultimate use of the iMOD model was for policy analysis for surface-and ground water regime planning; however, this case mainly focused on the interactive development of the model. A cooperative decision making context within the consortium was of critical importance in the collaborative modelling approach (i.e. in terms of the design of stakeholder workshops and participation mode, timing of participation, visualization of model results, information delivery medium, etc.). The application of the framework in this case has not only helped to classify this interactive modelling approach as collaborative modelling, but it has also helped identify its key characteristics, features and tradeoffs. This information can be relevant for decision makers, practitioners and researchers when deciding whether to use interactive modelling for a particular decision making process, project or study.
Results from such evaluations are useful for participatory/ collaborative modellers as well as decision makers, researchers and practitioners. For instance, Section 6 demonstrates that interactive modelling is most suited to modelling contexts where high levels of cooperation exist between the stakeholders involved and where all stakeholders possess the necessary level of technical expertise. In the Netherlands, the dominant governance setting fosters the use of this collaborative modelling approach. Given that such cooperative, participatory and high-tech contexts are not always common in other regions of the world, participatory/collaborative modellers, decision makers, researchers and practitioners must therefore assess if this or a similar approach would be suitable in these contexts.

Future research directions
A number of directions regarding the design and applicability of the generic framework should be explored in the future. Firstly, with this research the authors have demonstrated the potential of using the proposed generic framework to evaluate participatory and collaborative modelling approaches in WRM. To confirm the generic applicability of the framework, it must be applied to other participatory and collaborative modelling approaches and additional case studies in other contexts and situations. Secondly, an adapted and more specific (sub-)framework could be developed to evaluate the variety of participatory modelling tools (including computer-based models, visualization and communication tools). Finally, the proposed framework (in and of itself) does not provide detailed guidance in relation to selecting the most suitable approach. Once further approaches and tools have been analysed, a possible future research direction would be the development of a more detailed decision path based on a selected set of parameters.

Conclusions
Effective and sustainable WRM demands systematic planning and decision making processes that include stakeholder participation, are enabled by the use of computer-based models (informed decision making), and promote cooperation and negotiation. Participatory and collaborative modelling are an emerging set of approaches that cover a variety of ways to combine these elements. This is particularly important when addressing complex problems. A useful first step in examining these problems is to look at the existing levels of consensus among stakeholders and the degree of scientific knowledge related to the problem being addressed. These factors are critical contextual determinants in identifying the participatory and/or collaborative modelling approach(es) suited to each problem type.
In this article, a distinction is made between participatory and collaborative modelling based upon two determining dimensions: levels of participation and types of cooperation. Collaborative Modelling occurs when key stakeholders co-design and/or take joint decisions within the modelling process, and when stakeholder cooperation manifests itself as collaboration and joint action. Participatory modelling, in contrast, covers a wider spectrum of participation levels (from awareness to being involved in discussions) and types of cooperation (from coordination to joint action). In some planning and decision making processes a combination of approaches may be appropriate. For instance, a collaborative modelling approach could be used for key stakeholders and a participatory approach (lower levels of participation) used for other interested stakeholders.
Determining the suitability of a participatory or collaborative modelling approach for a particular planning and decision making process is dependent on a further set of critical factors relating to the local context and situation. A generic, detailed framework has been presented as a supporting tool in this article. It can be used by policy makers, practitioners, researchers, local stakeholders and decision makers to support the evaluation of participatory and collaborative modelling approaches in WRM. The framework was adapted and further elaborated from other previously published evaluation frameworks. The proposed generic framework comprises 20 parameters that have been grouped into six main factors: context and application, specific use, information handling, stakeholder involvement structure, modelling and organizing team, and means. This framework was illustrated by an example case: an interactive modelling approach adopted for a groundwater modelling study in the Netherlands. The application of the framework in this case shows promising results. However, these six factors may vary widely across different regions and subtle differences will likely exist between the different participatory and collaborative modelling approaches. As such, further analysis is necessary to demonstrate the general applicability of the framework for water resources planning and management around the world.