Learning from Your Neighbor : The Value of Public Participation Evaluation For Public Policy Dispute Resolution

Public policy dispute resolution [PPDR] and public participation [PP] are closely related fields of practice. Despite many similarities, we identify gaps in most evaluation studies conducted in the two fields. Evaluators of PPDR can better draw upon PP evaluations. The values, concepts, objectives, and practices of PP and PPDR are compared for similarities and differences. Focusing on the last 15 years, PPDR and PP evaluation literature is analyzed, with attention to four PP evaluation studies of special relevance to PPDR. Five ways that PPDR evaluation can be improved by drawing on PP evaluation studies are identified. Two promising works that begin to bridge the PPDR-PP evaluation gap are summarized.


Introduction
Public policy dispute resolution [PPDR] and public participation [P2] are closely related fields.While both fields promote evaluation of processes and outcomes, we find a gap in most evaluation studies in the two fields.Our argument is that PPDR evaluation approaches can learn from P2 evaluations.
We first compare the goals, values, concepts, objectives, and practices of P2 and PPDR.We highlight selected similarities and differences.We then summarize the recent PPDR evaluation literature, focusing on the gap of public participation dimensions of PPDR practice.By then examining P2 evaluation literature, we find that the evaluation gap in PPDR can be addressed by more explicitly incorporating the theory and methods from public participation evaluation.Some recent PPDR evaluation efforts show promise in bridging the gap.

Overview of PPDR
P2 practitioners may not be familiar with the PPDR field.Practitioners in PPDR largely draw from alternative dispute resolution (ADR), collaboration and facilitation concepts.The work consists primarily of assessing situations for the potential of stakeholders to negotiate a dispute relating to policy or regulation (although non-agreement seeking and adjudication-related work are parts of the field), and then professional facilitation or mediation to seek an agreement.
Seminal texts identify the development of the field, typical kinds of disputes, case illustrations of mediation and facilitation, and common processes of PPDR (Bingham, 1986;Carpenter and Kennedy, 2001;Dukes, 1996;Gray, 1989;Susskind, McKearnan & Thomas-Larmer, 1999).Competencies to distinguish PPDR from other forms of ADR were established in 1992 (SPIDR, Environmental/ Public Policy Sector, 1992), followed by guidance for government officials using PPDR (SPIDR Environment/Public Disputes Sector Critical Issues Committee, 1997).While much of the focus is on state or federal agency-sponsored processes, providers range from national firms to community mediation centers (Sachs, 2000).
The strongest policy strand in PPDR is environmental conflict resolution (ECR), with rosters of mediators and facilitators maintained by two federal agencies, and the creation in 1999 of the U.S. Institute for Environmental Conflict Resolution (See: www.ecr.gov ).

Similarities of PPDR and Public Participation Fields
Both PPDR and P2 fields share an overarching purpose and values, employ many common concepts and objectives, have similar practices, and hold complementary differences on some objectives and practices.None of these observations are new,1 but they have been significantly developed over the last decade.
We will concentrate our review of evaluation efforts over the last 15 years, focusing first on distinctions between PPDR and P2 evaluation literature.Later, we offer examples of bridging of the two areas.Before doing so, it is important to summarize the similarities and differences in the practices of the PPDR and P2 fields which inform some of the differences in evaluation.

Common Goals and Values
The general purpose of both fields is to help participants in the public arena reach better decisions.Specifically, P2 and PPDR literature identify common aspects of what "better" means: greater perceived legitimacy of the decision based on involvement by stakeholders, open-minded consideration of diverse information and perspectives in the decision-making process, the opportunity for new proposals to be raised and considered in the process, and having the ultimate decision-maker acknowledge the input of the participants.
James Creighton provides an elegant framing of the purpose that joins P2 and PPDR: "What does it take for a decision to count?" (Creighton, 2005).This common touchstone for both fields is played out on a range of thorny issues: representation of stakeholders, access to the involvement/negotiation process, information distribution, sufficient time for education and creative problemsolving, and third party responsibilities for superintending over a P2 or PPDR process.Nonetheless, "legitimacy of the decision" is at the center of design and evaluation considerations., 2009a, 2009b). 2 A comparison of these documents demonstrate the two fields are best considered as complementary perspectives on many of the same needs, organizing concepts, activities, and desired outcomes.

Common concepts, objectives and practices
We highlight two core concepts and objectives: third party assistance, and ability to influence a decision.
PPDR and P2 practice often employ a third party, separate from decisionmaker(s) or stakeholders, who assists in designing and managing a process that pursues a variety of goals beneficial to the stakeholders and decision-maker(s).The third party is "a protector of the process."(Moore, 1998) This means that she has a fiduciary role in defining, with substantive (if not total) stakeholder codetermination, the purpose, parameters, procedures and expectations for what will be mutually acceptable and beneficial procedures for engaging on an issue or pending government action.Much of the professional ethics in the documents listed above, and practical applications, go to the core of third party as "helper of all" who guards against participants or interlopers eroding the value of the participation/ negotiation process (Carpenter, 1995).
Stakeholder involvement for practical or moral reasons animates PPDR and P2 practice: there is a high expectation of some kind of influence.
Finally, there is a range of shared or overlapping practices.To note briefly: a) An early issue of IAP2's4 journal, Interact, included an article about a process that moved from public involvement to mediation, and examined the potential pitfalls of such a change in third party assistance (Moore, 1997) (Orenstein, 2009) and a spectrum was developed to describe similarities and differences (Orenstein, Moore & Sherry, 2008).
Concerning the goals and best practices of P2, Larry Susskind-a PPDR leader-engaged five P2 leaders over the IAP2 spectrum, the relative value of consultation vs. collaboration, and the aspirations vs. realities of P2 (Carson, 2008).

Differences of PPDR and Public Participation Efforts
There are four main differences between PPDR and many forms of P2: the goal of agreement, the scope of participation, direct or representational involvement, and decision authority.Each is summarized.
Reaching an Agreement -the most striking difference between PPDR and P2 efforts is that PPDR processes seek an agreement.In contrast, agreement is not a specific goal of most categories of public participation.IAP2 seeks public participation "to make better decisions that incorporate the interests and concerns of all affected stakeholders and meet the needs of the decision-making body" (IAP2, 2009a).Nonetheless, two continuums of public participation, one from IAP2 (IAP2, 2009c) and one from prominent P2 writer and practitioner, James Creighton, (Creighton, 2005, p. 9) place the idea of reaching agreement as only one of the possible goals and purposes of P2.
Scope of participation -PPDR allows for a narrower scope of stakeholders if that is likely to contribute to a successful resolution.From all potential stakeholders, selection depends on the degree that an individual, organization or community is affected, how well they can be represented, and which entities have power and/or resources relevant to the situation.By contrast, IAP2 seeks involvement by "all affected stakeholders," emphasizing open access to the process.
Representation and Authority -the other two main areas of difference for P2 and PPDR are representation and decision authority.First, representation of interests is established as an initial step in PPDR, but as a mid-to late-process outcome in many kinds of P2 efforts.PPDR commonly seeks established leaders of identifiable constituencies to be negotiators, consistent with the goal of seeking agreement.IAP2 seeks involvement by "all affected stakeholders."The definition of stakeholder is purposefully broad, and eschews the idea of legal standing.For instance, one of the IAP2 Core Values is: "Public participation seeks out and facilitates the involvement of those potentially affected by or interested in a decision."(IAP2 Core Values).
Negotiation among representatives leads to the other common difference for public participation and PPDR: the concept of decision authority.PPDR works comfortably with regulatory negotiation, compliance mediation, and policy dialogues, where the participants either have the power themselves or have been granted the authority to direct implementation of an agreement.Public participation makes no such promise.For example, IAP2's central documents focus on responsiveness, i.e., incorporating interests, concerns and suggestions.
"Suggestions" makes clear that someone else holds decision authority, albeit with some sense of altruistic or practical desire for assent (or lessened dissent) from stakeholders through their participation.6

Despite Similarities, Gaps Exist in PPDR Evaluation
A review of the goals, concepts, and practices show that there are great similarities in the fields of PPDR and P2.We would expect, therefore, to have great similarities in evaluation.However, we find that the differences in goals result in different attention and criteria used for evaluation.In particular, we find that PPDR evaluation gives little attention to the quality of public participation.We first consider approaches to PPDR evaluation, then approaches to P2 evaluation.

PPDR Evaluations -recent assessments
The most comprehensive PPDR evaluation summaries in the last ten years do two things.First, they overwhelmingly focus on agreement-seeking settings for what is counted as "cases" in most individual PPDR evaluations.Second, they cite one prominent P2 evaluation as clearly relevant to PPDR: a meta-analysis by Beierle andCayford, (2002, and2003) Birkhoff, and a later comprehensive review by Dukes (2004), cite Beierle and Cayford's meta-analysis of 239 cases seeking to understand how a range of P2 mechanisms perform on five social goals.One of the five social goals is "resolving conflict among competing interests" (Beierle & Cayford 2002, p. 15).Obviously, the agreement-seeking nature of PPDR makes Beierle and Cayford's approach to "public participation in environmental decisions" attractive.Three of the four other social goals are related or subsidiary components and benefits of PPDR: • Incorporating public values into decisions • Improving the substantive quality of decisions • Building trust in institutions The fourth goal, "educating and informing the public" is not as central to PPDR.
Stakeholders can be better informed, and different ideas integrated (or compromised) into an agreement.But broad scale public education is not typically a goal for most PPDR processes.
Beierle and Cayford, from our research, is the one outlier in bridging P2 and PPDR evaluation interests.Of the 239 cases they examined, 47 involved negotiation or mediation.Dukes notes an evolution in how ECR draws borders to its work.Early ECR research focused on mediation.Now, he writes, the vocabulary has expanded to consensus-building, collaboration, collaborative learning, collaborative planning, collaborative natural resource management, community-based collaboration and community-based conservation (Dukes, 2004, p. 191).Dukes states "Many practitioners would include enhanced public involvement within their practice as well.Some work encompasses a combination of such processes."(Dukes, 2004, p. 192, emphasis in original) While Dukes portrays ECR such that it may or may not include a mediator or facilitator, it does include "consensus or some variation other than unilateral decision making as the basis for agreements" (Dukes, 2004, p. 192).So, while Dukes opens the door to enhanced public involvement with one hand, with the other hand he moves it to the margins with the ECR definition of agreementseeking.
Turning to a 2008 example of PPDR evaluation, a large-scale performance evaluation (PE) system drew from federal-and state-level cases to document and evaluate environmental conflict resolution (Orr, Emerson & Keyes, 2008).A summary is offered here, but we return to this study as one promising bridge between P2 evaluation and PPDR evaluation.
The evaluation framework effort was initiated in 1999, with collaborative work among the authors' home institution (the U.S. Institute for Environmental Conflict Resolution), the Policy Consensus Initiative and state-level PPDR programs.
While citing Beierle and Cayford (2003), and Dukes (2004), Orr, Emerson and Keyes' definition of cases for inclusion in the dataset have an agreement-seeking intent, and "involvement of an independent, third-party facilitator or mediator" (Orr, Emerson & Keyes, 2008, p. 287).Helpfully, the framework does have elements for how "participants are effectively engaged" such as communication, understanding different viewpoints, identification of information needs, and narrowing and clarifying the issues in dispute.(Ibid., pp.289 and 295-96) Nonetheless, those "quality of participation" components are examined only within the overriding goal of agreement-seeking.
In sum, P2 is noted as a part of ECR and PPDR evaluation via a meta-analysis by Beierle and Cayford.Nonetheless, P2 continues to be placed on the edge of PPDR practice, at least when it comes to considering what past evaluations count in the PPDR arena.

Evaluating Public Participation
We concentrate on more recent studies of public participation for purposes of brevity.We do not wish to diminish earlier efforts to develop categories and ways to assess P2 (such as Rosener, 1978;Sewell & Philips, 1979;Thomas, 1995;and Irvin & Stansbury, 2004), but choose studies that build on earlier work and offer more specific indicators of effective public participation.First, one effort at synthesis is noted.
Hendricks ( 2009) crafted an evaluation framework for community engagement which moves from six principles of engagement (such as integrity, inclusion, influence, and sustainable decisions) to indicators of the principles, to draft questionnaires for gathering data on how well those indicators are met in a particular engagement process.Different questionnaires are offered based on whether the respondent is a stakeholder, process implementer, decision maker or participant (Hendricks, 2009, pp. 4-8).Hendricks writes she is a "representative" of IAP2-Australasia, but it is unclear how widely this framework and questionnaires (or modifications of either part) have been used (beyond some testing of the tool and some "university assignments…based on the framework," Hendricks, 2009, p. 1).
A more general issue has been seeking a consensus in the literature on what is considered P2 "effectiveness."For example, King, Feltey, and Susel (1998) define an effective citizen participation process as an authentic process.They put forth guidelines for structuring an authentic process, and enumerate common barriers that administrators face to authenticity.They do suggest potential indicators, but no framework for incorporating them into a larger evaluative model.
We now highlight four studies most relevant in our mind to P2 evaluation and links to PPDR.
First, Thomas Beierle (1999)7 attaches specific social goals to particular participation methods in order to measure effectiveness.However, Beierle's model is driven by the end-result-"Was it a good decision?"-notmerely by the process itself.Beierle's objectives for an evaluation framework are: 1. Identify strengths and weaknesses of a number of public participation mechanisms; 2. To be "objective" in the sense of not taking the perspective of any party to a decision 3. Measure tangible outcomes to the extent feasible Beierle argues that these objectives preclude a process or interest oriented approach to evaluation.However, we find that Beierle does identify a variety of "mechanisms" that could be classified as process-oriented indicators, as shown in Table 1 below.Beierle explores the relationship between these particular mechanisms of participation and the general likelihood of achieving different social goals.dimensions of cumulative time on a public involvement project, the perceptions about difficult or distasteful parts of the process, and the degree of overall attention needed for the process in the context of other work and priorities of the process sponsor and process participants.Similarly, they note emotional investment and discomfort as an indirect cost stated by their interviewees (Lach & Hixson, 1996, pp. 62-63).Although widely noted by PPDR third parties and participants, there has not been detailed coverage of the range of perceptual values on time, emotional investment and discomfort, or opportunity costs.
In discussing their findings, Lach and Hixson make two points.First, there is a central dilemma: almost all respondents thought the wide diversity of views represented in the public involvement process was valuable and important to them, and almost all reported that accommodating those different values and needs was difficult, if not impossible.The PPDR challenge is similar: seeking agreement among divergent interests is often painful, but achieving a committed, often innovative agreement, creates great value.Second, the "savings" achieved through the public involvement process are typically framed as avoided costsi.e., litigation, or other elements of the implementation not needed.The authors observe that these are largely hypothetical savings.
Lach and Hixson report that it was hard to determine and measure indirect costs.For example, the time commitment felt quite large to several interviewees.However, when pressed to document the time spent, "…the tallies for all participants appeared to us to be quite modest" (Lach & Hixson, 1996, p. 64).
Beierle's contribution has been noted earlier as often referenced by PPDR evaluators.His four process indicators overlap with at least one of the other studies.For outcome factors, his five "social goals" cover all but one of the categories we use to compare the studies.
Finally, Schweitzer, Carnes, and Peel (2000) focus on attributes and performance indicators of successful public participation efforts.Their field data comes from over 100 interviews of individuals involved in one of nine remediation sites with active U.S. Department of Energy participation efforts.Like Lach and Hixson, the applicability of their work may be limited by the nature of Department of Energy public involvement efforts or the features of P2 on remediation sites.However, given the public administration field's interest in performance measurement, Schweitzer, Carnes, and Peel's work can contribute to PPDR evaluation aimed at government agency performance concerns.
Schweitzer, Carnes, and Peel offer two dimensions for measuring legitimacy: process and key decisions.Like Beierle, they look to a more general goal of individual public participation efforts: public trust in the sponsoring agency.Both are useful contributions to central values for doing PPDR or PP.

Findings
We think analysis of the highlighted studies of P2 evaluation yields five valuable points for guiding further development of PPDR evaluation.
First, the studies describe some shared and some different criteria by which to measure PPDR efforts.Some criteria are clearly behavioral, others are perceptual.
Rowe and Frewer identify an unbiased, transparent process as an indicator.This item is a key part of most PPDR processes, and is largely perceptual.However, Rowe and Frewer's participation indicator, which is defined in terms of "resources to participate," is a useful addition to PPDR models in the areas of information exchange and comprehension.Also, resources may be more objectively or behaviorally measured.
Second, the studies cover both inductively created criteria drawn from interviewing public participation participants, and deductive social goals drawn from the shared purpose and values of P2 and PPDR.This area of uniformity vs. variability in approach to PPDR evaluation is an important contribution.
Third, Beierle offers some particular relationships between goals and measures that can be compared to past and prospective PPDR evaluation.As noted earlier, reducing conflict, incorporating public values, improving the substantive quality of decisions and increasing trust in institutions fit well within PPDR evaluation concerns.Moreover, as PPDR work gives more attention to consultation, outreach and some degree of education outside of the representatives at the table, the fifth goal from Beierle -educating and informing the public -becomes increasingly relevant.
Fourth, Schweitzer, Carnes, and Peel offer a specific link to performance measurement concerns.Such a link is notable because performance measurement is an important topic in public administration program evaluation.For example, Epstein, Coates, & Wray (2006) present a model that focuses on the overlaps of effective citizen engagement, competent project implementation, and performance measurement.
Finally, there is an iterative feature of public participation that is increasingly relevant to PPDR.Public hearings are a common, often repetitive, form of P2.Citizen advisory committees are legion.Attending to multiple P2 cases (or several P2 elements in a single government project or decision) highlights the longer-term variables relevant to judging the value of P2 and PPDR.
One such factor is trust.Beierle denotes "increasing trust in institutions" as one of his social goals.An indicator for Lach and Hixon is "mutual respect," and Schweitzer, Carnes, and Peel identify "public trust and confidence" in the sponsoring agency.Akin to moving away from grading the individual trees in a forest, there is an interest in trying to assess the accumulated value of P2 and PPDR in terms of legitimacy of decisionmaking on public issues.

Promising Examples of Bridging PP-PPDR evaluation
We are happy to note two very recent efforts which have begun to bridge the PP-PPDR evaluation divide and embody some of our recommendations.Under the leadership of the U.S. Institute for Environmental Conflict Resolution, a multicase evaluation pilot has provided a promising integration.While still focused on agreement-seeking outcomes, the Orr, Emerson, and Keyes (2008) model attends to pre-process factors, interaction of participants, and the outcomes are nuanced for values of durability, quality, and capacity building.In particular, data gathered from participants includes items on trust, comparison to other processes' cost and (admittedly speculative) and effects on working relationships for future conflict.For example, the item "I had the resources (e.g., time, money) needed to participate effectively in the process" (U.S. Institute for Environmental Conflict Resolution, 2007) fits perfectly with the Rowe and Frewer process indicator.
The second effort was a comprehensive, multi-year National Research Council panel that examined the value of P2 in environmental assessment and decision making.The 2008 report identified three goals for public participation: quality, legitimacy and capacity (Dietz, & Stern, 2008).These categories include many of the points we summarize of the four public participation studies.Quality includes consideration of values and concerns of affected people and organizations, and the combination of the best available knowledge and incorporating new information, among other items.Legitimacy is elaborated as perception of a process that is fair and competent, and that follows the governing laws and regulations.Capacity incorporates outcomes of being better informed, being more skilled at effective participation and having a more widely shared understanding of issues.Most notably, the capacity entails not just general communication skills, but more typical PPDR "mediation skills" and mutual trust.

Conclusion
The quality of P2, using at least some of the indicators outlined above, should be incorporated more explicitly into PPDR evaluations.We note a number of opportunities to transfer these frameworks to PPDR.
Impartial "process expert" third parties are increasingly drawing on techniques from P2 and PPDR to help stakeholders be engaged in a way that yields a decision that is more legitimate than a decision with lesser participation or agreement (as noted above, Connor, D.M. & Orenstein, S.G. 1997/1995, among others).Such "blending," even if with some hard edges (Carson, 2008), is exemplified by the Spectrum of Processes for Collaboration and Consensus-Building on Public Decisions.This spectrum shows separate categories, but notes the important relationships between the desired outcomes within those categories (Orenstein, Moore & Sherry, 2008).
The P2 evaluation studies reviewed identify important factors for PPDR evaluators.The factors include both project-specific and more general social values for attention in assessing the effects of PPDR.Finally, with negotiation, collaboration and participation models becoming a greater interest of public administration and policymaking over the long-term, (Carlson, 2007;Lukensmeyer, & Torres, 2006;O'Leary & Bingham, 2007), P2 studies of iterative participation work hold great promise for providing a more complete perspective for guiding PPDR evaluators in judging PPDR policies and programs.
All of these models need testing.Beierle notes that his proposed relationships are drawn from the literature and common sense, "not on any empirical evidence" and invites testing of the hypothesized relationships (Beierle 1999, p. 88).
Beierle's focus on social goals that transcend the actual process itself resonates with Judith Innes's framework for PPDR and consensus-building evaluation (Innes, 1999), and with a guide for P2 evaluation directed to United Kingdom central government officials (Warburton, Wilson & Rainbow, 2010).
Lach and Hixson conclude their work by calling for these prototype indicators to be applied in other public involvement efforts "…to determine whether the indicators truly capture the values participants have experienced" (Lach & Hixson, 1996, 65).To our knowledge, neither Lach and Hixson or other researchers have applied their framework.
Specifically, we recommend a focus on legitimacy.Among the many interesting ways to formulate and investigate legitimacy, one is to research longer-term collaborative groups of greater or lesser formality that seem to straddle at least two distinctions in P2 and PPDR work: representatives vs. wide, direct participation; and influence vs. decision authority.Three of many examples come to mind.First, there is a line of research specific to watershed partnerships or collaboratives (Leach, Pelkey & Sabatier, 2002;Leach & Sabatier, 2005;Sabatier, et.al, 2005;Leach, 2006).This area is particular promising because of the range of goals, participation, influence and decisions that vary by maturity of the partnership, resources available, and trust among the participants.Second, for 2002-2006, there was a grant-funded clearinghouse seeking to bridge the practitioner-researcher divide on a broad range of "community-based collaboratives."(CBCRC, 2009).Finally, Thomson, Perry and Miller (2009) offer a conceptualization and measurement of collaboration with their 17 factors categorized by governance, administration, autonomy, mutuality and norms/trust.
Drawing from the four P2 evaluations highlighted in Table 1, we see the seeds for continuing to build strong evaluation into the development and interweaving of the P2 and PPDR fields.Using a cuisine metaphor, it is as if one field has emphasized the quality of the food in a restaurant and the other has sought to measure the quality of the service.Both are valuable, but a truly good meal requires excellent service and scrumptious food.
From a shared purpose, common values guide what it takes for professionals in both fields to function effectively and ethically.Both fields have standards of practice documents.For PPDR, the values and principles are contained in Competencies for Mediators of Complex Public Disputes (SPIDR, Environmental/Public Policy Sector, 1992).A related document addresses government officials as conveners of agreement-seeking processes.(SPIDR Environment/Public Disputes Sector Critical Issues Committee, 1997).For public participation, the International Association for Public Participation's [IAP2] Code of Ethics for Public Participation Practitioners and the IAP2 Core Values for the Practice of Public Participation are the comparable documents (IAP2

Table 1 :
Public Participation Evaluation Indicators: A Comparison 9Time, Opportunity, Authority and influence, Emotional