Who Moderates the Moderators? The Effect of Non-neutral Moderators in Deliberative Decision Making

Ideal deliberative democracy seeks to employ unbiased moderators. Yet, a large literature in the field of mediation suggests the elusiveness of perfect neutrality. Our study thus addresses the following question: when moderators of deliberations express their own views – even in a limited manner – can they change the preferences of participants? Using a novel experimental design in a real deliberative decision-making process, we find that moderators can significantly influence the attitudes and behaviors of participants by expressing views in a constrained manner. The results of our study have implications for refining epistemic conceptions of deliberative democracy and for designing more precise empirical investigations of the effects of deliberative processes on attitudes and behavior. The results also warn of a simple mechanism by which interest groups might hijack the deliberative decision-making processes used in community driven development projects all over the world.


Introduction
Ideally, the procedures governing deliberative democracy -where public deliberation leads to a binding decision -should promote the inclusion of all participants in the discussion.These procedures should prevent disruptive behaviors and induce participants to listen to each other, allowing for as much diversity of opinion as possible.No single opinion should be privileged beyond the merits of its inherent rationality and logical appeal. 1 To achieve this set of normative goals, many practical applications of deliberative democracy divide the participants into small discussion groups of five to ten people, and assign moderators to facilitate the discussion.The ostensible goals of the moderators are simply to stimulate discussion and guarantee that all participants have the opportunity to speak -they are thus introduced as neutral interveners.
The procedures of deliberative democracy thus appropriately endow moderators with coercive power so that they can prevent domination by vociferous participants and thereby induce higher-quality group discussion. 2 Yet, the actual extent and form of moderator intervention varies significantly.Real-world applications of this approach include participatory budgeting and community-driven development projects used in emerging markets and developing economies throughout the world.In these contexts, moderators sometimes intervene in discussions by summarizing or reframing the opinions of participants, providing additional information, or even explicitly supporting one option over another. 3Our question is the following: What happens to the preferences of participants when moderators intervene in the discussion in support of one view over another?Raising this question highlights an often overlooked fragility of real-life deliberative decision-making processes.Most critiques of empirical applications of deliberative democracy focus on the disruptive effect of inequality among the participants, the biasing effect of outside interest groups and media on deliberation, and the polarizing effect of group dynamics. 4Even in the absence of these problems, however, moderators themselves might bias the outcomes of deliberative democracy.In real-world applications of deliberative democracy, moderators rarely hold neutral positions.In particular, if moderators have a stake in the outcome of the deliberative decision, they have an incentive to employ their coercive powers in ways that violate ideal deliberation.One of the authors of this study participated in the weekly meetings of the Participatory Budgeting Council in Porto Alegre, Brazil, observing moderators attempting to influence the outcome of discussion via interruptions, shouting, strategic agenda setting, and even direct threats towards participants.Now, academically-inspired experiments -such as Deliberative Polls®, citizens' juries, and America Speak -employ neutral moderators, but thousands of participatory-budget deliberations and community-driven development projects employ moderators who can express explicit viewpoints.
This study thus tests the effect of moderators in a controlled field-experiment setting, where the views of moderators are randomly assigned to different discussion groups.We asked the participants (107 students of a political science class) to indicate their preferences over the number of writing and exam requirements for the course -issues that they cared about.They then deliberated in groups of about five students each.In some groups, the moderator expressed neutral views, while in others the moderator advocated for one option over another.After deliberating, the students cast final votes.The outcomes of the elections were binding on all students.We stress the binding nature of the decision because it is a rare feature in deliberative and participatory experiments.Many deliberative processes are only consultative, but in many high-stakes cases, the process is binding.We also stress that the participants were not aware that they were partaking in a study and thus behaved as they naturally would in a typical deliberative setting.
The experiment was implemented in the fall of 2007 with the approval of the Yale University Human Subjects Committee. 5The balanced nature of our experiment helped to ensure that the final results of the deliberative process would not be impacted -indeed, they were not.Moreover, the vast majority of the students reported that they favored the opportunity to participate in the decision-making process for the course requirements.Most of them recommended that we continue to use the process with future classes.No student expressed objection in the anonymous evaluations or in person.
As expected, the results of our experiment show that when moderators intervene in favor of one option, they can have a substantively and statistically significant impact on the preferences of participants.For each decision in our experiment (writing and exam requirements), participants could choose between two options (A or B) or declare indifference.Consider the basic results of the first discussion (the writing requirement).
In the control group (where moderators exerted no influence), 83 percent of the students voted for option A (3 short response papers), while 17 percent voted for option B (1 short response paper and 1 final paper); no student was indifferent.In groups where moderators influenced toward option A, the average treatment effect appears to increase the number of students in favor of option A by 14 percentage points.In groups where the moderators influenced toward option B, the average treatment effect of treatment B appears to increase the number of students in favor of option B by 20.5 percentage points.
Similar results emerge from the second discussion (the exam requirement).In the control group 6.1 percent of the students voted in the referendum for option A (cumulative final exam), and 93.9 percent voted for option B (midterm examination and non-cumulative final examination); no student was indifferent.In groups where moderators influenced toward option A, the treatment appears to have reduced the number of students in favor of option B by 19.8 percent.Where moderators influenced toward option B, no student voted for option A, and the treatment apparently reduced the number of students in favor of option A by the full 6 percentage points.
Of course, in our setting, both of our discussions had unexpectedly large majorities in favor of one option over the other, which limited the effect that our moderators could have when influencing towards the popular option.So, the effectiveness of the moderator-effect can be more precisely estimated when considering cases where the moderator influenced towards the less popular option as we show below and in appendix 3. 6 The results of our study have implications for refining epistemic conceptions of deliberative democracy and for designing more precise empirical investigations of the effects of deliberative processes on attitudes and behavior.The magnitude of our results implies that empirical studies investigating the effect of deliberation on attitude formation and behavior may need to disentangle the effect of moderators from the effect of deliberation.Without properly understanding the role of moderators, we might overestimate or underestimate the effect of deliberation.We return to this issue in the conclusion.
Additionally, this article details a relatively inexpensive methodology that can be applied to investigate other elements of the deliberative decision-making processes -from subtle moderator-interventions (including the use of body language), to the impact of providing information packages to participants, framing messages, and group composition.Future work could, for example, test for the separate effects of a moderator endorsement of one option over the other and a rational argument.For example, an excellent recent laboratory experiment by Park (2012) builds on our methodology to investigate the effect of moderators with different levels of expertise, finding, indeed, that greater expertise leads to more influence.Note that our treatment is composed of both a moderator endorsement combined with a moderator argument.Our experiment is not designed to separate these two components.Nor is the experiment designed to study the strength of different types of persuasive messages.A large literature in 6 Our experimental results measure the difference in the average post-discussion opinions between a treatment group, in which moderators expressed their views, and a "no influence" group, in which moderators did not (see section 3 for a detailed discussion and appendices 2 and 3 for a more detailed presentation of the data).By construction, the distribution of post-discussion votes in the control group dictates the upper bounds of the treatment effect and the possibility of distinguishing the effect of moderators from the effect of group discussion -that is, the precision of our estimate.In an extreme case, if there were a 100 percent majority in the control group, it would be impossible to detect any positive moderator's effect influencing toward the majority option.Thus it is not surprising that when we study the effect of the moderators influencing toward the more popular option, we find smaller and less statistically significant results than when we study the effect of moderators influencing toward the less popular option.
social psychology investigates such issues in non-deliberative settings, and our methodology could be employed to test the conclusions of this literature as they apply to deliberative democracy. 7 the first study to randomly assign moderator-preferences within a deliberative decisionmaking process, our objective is to simply establish whether moderators can impact the preferences of participants with an intervention that is similar to -although more conservative than -those we have observed in real-world settings.The most interesting feature of our methodology is that the participants behave more naturally than in laboratory experiments.
As for practitioners, who employ the tools of deliberative democracy, our results flag the importance of moderators as a potential vulnerability that interest groups may seek to exploit.These phenomena are most relevant in community-driven development projects that allocate significant resources.If a reasonable degree of disagreement existed in a community, and organized interest groups had influence over moderators, they could hijack a deliberative decision-making process.
The rest of our article proceeds by first discussing the experimental literature on deliberative democracy (section 2).We introduce the details of our experimental methodology in section 3. We then present and discuss the results in sections 4 and 5, respectively.Section 6 concludes with a discussion of the implications of our research for the study and implementation of deliberative democracy.

Experimental studies
Scholars have conducted experiments to test the impacts of various facets of deliberative discussions on the post-discussion preferences of participants.8Some researchers, for example, find that the effect of deliberation is smaller for salient issues and when the participants are better informed (Farrar et al. 2010, Luskin, Fishkin, and Hahn 2007).Others show that deliberation makes the preferences of participants more structured and orderly (List et al. 2006).Some studies have tested the effect of inequality among the participants on the deliberative outcome.Morrison and Singer (2007) find that the perception of inequality among the participants influences whether participants find the outcomes of deliberation to hold legitimacy.Other studies have investigated the effect of group pressure and group composition on participants' behavior.Farrar et al. (2009), List et al. (2006), and Luskin et al. (2007) all find that participants modify their preferences to conform to the preference of their group.Their results contrast with the psychology literature that analyzes group polarization effects (see Sunstein 2000).Karpowitz, Mendelberg, and Shaker (2012) study the voice of women in a deliberation, finding a substantial gender gap biased against women (unless there are many women or a unanimous rule is employed in settings with few women).
Sophisticated research is also emerging on strategic behavior in deliberations.In a nonexperimental study, Muhlberger (2007) finds that participants are minimally strategic.But several studies disagree with this conclusion (see, for example, Landa 2005; Hafer and Landa  2005, 2007; Meirowitz 2007; Dickson, Hafer and Landa 2008a,b; and Landa and Meirowitz  2009).For a recent study that combines game-theoretic and psychological approaches tested with an experimental design, see Myers (2010).
Pierce, Neeley, and Budziak (2008) find that neutral moderators, who do not express their views during a deliberation, increase perceptions of fairness in the procedure, confirming the important role that moderators can ideally play.In his work, Steiner (2012) considers a rich set of experiments and case studies and concludes that the style of moderator-intervention impacts the ways participants contribute to discussions.In an experiment that randomly assigns groups to deliberate with and without a moderator, Fulwider (2005) finds that the presence of moderators makes opinion change more likely.As mentioned above, Park's (2012) laboratory experiments show that moderators perceived as having more expertise have more persuasive effects on participants.
Similarly, Humphreys, Masters, and Sandbu (2006) find that group responses are correlated with the preferences of moderators.They analyze the results of a national deliberation on country-wide economic priorities organized by the UN Development Program in São Tomé and Príncipe.The moderators were randomly selected from a group of civil society leaders and public officials, with some moderators randomly assigned to more than one group.The authors infer the importance of moderators from the share of the variance explained by moderatorspecific effects.Moderator effects account for over one third of variation in the outcomes.The authors also provide evidence that opinions changed in the direction of moderator influence.Moderator preferences -which were not manipulated by the experimenters -were recorded one week after the deliberation and may thus have been influenced by the group discussions themselves, but this is unlikely as it would imply strange moderators who are both highly influential and highly malleable (Humphreys et al. 2006:598). 10 Still, in contrast to these studies, which all detect various moderator effects, Farrar et al.  (2009) find no robust differences in the policy views of participants across groups with moderators employing passive and active approaches (although in this experiment, moderators did not express their views).None of the above studies randomly assign preferences to moderators in a field experiment setting.What happens when we manipulate the views of moderators in a natural setting?Can moderators influence participants to change their views when they argue for one option over another -even if they do so in a rather limited manner?Our experiment is designed to address these questions directly by scripting the intervention of moderators.It is the first study to randomly assign the direction of moderator-influence and the only existing study to do so in a field experiment.

The deliberative decision-making process and the experimental design
The experiment took place in an Introduction to Comparative Politics course at Yale University with an enrollment of 107 students.The subjects were not aware they were participating in an experimental study. 11The course syllabus noted the following: "After studying elections, we will be holding elections in class ourselves!Students will be able to vote for alternative writing requirements and alternative exam requirements.Majority rules."The syllabus provided options for the writing and exam requirements (see table 1).Note that the outcomes of the elections were binding on all students in the class, thus the experiment represents a binding deliberative decision-making procedure.At the beginning of the semester, each student selected one of eight possible section-meeting times.During the fifth week of class, students were randomly assigned to groups within their respective section-meetings.There were a total of 24 groups with an average group size of 4.5 students. 12fore the group discussions took place, we conducted a confidential pre-survey to record the initial preferences of each participant.Students then discussed the writing requirement for 15 minutes in their groups before casting their secret ballot on the issue.Next, they discussed the exam requirement for 15 minutes before casting their secret ballot on this issue.A discussion rule was also randomly assigned to each group.Some groups had no rules, while others had "strict discussion rules," which limited students to speaking for one minute at a time and required all students to have the opportunity to speak before a student could speak twice.
At the end of the 30-minute session, the students filled out a survey, which asked standard questions about the participants' characteristics and to evaluate their perceptions of the process.
In particular, we collected data on gender, graduation class, and membership in a varsity team, which we address in the analysis below.
Note that because the writing and exam requirements were discussed and voted on separately, we observed a total of 48 separate group discussions (two for each of the 24 groups).In the results section below, we place more confidence in the results from the first discussion because the second discussion may have been contaminated by the first.In fact, however, the qualitative results hold across both discussions and are quantitatively similar.
Each of the groups had one moderator who was introduced as a graduate student. 13We randomly assigned moderators to intervene in discussions by expressing opinions (or not) over the various options."Preferences" were randomly assigned as entire scripts per discussion.So moderator preference was consistent throughout a single discussion.A given moderator could, however, be neutral (placebo) in the first discussion and biased (treatment) in the second, and vice-versa.We scripted the interventions and report exactly what the moderators said for the first discussion in appendix 1 (the script for the second discussion is available on request)."Treatment" moderators made three statements supporting their "preferred" option.The first statement was read at the beginning of the discussion; the second, halfway through the discussion; and the third at the end.The fact that the moderator would intervene in the discussions was unknown to the students ex ante.
Other than reading the script, moderators assigned to influence the students could only attempt to influence the discussion using body language, nodding their heads during the discussion when students raised points that agreed with their "preference," shaking their heads when students disagreed.Moderators assigned to be neutral also read three statements at exactly the same time intervals as the moderators assigned to influence the students, but these moderators expressed no opinion about the options and did not use body language.
The students were instructed to justify the preference they had written in the pre-survey during the discussions.This type of deliberative exercise, which stresses justification and critical evaluation, is used in many deliberative discussions.Additionally, we reminded the students that the decision was important because it would affect their coursework -the results of the referenda were binding on the entire class.
Regarding our randomization procedure, we did not generate significant imbalances in treatment assignment (see appendix 2).The pre-discussion preferences of the participants do not significantly predict treatment assignment, nor do any other individual characteristics that we investigated with the exception of membership in a varsity team.We flag this factor because varsity practice-times influenced the schedules of athletes, as a group, which influenced their section-meeting choice and may have also influenced their preferences on course requirements.We therefore present robustness checks that control for this factor in our analysis of the data below. 14s we conclude this section, we emphasize that the decisions of the participants on the course requirements were binding on the entire class.We also highlight that the participants were unaware of the experiment and of the research project.The students believed only that they were participating in these discussions and the subsequent referenda to decide the requirements of their class (which was true -and our balanced experimental design did not impact the ultimate outcomes).Moreover, all scripted interventions were pedagogically justifiable.While different students were exposed to different arguments, none of the arguments were misleading.Different students were just prompted to think about different considerations, as is commonly done in small-group class discussions.We justified the presence of surveys as an evaluation tool to help the professor decide if the process should be replicated the subsequent year.Students are familiar with this type of survey because professors introducing innovation in the curriculum often conduct their own custom evaluation to obtain data mid-semester.In short, we made the setting feel completely natural.
As we turn to the results, it is noteworthy to mention the informal feedback that we received from our moderators.Treatment-moderators reported they felt paralyzed by the script and felt this limited them from having influence.We decided in advance that scripting the interventions of the moderators would limit the effect of particularly charismatic or persuasive moderators.We also felt that using a script would make clear the level of intervention, and make the experiment easy to explain and to replicate.Moreover, we wanted to see if even minor intervention could have an effect.

Model specification and results
To analyze the results of the experiment, we employ parametric and non-parametric methods that isolate each discussion in turn.We present the descriptive data and non-parametric tests in appendices 2 and 3, and focus in this section on the results of our regression analysis, presented in table 2. Columns 1 through 4 present the results from the first discussion (the writing requirement), and columns 5 through 8 refer to the second discussion (the exam requirement).As mentioned above, we place more weight on the first discussion, as the second may have been contaminated by the first.In fact, however, the main results hold across both discussions.
Columns 1 and 5 present the results from a linear probability model that estimates the probability that a student votes for option B in the post-discussion referendum.These baseline models include only the randomized treatment -there are no control variables.This clean approach is equivalent to a non-parametric estimate of the difference in means between the treatment effect in the treatment group and in the non-treatment group (see appendix 3).
Columns 2-4 and 6-8 present the results from various regression models.We first introduce a non-linear model (probit) and then introduce pre-treatment covariates as controls in the linear model.Some scholars argue against using a regression model that includes pre-treatment covariates to analyze experimental results because it may introduce bias (for an excellent discussion, see Freedman 2008a, 2008b).In some applications, however, the inclusion of pretreatment covariates can improve the precision of the estimation of the treatment effect (Green  2009, Gerber and Green 2012).We present both approaches to test the robustness of our results. 15lumns 2 and 6 present the results of the non-linear specification (probit).We do not include pre-treatment covariates because our objective is merely to show that our results hold in the non-linear specification. 16Columns 3 and 7 present an ordinary least squares (OLS) specification that controls for the pre-discussion preferences of students.17Columns 4 and 8 introduce pre-treatment covariates that control for specific characteristics of students: gender, graduation class, and membership in a varsity team (due to the constrained schedules of athletes, as mentioned above). 18These models further include indicators for the eight time slots during which the experiment was conducted and a variable that captures the percentage of students in each group in favor of option B. 19 We include this last variable to ensure that the treatment effects are not driven by group pressure.Note that when we introduce the control variables, the interpretation of the coefficients differs subtly.In the baseline models (without controls), the coefficients indicate the effect of the treatment on the post-discussion preferences.In the models with controls for the pre-discussion preferences, the coefficients indicate the effect of the treatment on the difference between pre-discussion preferences and post-discussion preferences.
Note that the dependent variable in the second discussion is coded differently from the coding employed in the first discussion due to the presence of three indifferent students.It is coded zero if the student favors option A or is indifferent; it is coded one if the student favors option B. Thus the coefficient of treatment A shows the effect of treatment A on reducing the probability of the student voting for choice B. The effect is robust to switching the way we code the three indifferent students.

Results from the first discussion (writing requirement)
As table 2 shows, the effect of a moderator influencing toward option A in the first discussion is significantly different from zero (at the ten percent level) only in the linear and non-linear baseline models (columns 1 and 2).The effect of a moderator influencing toward option B is statistically significant across all specifications at the five percent level or stronger.
Importantly, the majority of students held a pre-discussion preference for option A in the first discussion.Thus the imprecise estimate of moderator-influence toward option A is not surprising.We simply do not have enough data to isolate the effect of the treatment A when considering the difference from pre-discussion preferences.
So, the analysis of the first discussion shows a statistically significant moderator effect with regard to treatment B that is robust to a variety of different specifications.The effect of treatment A, which is constrained by the distribution of pre-discussion preferences in our sample, is weakly statistically significant only in the baseline models.
Interestingly, but maybe not surprisingly, the presence of a strict rule of discussion, which allows all participants to express an opinion, favors a change in opinion toward the less popular option B, independent of moderator influence.Thus a strict rule of discussion seems to have had a role in protecting the minority position -at least in this first discussion.

Results from the second discussion (exam requirement)
The second part of table 2 (columns 5 through 8) presents similar results for the second discussion.In the control group, a large majority of students (97 percent) held pre-discussion preferences in favor of option B, and therefore we observe that moderator influence toward option B is not statistically significant.Moderator influence toward option A is statistically significant at the ten percent level in the probit model and at the one percent level in all of the linear specifications.The discussion rule in the second discussion has no statistically significant impact.Notes: *** indicates significance at the 0.01 percent level, ** at the 0.05 percent level, and * at the 0.10 percent level.For the probit models, we present the marginal effect for a discrete change of each indicator from zero to one (holding other indicators to zero).The numbers in parentheses are t-statistics (columns 1, 3, 4, 5, 7 and 8) or z-statistics (columns 2 and 6)."Student characteristics" control variables include the graduation class of participant, an indicator for gender, and an indicator for membership on a varsity team.

Discussion of results and suggestions for future research
Our experiment builds on the findings of previous research, which suggests the importance of moderator-influence (Fulwider 2005, Humphreys et al. 2006).We find that the expressed opinions of moderators can have an effect on deliberative discussion.The magnitudes of our estimated effects suggest that moderator influence could be decisive in a deliberative discussion.If moderators express an opinion even in a limited manner, as in our experiment, they can have a statistically and substantively significant impact on deliberation outcomes.
Future research should address a number of issues that our experiment does not.For example, we estimate smaller treatment effects in the second discussion.Given our experimental set-up, the first discussion treatment effects are unbiased -the students had no anticipation of treatment.The treatment applied during the first discussion may, however, have influenced the second discussion.We thus flag the attenuated moderator effect in the second discussion as a question for future investigation.
Our results also indicate smaller treatment effects when influencing towards the prediscussion majority preference.This result may simply be driven by the constraints of the sample, as discussed above.Still, our experiment is not designed to identify differential effects between influencing toward majority and minority opinions.An experiment designed to vary the majority opinion in the groups could address such a question.Still, our regressions (column 4 and 8) do show that our results are robust to the inclusion of controls capturing the effect of group pressure.When isolating the effect of moderators from group pressure, the magnitude is greater (see table 2).The result suggests that a moderator might be particularly effective in defending minority opinions from group pressure.
Another result of our experiment worth further consideration is the effect of a strict rule of discussion -one that explicitly invites each participant to speak once before any other participant can speak twice.In this first discussion, the rule appears to promote the minority opinion, but the finding does not hold in the second discussion.Our discussion rules were not re-randomized for the second discussion because it would have appeared unnatural to change procedures between discussions.Thus we did not investigate this result experimentally.One possible explanation is that the number of students in the minority in the second discussion was so small that the rule of discussion did not have an effect. 20nally, we investigate the interaction of moderator-gender and student-gender.Surprisingly, the interaction is not statistically significant.Karpowitz, Mendelberg, and Shaker (2012) find an effect of gender group-composition on deliberation, so we expected the gender of the moderator to matter.Yet, we did not randomize the gender of the moderator or the group composition, so we flag this question for future research.
As for the external validity of our results, we highlight several issues to consider: First, undergraduate students might be particularly susceptible to persuasion due to their young age.Some studies from the 1980s and 1990s, which examine attitude strength over an individual's lifecycle, show that age is inversely correlated with openness to changing opinionalthough other studies challenge this finding. 21When we turn to laboratory experiments on the impact of framing, studies by Nelson, Clawson, and Oxley (1997) and Kühberger (1998) do not detect different effects across students and non-students.In their study of younger and older adults, Kim et al. (2006) find that senior citizens are actually more susceptible to the influence of framing than are undergraduate students.Note also that in laboratory experiments where participants must provide a rationale for their choices, the impact of framing diminishes. 22Of course, all these studies are conducted in a laboratory with subjects who are aware of being studied. 23Thus, the relationship between age and malleability in a deliberative setting requiring justification is not obvious, and the results of the above laboratory experiments may not apply to our field experiment setting.
Second, if undergraduates perceive graduate students as experts, their preferences might be particularly malleable.Numerous studies in social psychology show that persuasiveness generally increases with communicator expertise and credibility. 24Our experience at Yale University does not suggest that undergraduate students perceive moderators as figures of authority or as experts, but we cannot exclude this possibility.Still, in community settings, moderators are often local leaders, officials, or representatives of donors, who clearly do have some direct power over the participants -or they come from non-governmental organizations and have real expertise.In contrast, none of our moderators was serving (or had ever served) as the teaching assistant for anyone in his/her group, so the moderators had no grading power over their students.Our moderators had no experience in devising class requirements, and the information contained in their persuasive statements was common knowledge.The scripts were built using information provided to all students by the professor in class.Overall, while it is possible that undergraduate students perceived graduate students as experts, we believe that in many community settings the relative perception of the authority and expertise of the moderators is higher.Ultimately, however, this empirical question cannot be tested without replicating our experiment in a community deliberative decision-making process.Third, our participants had well-defined a priori preferences over issues that they cared about and understood thoroughly (course requirements).In real-world applications of deliberative democracy, the knowledge and motivation of participants vary.In participatory budgeting, for example, only a small minority of participants understands the budgeting process.Numerous studies on persuasion in social psychology have shown that highly-motivated subjects who understand the topic of discussion are less prone to persuasion. 25If this caveat holds, our findings likely understate the impact of moderator-influence.
Fourth, we deliberately constrained the ways in which our moderators could intervene in discussions (limiting them to three scripted statements).In community settings, such restrictions are absent.As noted in the introduction, from February to April 2009, one of the authors of this study participated in the weekly meetings of the Participatory Budgeting Council in Porto Alegre, Brazil (see Spada 2012).The author observed moderators who were not neutral and even used direct threats against participants.For example, they warned that participants who criticized the city government would not receive city funding for their projects.
While our experiment does not directly address the above issues and thus questions of external validity remain, our observations of real-world applications of deliberative democracy in community settings lead us to suspect that our study underestimates the effect of non-neutral moderators in many real-world applications of deliberative democracy.

Implications for the theory and practice of deliberative democracy
Endowed with the coercive power to intervene in discussions, moderators play a crucial role in deliberative democracy.Idealized as the neutral arbitrators of deliberations, theorists and practitioners have often ignored the potential misuse of moderator power.The results of this study suggest, however, that we may wish to rethink the role of moderators, lest we confound the effect of deliberation with the effect of moderator influence in real-world applications of deliberative democracy.While our study is certainly not designed to test various means of controlling moderators, the estimated impact of moderator influence does indicate a need to consider the issue.
Our experiment also provides a methodological contribution to the pragmatic research agenda on deliberative democracy (Fung 2007).Researchers can apply our fairly simple experimental design to analyze many elements of a real deliberative decision-making process.We suggest extensions of our work, from analyzing more in detail the effect of the provision of information within the discussion, to the effect of group size and composition, the length of the discussion, and, indeed, all aspects of deliberative decision-making processes.
From the standpoint of practitioners, we note that communities around the world are increasingly turning to deliberative and participatory governance processes, where moderators engage people in direct policymaking.The approach has been advocated by the World Bank, the International Monetary Fund, the United Nations, many non-governmental organizations, and municipal governments in developed and developing countries all over the world.According to estimates by Mansuri and Rao (2004, 2007), World Bank lending for "community driven development" projects based on deliberative and participatory processes grew from US$325 million in 1996 to US$7 billion in 2010. 26holars have warned us that participatory and deliberative approaches may be vulnerable to capture by local elites (see Platteau and Gaspart 2003, Baierle 2007, Wampler 2007, and Spada  2012).The specific ways in which elites may hijack these processes, however, have not been fully explored.One potential mechanism runs through the influence of moderators.As shown by our experiment, moderators might serve as an effective but subtle tool -one that is difficult to detect.
Even in cases where there is no manipulative intent by an interest group, the recruitment methods, backgrounds, and training of moderators might all play a role in deliberative processes.Often, for example, moderators are recruited from local civil society organizations, which deal with specific social issues (for example, education).Such knowledgeable and partisan moderators might unintentionally bias discussions towards their preferred outcomes.Our scripted interventions suggest that even a seemingly minor suggestion can impact the preferences of participants.
Paying careful attention to potential moderator influence may prove particularly useful for the study of the epistemic benefits of deliberative discussions and also for studies that analyze changes in attitudes, polarization, and the formation of consensus.At present, a puzzle exists in the literature.Consider the fact that laboratory research in political psychology has found that group discussions have a polarizing effect on the attitudes of participants (Sunstein 2000).In contrast, Deliberative Polls, and other similarly structured deliberations, find that group discussions do not lead to polarization (Luskin, Fishkin, and Hahn 2007).Note that the laboratory experiments do not employ moderators, while the Deliberative Polls do.Perhaps the difference in outcomes -polarization in un-moderated laboratory discussions and nonpolarization in moderated Deliberative Polls -is indeed a moderator-driven effect.Deliberative Polls do employ professional moderators who are instructed to be neutral and not force consensus.Yet, many professional moderators might have preferences favoring consensus over polarization due to their extensive training in conflict resolution and their well-intentioned desire to provide an enjoyable experience for the participants.Such preferences might unintentionally bias the discussion against polarization.In his critical analysis of a European Deliberative Poll, Steiner (2012: 273) observes that paying the expenses of participants may further encourage a pleasant atmosphere.He concludes that a good atmosphere, reinforced by moderators who intervene in discussions to promote deliberation -plus the dissemination of pre-discussion information packages -may all work together to induce changes in attitudes and preferences.From an epistemic perspective, it is important to distinguish the effects of each of these various aspects of the experience from the inherent effects of the deliberation itself.Our research here specifically suggests greater scrutiny of the behavior of moderators in studies that aim to measure the effect of deliberation on attitude change.Researchers should seek to identify the separate effects of group discussions and moderator influence.
Our study thus aims at reducing the distances between the theory of deliberative democracy, the empirical testing of its consequences, and the actual practice of deliberative democracy in the real world.With this in mind, we conclude by highlighting simple policy recommendations that practitioners can consider to minimize the risk of manipulation by moderators and thereby strengthen deliberative processes.
First, we recommend that organizers of deliberative processes simply make their participants aware of the potential for moderator influence.The literature on persuasion shows that forewarning people of a speaker's persuasive intent can promote resistance (see Freedman and  Sears 1965, Hass and Grady 1975, and Petty and Caccioppo 1979).Highlighting the possibility of misusing the moderator position may put the conveners and participants engaged in a deliberative procedure on guard against intentional and unintentional manipulation.
Second, several studies have shown that participants in discussions can learn resistance to suggestive messages (McGuire and Papageorgis 1961, Pfau et al. 1990, Compton and Pfau 2004,  Banas and Reins 2010).Following this literature, deliberative processes might include capacitybuilding exercises that expose participants to weakly persuasive messages.Exposing participants to weak messages -such that they do not actually change attitudes -can teach participants to resist stronger forms of persuasion.This approach, of course, could make consensus-building more difficult, so we raise it with caution.
Third, we suggest assessing the preferences of moderators before conducting the deliberative procedure.In some settings, moderator influence may not be an issue.Where the moderators have little stake in the outcome of a deliberation, implementing complex moderation schemes might be unnecessary and actually prove disruptive.Where moderators have high stakes in the outcome, however, sterilizing the potential impact of moderators should take priority.Conveners could employ multiple moderators who represent different points of view and work together to balance their respective biases.We know from recent literature in political psychology that the effects of opposing framing messages can offset each other (Chong and Druckman 2013).
Finally, we recommend the random selection and assignment of moderators, a procedure that was effectively employed in São Tomé and Príncipe (see Humphreys et al. 2006).The random assignment of moderators may limit the ability of external interest groups to target them, and biases amongst the moderators may counterbalance each other.As an added benefit to scholars, increasing the use of random assignment would expand the prospects of conducting more field experiments on the role of moderator influence.discussion rule that had been randomly assigned for the writing requirement deliberation was maintained for the exam requirement.We omit the scripts for the second discussion; they are available on request.
Table A2: First discussion, opening intervention

Placebo
As the TA has said, we have to deliberate first on the decision regarding the writing requirement.You have to decide between 3 short papers, 3 pages each, and 1 short paper and 1 long paper, 2 pages and 8 pages.
In favor of three short papers Before letting you discuss, I want to share with you one little tip from my experience regarding this type of assignment.I take my job as moderator seriously and have notes: Don't be fooled by the fact that three assignments are more than two.Take into consideration the fact that writing a short paper might take less time than writing a long one.You might be able to write two good 3-page papers in less time than an 8-page paper takes.
In  In favor of three short papers At this point of the discussion I want to focus your attention on one important element: the preparation for the exam.Writing three short papers on three different topics means that you have already thoroughly prepared three major topics that will be covered in exam questions.This will give you more mastery of the subject and might help make studying for exam questions easier, regardless of the structure of exams.

In favor of one long paper
At this point of the discussion I want to focus your attention on one important element: the preparation for the exam.Writing three short superficial papers on three different topics means that you haven't mastered any single topic.On the other hand writing a long paper allows you to focus and master a topic.Thus, when the time of the exam comes, you have already prepared in depth some of the material for the final exam.This will give you more mastery of the subject.In favor of three short papers Before voting I want to (point out)/(remind you of)28 one final element: the importance of learning parsimony.When you compete for grants or make presentations, you will discover that often you are allowed to write only a couple of pages.Thus, from the point of view of pure usefulness for your future, the 3 short page papers is probably best.

In favor of one long paper
Before voting I want to (point out)/(remind you of) one final element: the importance of learning how to write a draft and then learning how to expand this draft into a proper project.This will be the process you will follow to write your senior essay and any type of other project in the future.Thus, from the point of view of pure usefulness for your future, one short paper and one long paper is probably best.

Appendix 2: Pre-discussion preferences and randomization checks
Table A5 (below) presents the pre-treatment preferences of the students and their discussion treatment assignment.As discussed in the main text, the majority of the students in the first discussion favored option A (80.3 percent), while a similar majority of students in the second discussion favored option B (79.4 percent).Note that there were five indifferent students in the first discussion and twelve in the second.
Regarding the randomization checks, we use Fisher's exact test to investigate the relationship between treatment assignment and pre-treatment preferences, as well as treatment assignment and the other covariates -see table A5. (Fisher's exact test is appropriate because we have multiple cells with an expected frequency of five or fewer.)The null hypothesis of the test we present in table A5 is that treatment assignment is independent of pre-discussion preferences.The analysis reassuringly reveals non-significant results for each discussion.Analysis with respect to the other individual characteristics that we consider -gender, age, and varsity teammembership -produces similar non-significant results.Finally, employing a multinomial logit to construct a joint chi-square test of the statistical significance of the pre-treatment covariates and treatment assignment indicates that none has a statistically significant relationship (results not presented but available in the replication materials).Again, we obtain similar results when analyzing the discussion rule, which we randomly assigned to some groups (results available upon request).Overall, our randomization procedure did not generate significant imbalances in treatment assignment.The multinomial logit does show, however, that membership in a varsity team weakly predicts treatment assignment.Thus, in table 2, we include a multiple regression model that controls for this variable.The post-deliberation preferences for the first discussion (writing requirement) are as follows.
A3.1 First discussion (writing requirement): In the control group (where moderators exerted no influence), 83 percent (34/41) of the students voted for option A (3 short response papers), while 17 percent (7/41) of the students voted for option B (1 short response paper and 1 final paper); no student was indifferent.
In groups where moderators influenced toward option A, 97 percent (33/34) of the students voted in favor of option A, while 3 percent (1/34) of the students voted in favor of option B; no student was indifferent.The average treatment effect appears to increase the number of students in favor of option A (or decreases the number of those in favor of option B) by 14 percentage points.A difference-in-means test indicates that this result is statistically significant (the 95 percent confidence interval suggests that the effect is between 0.05 percent and 28 percent).
In groups where the moderators influenced toward option B, 62.5 percent (20/32) of the students voted in favor of option A, while 37.5 percent (12/32) of the students voted in favor of option B; no student was indifferent.The average treatment effect appears to increase the number of students in favor of option B (or decreases the number of students in favor of option A) by 20.5 percentage points.A difference-in-means test indicates that this result is statistically significant (the 95 percent confidence interval suggests that the effect is between 0.07 percent and 41 percent).

A3.2 Second discussion (exam requirement):
Similar results emerge from the second discussion, but the interpretation is a bit more complex due to the presence of students who declared indifference.In the control group 6.1 percent (3/49) of the students voted in the referendum for option A (cumulative final exam), and 93.9 percent (46/49) voted for option B (midterm examination and non-cumulative final examination); no student was indifferent.
In groups where moderators influenced toward option A, 18.5 percent (5/27) of the students voted for option A, 7.4 percent (2/27) were indifferent, and 74.1 percent (20/27) voted for option B. The treatment appears to have reduced the number of students in favor of option B by 19.8 percent.This result is driven by (1) an increase in the number of students in favor of option A (by 12.4 percentage points) and ( 2) an increase in the number of indifferent students (by 7.4 percent).A difference-in-means test indicates that the result is statistically significant (the 95 percent confidence interval suggests that the effect is between 4 percent and 36 percent).
In groups where moderators influenced toward option B, no student voted for option A, 3 percent (1/31) of the students were indifferent, and 97 percent (30/31) voted in favor of option B.
Thus the treatment reduced the number of students in favor of option A by 6 percentage points.This result is driven by (1) an increase in the number of students in favor of option B (by 3 percentage points), and (2) an increase in the number of indifferent students (by 3 percentage points).A test of the difference in means shows, however, that this result is not statistically significant (the 95 percent confidence interval suggests that the effect is between -3 percent and 15 percent).Again, the non-result is partly due to the fact that an overwhelming majority preferred this outcome to begin with (79.4 percent -see table A5).

Table 2 : The effect of moderator-influence
favor of one long paperBefore letting you discuss, I want to share with you one little tip from my experience regarding this type of assignment.I take my job as moderator seriously and have notes: Don't be fooled by the fact that writing a short paper requires less time than writing a long one.Writing three different short papers requires coming up with three different ideas.Coming up with ideas is the hardest part of this assignment.If you do one short and one long paper, Prof. Vreeland will allow you to use the short paper as a draft for the long paper.This means that in the end, if you choose the option of one short and one long paper, you simply have to come up with ideas for one paper and then just write one longer paper using the same ideas.

Table A3 :
First discussion, midway intervention

Table A4 :
First discussion, final Intervention

Table A5 : Pre-discussion preferences and randomization checks
Pre-treatment preferences in the first discussion (writing requirement)