Beyond Engagement Exercises: Exploring the U.S. National Citizens' Technology Forum from the Bottom-Up

Exercises intended to engage laypeople in deliberations about emerging scientific and technological issues have become very popular in recent decades. These exercises are typically organized by political or intellectual elites, and often assessed in a top-down fashion as well. This paper disrupts that pattern by using a mix of complementary qualitative approaches to explore the experiences of citizen participants in a large exercise on emerging technologies, the 2008 U.S. National Citizens Technology Forum (NCTF), which included both face-to-face and online deliberations. Research questions explore participants’ perspectives on 1) the quality of the deliberations, 2) the potential for the exercise to have impacts, and 3) the degree of empowerment they experienced. While most participants had positive experiences in the exercise, and did not feel that anyone dominated deliberations, at times tensions and conflicts simmered under the surface. Further, the majority of the participants were highly critical of what they felt were chaotic online interactions that failed to engage with some of their key questions. Though many mentioned gaining some personal efficacy, most categorized the exercise as a research project and therefore did not feel it would have many broader societal or political impacts. Finally, participants’ reflections on their experiences in the exercise revealed interesting insights that went beyond the focal research questions—such as their awareness of the top-down power dynamics in the exercise—and how they actively negotiated these dynamics in ways that shaped the quality of deliberation, their sense of empowerment, and assessments of the exercise's potential impacts.


Introduction
Exercises to democratize science by engaging laypeople "upstream" in emerging scientific and technological issues have begun to blossom in recent decades (Bowman & Hodge, 2007;Chilvers, 2008;Einsiedel et al., 2001;Fischer, 2000;Gavelin et al., 2007;Goven, 2003;Hamlett & Cobb, 2006;Kleinman et al., 2007).Exercise organizers have employed a variety of mechanisms to structure deliberations, including citizens' juries, citizens' panels, scenario workshops, technology forums, consensus conferences, and combinations of these approaches (Rowe et al., 2008).Representing a particularly popular model, consensus conferences, developed in the 1980s by the Danish Board of Technology specifically as mechanisms to engage lay publics in decisions about emerging technologies, have been carried out in countries all over the world in the last two decades (Einsiedel et al., 2001;Goven, 2003;Grundahl, 1995;Guston, 1999;Kleinman et al., 2007).Further, in recent years, engagement practitioners and scholars have collaboratively facilitated and evaluated several large deliberative events focusing on a range of societal issues, such as AmericaSpeaks events,1 National Issues Forums,2 21st Century town meetings3 and Deliberative Polls. 4itizen deliberations on science and technology are typically organized by political or intellectual elites-and usually assessed in a top-down fashion as well.This paper seeks to disrupt that pattern by privileging and foregrounding the experiences of citizen participants in a large deliberative exercise on emerging technologies, the 2008 U.S. National Citizens' Technology Forum (NCTF).In particular, our research questions explore participants' perspectives on 1) the quality of the deliberations, 2) the potential for the exercise to have impacts, and 3) the degree of empowerment they experienced.By engaging a mix of complementary qualitative methods, we create space and flexibility for participants' experiences to come alive in our analysis-to the extent that our data speak to issues well beyond our initial research questions.Most strikingly, our data reveal participants' awareness of the top-down power dynamics in the exercise-and how they actively negotiated these dynamics in ways that shaped the quality of deliberation, their sense of empowerment, and assessments of the exercise's potential impacts.
Gaps in Research on Deliberative Exercises: Where are Participants' Perspectives?
Organizers of science and technology engagement exercises utilize diverse criteria in their research and/or assessments of the deliberations, depending on their scholarly backgrounds, goals, engagement frameworks, and access to resources.While most scholars and evaluators use assessment criteria or research strategies based, at least in part, on traditional deliberative and/or democratic deliberative ideals (e.g., related to or based on conceptualizations of good deliberation described above), some science and technology engagement researchers also base criteria on upstream technology engagement models that explicitly aim to go beyond the deliberation itself and have science policy and/or other societal impacts (Chilvers, 2008;Rowe et al., 2004Rowe et al., , 2008)).
We do not describe types of formal evaluative approaches to deliberative events here, since evaluation is not the focus of our paper.Instead, we highlight two critical research and methodological gaps in previous studies of these events, particularly those focused on scientific and technological developments: 1) little attention to participants' perspectives, and 2) reliance on primarily (or only) quantitative methods.A key premise of this study is that engagement exercise participants' perspectives on deliberations should be top priorities in research on and/or evaluation of the events, especially given that many of these exercises are supposedly intended to foster healthy deliberation among them, empower them, and give them a meaningful voice in scientific and technological decision-making (Phillips and Orsini, 2002;Powell & Colin, 2008;Rowe et al., 2004Rowe et al., , 2008)).
Oddly, few engagement scholars or evaluators explore how the lay citizen participants-arguably the least powerful but most important actors in deliberations-experience deliberation in technology engagement exercises or how they feel about the quality of the deliberations (with a few exceptions, e.g., see Delborne et al., 2011;Harvey, 2009;Powell et al., 2011a;Powell & Kleinman, 2008).Further, perhaps stemming from a desire for "clean data," researchers who do assess citizen participants' perspectives often rely on quantitative and/or quasi-experimental methodologies (e.g., numeric survey responses) that do not explore participants' actual discourse during deliberations, what happens during the deliberative process (instead they only compare opinions before and after the process has taken place), and/or nuanced perspectives and emotions of participants (Harvey, 2009).
Quantitative survey approaches, moreover, often overlook important perspectives and experiences of participants, because most survey instruments do not allow participants to express their thoughts and reflections on their own terms, but constrain them to numerically answering questions on issues deemed important by researchers (which may or may not be the most important issues to participants) (Harvey, 2009, p. 147).Yet engagement exercise participants not only help construct the actual deliberations, they also directly experience the structures and processes designed and facilitated by organizers and can reflect on the quality of these processes, their roles within them, and the power dynamics among involved actors.Participants are also aware of various aspects of the institutional, societal, and political contexts of the issues at hand, and can reflect on potential impacts these exercises might have within these contexts.Consequently, they have invaluable insights to bring to the process of evaluating deliberative exercises.Accordingly, we argue that citizen participants' perspectives on deliberations should be prioritized by deliberative scholars and organizers in research and evaluations-and that grounded, qualitative approaches are ideal for exploring their perspectives.
In this light, we focus this paper on citizens' perspectives on their experiences in a large deliberative exercise on an important set of emerging technologies.Our analysis is not intended to be a formal evaluation of the exercise, but rather, an in-depth look at participants' experiences in the NCTF and their assessments of the exercise in relation to several of the exercise's goals.Using a combination of primarily qualitative methods, we explore the following broad questions based on some of the exercise's central goals: 1) How did participants assess the deliberative quality in the NCTF? 2) Did they expect the exercise to have any impacts?and 3) Did they feel empowered by participating in the exercise?
Case Study: The U.S. National Citizens' Technology Forum In 2003, the United States 21st Century Nanotechnology Research and Development Act mandated that nanotechnology research and development integrate "public input and outreach…by the convening of regular and ongoing public discussions, through mechanisms such as citizens' panels, consensus conferences, and educational events..." (U.S. 21 st Century Nanotechnology Research andDevelopment Act, 2003, p. 1480).This legislation has been an impetus for the organization of citizen engagement exercises throughout the United States, including the exercise that is the focus of this study, the 2008 National Citizens' Technology Forum (NCTF).
Funded by the United States National Science Foundation (NSF), the NCTF took place in March, 2008 in six cities across the U.S.: Atlanta, GA; Berkeley, CA; Durham, NH; Golden, CO; Madison, WI; and Tempe, AZ.Deliberations addressed human enhancement applications enabled by converging technologies such as nanotechnology, biotechnology, information technology, and cognitive science (NBIC).The project was led by the Center for Nanotechnology in Society at Arizona State University (CNS-ASU).Two professors at North Carolina State University took lead roles in organizing the event and developing background materials.The exercise involved 352 citizens surveyed before and after the event, including the eighty-six citizens who actively participated in deliberations at six sites across the country (seventy-four completed the entire process from start to finish).The Madison group, for which the authors of this paper had responsibility, included 14 of these 74 people.Participants who took part in the entire exercise were paid $500 each for their participation in the threeweek event (in addition to filling out pre-and post-surveys).The five content experts were each paid $1000 for participating in one of the online sessions.
Face-to-Face (F2F) and Keyboard-to-Keyboard (K2K) Deliberative Sessions The exercise included four all-day in-person or face-to-face (F2F) deliberative sessions (spread over two weekends) at the local sites, as well as nine two-hour online or keyboard-to-keyboard (K2K) deliberative sessions, held on weekday evenings with experts and citizen participants from all of the sites.In-person (F2F), which were 6-7 hours each (including lunch/snack breaks), were facilitated by professors, graduate students or postdoctoral researchers from universities at each site (referred to here as "local site facilitators"). 5Online (K2K) sessions were facilitated by professional moderators and national NCTF organizers from North Caroline State University who did not participate in any of the local F2F sessions.
On the first weekend, participants met with the local site facilitators for the first F2F sessions, which were comprised of structured and facilitated group discussions (sometimes the whole group, sometimes broken into smaller groups).They were introduced to the NCTF project and asked to share their comments, reactions, and questions about the process, the issue at hand, and the background information that was sent to them beforehand.
During the nine following K2K sessions, conducted in an online interface ("Elluminate Live!") that resembled a chat room with some additional features, participants from the different sites generated and prioritized questions to pose to experts, and then chatted with the latter about these issues. 6Given the large number of participants, only sub-groups of participants were "chat-active" during any given online session-meaning that participants experienced a majority of the online deliberations as observers who had the occasional opportunity to participate in an online poll.The online sessions were planned and structured by the national organizers and co-facilitated by the national organizers and two moderators familiar with the software (for a detailed analysis of the K2K sessions, see Delborne et al., 2011).
During the final F2F weekend at the end of March, each local group met again to discuss and formulate their own recommendations on NBIC human enhancement technologies.These final discussions were structured and facilitated by the local site facilitators.Following the usual consensus conference model, the Madison group, along with each of the other local sites, produced a final report with their consensus recommendations during their final F2F sessions. 7National 5 The first and third authors of this paper were independent observers (not funded through the project) and played no role in the design of the event, facilitation of the sessions, or interviews with participants.Although they were present in all of the in-person and were online for the K2K sessions, they did not participate in the discussions.The second author was funded (through ASU) as a postdoctoral researcher and facilitated Madison meetings as well as doing both pre-and post exercise interviews (along with Ashley Anderson, a graduate student and Professor Daniel Lee Kleinman). 6See more about the online platform used here: http://www4.ncsu.edu/~pwhmds/discussion_room.html(last accessed Oct. 4, 2011).Transcripts of online discussions are available at: http://www4.ncsu.edu/~pwhmds/online_session_notes.html(last accessed Oct. 4, 2011) 7 The final reports from each of the six sites are available at: http://www4.ncsu.edu/~pwhmds/final_reports.html(last accessed Oct. 4, 2011) organizers compiled these site reports into a final report made available to federal decision makers, researchers, and media several months after the event (Hamlett et al., 2008).

NCTF Goals and Evaluation
NCTF organizers based the structure of the deliberative processes on a combination of several interrelated frameworks and goals.Firstly, the exercise was a Citizens Technology Forum (CTF) process, based on key elements of the Danish Consensus Conference (CC) model (Hamlett & Cobb, 2006).Both the CTF and the Danish CC are partially conceptually rooted in Habermasian ideas (Habermas, 1984).Reflecting Habermasian goals of ideal speech and equal power, the NCTF Handbook describes the project's goals for deliberations as "informed, respectful reason-giving among participants who have equal standing-social, political, and informational-to speak" (Hamlett, 2007, p. 4).In other words, organizers conceived of healthy deliberations as calm, cognitivelybased processes in which there are no significant knowledge or power inequities among participants.
Further, lead organizers structured the exercise in attempt to avoid what they called group "deliberative pathologies."Texts in NCTF background materials (and lead organizers' publications supporting their approach) state that they expect ordinary people to be more subject to emotional, social, and intellectual errors in their thinking that are likely to be accelerated in group interactions, resulting in pathological "polarization cascades" and, ultimately, incorrect decisions (Hamlett & Cobb, 2006; also see Kuran & Sunstein, 1999and Sunstein, 2002, 2005).These polarization cascades are likely to occur, they propose, "when individuals holding the minority opinion in a group adopt the majority opinion for normatively undesirable reasons after deliberating" (Hamlett & Cobb, 2006, p. 631).They argue, however, that while polarization and other pathological patterns in group deliberative processes normally happen, they can be "held at bay by manipulating key facets of the deliberative environment," including "the operational structure of the deliberations" (Hamlett & Cobb, 2006, p. 632).In other words, if they are properly structured and facilitated by experts, pathological deliberative processes and outcomes can be avoided.
The NCTF exercise also had a central research component.National organizers articulated in background materials that they wanted to see if citizens, with proper facilitation, could "master the give-and-take of deliberative exchanges," noting that if they could not master deliberation, then democratic calls for citizen engagement in technology must "fail on the grounds of public incompetence" (Hamlett, 2007, p. 2).Organizers developed a large quantitative survey with numerical response options to evaluate this goal and to obtain public opinion data.Pre-and post-survey questions, assessing basic definitional knowledge about NBIC, as well as opinions, feelings, and attitudes on a variety of issues related to NBIC, were designed largely to evaluate whether or not polarization cascades occurred among citizen participants and to assess variables such as risk perceptions and attitudes, political beliefs, and political efficacy.In many ways this aspect of the exercise was similar to a deliberative opinion poll.
Finally, the exercise aimed to have some impacts on political and societal decision-making.Online background materials state that "The report will be widely circulated to government, industry, and to the general public,"8 and the NCTF Handbook notes that "We hope to provide decision makers-in the government, in business, and in society generally-with the informed, deliberative opinions of ordinary people who have taken the time and effort to study the issues carefully" (Hamlett, 2007, p. 2).
Final reports written by national organizers indicate that organizers felt successful in recruiting representative, ordinary people for the exercise, and in creating healthy deliberation and avoiding polarization cascades (Hamlett et al., 2008).National organizers based these assessments primarily on knowledge, attitude, and risk perception changes documented in quantitative survey data and the creation of consensus recommendations by each site participant team.They did not directly engage with or interview participants, analyze actual deliberations, or assess whether the deliberations had any policy impacts (as far as we know).Though citizen reports were presented by organizers to a group of U.S. policymakers, researchers, and media in a meeting in Washington D.C. later in 2008, it is not clear whether reports or other NCTF results were distributed more widely, or whether organizers felt the reports (or the event itself) affected policy or other societal decisions in any way.

Research Questions and Methodological Approach
Our research was structured by three broad questions related to some of the goals of the NCTF (described above), but we used methodological approaches focused primarily on participants' assessments of issues related to these goals.Our research and interview questions are outlined in Table 1.To address our research questions, our methodological approach combined several qualitative strategies.In line with a qualitative field approach, our methodologies were complementary, flexible, and reflexive/iterative (Babbie, 1998).While we developed general research questions up front, we remained open to new questions that emerged during our data collection and analysis.We describe the different methodologies we used in more detail below.

Review of NCTF Background Materials & Final Reports
As researchers and/or participant observers in the exercise, we had full access to NCTF background materials and reports.To understand the conceptual basis and goals for the exercise, as well as organizers' assessments of it, before and during the event each of us read through all background and preparatory materials written and disseminated by the national organizers at least once and discussed them together.After the event, we located and read presentations and reports written by NCTF organizers summarizing findings from the event, identifying texts and conclusions that were most relevant to understanding the goals, rationales, and evaluations of the exercise and to our central research questions.

Semi-Structured Interviews
Our primary methodological approach was to conduct face-to-face, semistructured interviews with all of the fourteen Madison NCTF participants both before and after the exercise.Interviews were done by the local site facilitators, including the second author of this paper.Semi-structured interviews, commonly used in participatory and field research, allow a flexible, conversational approach in which themes and ideas that emerge from perspectives and questions of the people interviewed can be fully explored (Babbie, 1998).
Our interviews included a broad range of questions developed by the Madison research team in service of this paper as well as several other related inquiries (Delborne et al., 2011;Kleinman et al., 2011;Powell et al., 2011a).Appendix 1 lists the full set of questions; focal questions for this study are listed in Table 1.Questions were first asked using the wording from the survey interview protocol, but if respondents appeared puzzled by a question, it was reworded as needed.Though some of our focal questions could be answered with "yes" or "no," all answers were followed by open probes (Why?Why not?Please explain; why do you feel that way? etc.) to encourage participants to explain and elaborate to whatever extent they wished.In addition to the focal questions, interviewees were asked several relatively open-ended questions and were allowed to elaborate and diverge from any questions as much as they wanted, and/or to ask questions themselves.They were also encouraged to discuss any issues that came up during interviews, whether or not they were directly related to questions.Immediately after the interviews, interviewers recorded their observations and thoughts about the interview process.

Analysis of Interview Data
All interviews were audiotaped and transcribed verbatim by a professional transcriber.In analyzing written transcripts, we focused first and primarily on answers to our research questions, while also looking through entire transcripts for participants' comments related to our focal questions.Analysis was carried out in three stages: 1. Coding yes/no answers.We coded answers as yes/no (for questions that were answered as such).When participants did not clearly say "yes" or "no," they were coded as "equivocal." 2. Explanations for yes/no/equivocal answers.Participants' explanations for why they said "yes" or "no" or were equivocal were examined, and we took note of explanations and connections repeated by more than one participant, as well as ideas that were clearly unique, different or divergent from others.3. Finding/connecting interview comments related to focal questions.We read through participants' answers to all interview questions, to locate comments directly or indirectly related to our questions.4. Identifying interpretive repertoires.After identifying and agreeing on yes/no/equivocal answers, we re-read the transcripts several times, identifying repeated or common (made by more than one participant) explanations for answers or common connections made among issues-or in other words, common interpretative repertoires.Interpretative repertoires are "basically a lexicon or register of terms and metaphors drawn upon to characterize and evaluate actions and events" (Potter and Wetherell, 1987, p. 138; see also Gilbert and Mulkay, 1984).While looking for common patterns in types of interpretative repertoires (e.g., "I didn't feel heard and comfortable in the online sessions because of X, Y, Z," etc.), we were also highly attentive to differences and exceptions among these common repertoires, and/or inconsistencies within individual's repertoires (Potter & Wetherell, 1987).In some cases, more than one participant drew connections among similar aspects of the exercise in answering questions that we hadn't anticipated as being relevant to our focal questions, or mentioned them when answering other questions (not our focal questions).Rather than discounting these as not relevant to our study, we documented them as well.The issues we highlight and discuss in our results section below are the most common interpretive repertoires we identified, as well as deviations to these common repertoires that we felt were most relevant to our focal questions.

Participant Observation
Another central methodological approach we utilized was participant-observation, a method commonly used in several social science fields that allows researchers to gain a close familiarity with a group of people and the processes and events they are experiencing, while participating directly in interactions among them (DeWalt et al., 1998).Our participant observation included the following components: 1.All three authors (particularly the second author, who was funded as a researcher on the project) were involved in design of and preparation for the Madison components of the exercise (F2F), including developing research questions, planning the processes during the deliberative sessions, and planning logistics (food, equipment, etc).2. The second author interacted before, during, and after the event with national NCTF organizers while coordinating the Madison components of the exercise.3. The second author was responsible for selecting Madison participants, communicating with them, and organizing and facilitating all of the Madison F2F sessions.
4. The first and third authors, not funded through the NCTF project, did not facilitate or actively participate in discussions in the F2F sessions, but were present for all F2F sessions-observing, taking notes, and helping with organizing breakout groups and logistical support. 5. None of us had any role in organizing or facilitating the K2K sessions, but one of us logged on to each of the K2K sessions, so that one of the Madison site team members was online for each session.Once online for the K2K sessions, we observed interactions and took notes, but did not moderate in any way or actively participate in the discussions.After participating in F2F and/or K2K sessions, we wrote notes and summaries of our impressions of the interactions and compared them among the team, highlighting interesting themes and issues relevant to our research questions.
Participants and national organizers were aware of our roles in the exercise and that our involvement was part of a research project.Our involvement in all aspects of the exercise gave us a depth of understanding of the processes included in the exercise and what the participants went through.

Analysis of Online (K2K) Transcripts and Quantitative Data
In addition to one of us participating in each of the online sessions, which gave us firsthand experience of what the sessions were like, we had access to transcripts from all of the online sessions.Transcripts included all postings to the online deliberation, including those from participants and moderators.While we originally were not intending to use the online transcripts for this study, in line with our reflexive and iterative approach, we went back to these transcripts to find examples of interactions that illustrated problems that were repeated by several Madison participants in their discussions of the quality of the online deliberations.In other words, participants' common interpretive repertoires and repeated comments about particular issues in the K2K led us to consider the K2K transcripts as important data for this study.
Our analysis of K2K transcripts of interactions among participants from all sites, along with our first-hand experience (participant observation) of the sessions, gave us considerable insights and data on the quality of the K2K deliberations, as well as the challenges and constraints of the K2K process.Participants also commented on the K2K sessions to us via email, and in group feedback discussions in later F2F sessions, as well as during casual interactions outside of the formal deliberations in the F2F sessions (e.g., snack breaks, etc).
We also had access to the data collected through online quantitative surveys administered to all 86 NCTF participants (74 after the event) across all sites by national organizers.We drew on the survey data minimally for this study-e.g., when discussing demographics and representativeness of the participants, and in discussing quantitative changes in participants' risk perceptions and internal and external efficacy before and after the event (to compare to our qualitative assessments).

Methodological Limitations of the Study
While we had access to national NCTF survey data and online transcripts from the entire participant group, neither national organizers nor researchers who organized sessions at other F2F sites conducted interviews with their participants, precluding our ability to perform consistent comparative analysis across sites.9Consequently, we need to be cautious in generalizing our findings, based primarily on interviews with 14 Madison participants, to the entire NCTF participant group.We recognize two limitations explicitly.First, the small sample size (N=14) of interviews compared to the whole population of NCTF participants (74 participants finished the exercise, including the Madison participants) creates the risk of sample bias.Secondly, because our interviews were conducted only with Madison participants, any unique characteristics of the Madison group or the Madison NCTF deliberations would dampen our ability to generalize to the NCTF as a whole.
Demographic and knowledge/attitude data from the NCTF survey of all participants reveal similarities and differences between the Madison group and participants at other sites. 10The Madison group, on average, had higher education and income levels and was more knowledgeable about NBIC technologies than participant groups at other sites-and the whole NCTF group was more privileged on average (on these variables) than the U.S. population.Specifically, several Madison participants had graduate degrees, science backgrounds or professions, and facilitation experience.In addition, the Madison group on average was much more positive about benefits of NBIC technologies and less worried about risks after the exercise than participants at other sites (Powell et al., 2011a).
With more education, knowledge, facilitation/group work experience, and positive feelings about science and technology, Madison participants may have been more confident in engaging with and posing questions to facilitators, moderators, and experts.Their knowledge and confidence may have made them more openly critical than other site participants.On the other hand, we suspect that groups at other sites, with somewhat higher proportions of less privileged people, were probably at least as, if not more, critical of the F2F and K2K sessions than Madison participants were.Comments in the K2K transcripts support this speculation, suggesting that other site participants' assessments of the K2K portion of the exercise were at least similar in valence (negative) to those of Madison participants (for a more thorough analysis, see Delborne et al., 2011).
Survey data also reveal that there was a significant decline among all NCTF participants after the exercise in preference for F2F deliberation as compared to K2K deliberation-suggesting that other site participants also negatively assessed the online portions of the exercise.Survey data showed an overall decline among the whole participant group in the sense that the exercise would have political impacts after the event, paralleling our results.However, again, we have no directly comparable data on how other participants felt about deliberations in their individual site F2F sessions, the potential for the exercise to have any political impacts, and/or how empowered they felt after the exercise.So, on those questions, comparisons with the whole group aren't possible.Nevertheless, we believe that in spite of these limitations, our rich data from the 14 Madison participants provide key insights into the experience of the NCTF deliberation.
Lastly, a challenge in comparing and interpreting participants' perspectives on the F2F versus the K2K sessions was the very different relationships lead NCTF organizers had with participants compared to our faceto-face engagement with them as site facilitators and/or participant observers.Participants engaged closely with us (and other local site facilitators) for several days over a period of five weeks, building relationships and trust with us and other facilitators.Related to this dynamic, because interviews were done by two of the Madison site facilitators (including the second author of this paper), there could have been some "social desirability" issues-e.g., the participants may have told interviewers what they thought they wanted to hear.Also, having developed rapport with the facilitators during the F2F sessions, it is possible that participant interviewees were less critical of them and/or of the F2F processes.In contrast, participants engaged with national organizers and moderators only for a couple hours at a time via a text-based, online medium-one that they found highly problematic.These factors probably made them more critical of the online process.

Results
Research Question 1: How did participants assess the deliberative quality of the exercise?
Participants' perspectives on deliberative quality in face-to-face sessions (F2F) Overall, most Madison participants were very positive about the face-to-face (F2F) sessions, noting that these sessions were successful in encouraging calm and respectful deliberations.Many said that they learned about NBIC and about deliberative processes and enjoyed interacting with other people.Comparing perceptions of the F2F with the K2K, more people said they felt heard and comfortable in the F2F sessions than in the K2K, and slightly fewer said they felt that anyone dominated in the F2F than the K2K (see Table 2).A previous analysis of the exercise also revealed that after the exercise, the majority of the Madison panelists, as well as the whole participant group, reported that they preferred the F2F over the K2K, even though their expectations before the exercise were reversed (see Delborne et al., 2011).Affect your personal efficacy as citizen? 13 5 5 2 Willing to participate in another CC? 0 8 5 11 One audio recording was lost, making it impossible for us to code all 14 participant interviews.
12 Answers were put in this category if participants qualified answers with statements such as "Yes, I would, if…" or "Sure, depending on…" or said both "yes" and "no" in the same response. 13One missing answer for this question Many Madison participants directly or indirectly attributed the calm deliberations and their comfort during the F2F sessions to the skill of the local facilitators in structuring and managing the conversations.For instance, several mentioned that they felt apprehensive going into the exercise about conflicts and/or strong individuals dominating the discussions, but that the good facilitation made them comfortable in this regard.Some were explicitly attentive to the ways the local facilitators' careful guidance of the processes assured this level of comfort-at times noticing potential conflicts simmering just under the surface that were avoided by careful facilitation, the structure of the processes, and having a cooperative group of people.One Madison participant commented: I thought that people were receptive and it was a very relaxed environment.There were not a lot of combative personalities.That was sort of a fear going into it that these, four or five people who just sort of dominated the whole thing…I think we had a good group.I've been on committees and there is so much paralysis that can occur and disagreements, and people take things personal and they get defensive or passive aggressive.Then they do things that sort of block the efforts of the group.…Youknow, you guys deserve a lot of credit for sort of keeping us on track, sort of kind of nip it in the bud before things could escalate.I think before there were flashpoints, but they were diffused before they could become bigger things (Madison panelist 4).
There were exceptions and interesting nuances, however, in participants' comments related to the perceived quality of deliberation.Several participants mentioned being uncomfortable with conflicts and/or avoiding conflicts by selfcensorship, and noticed that others did that as well: There was definitely a conflict.It seemed like people held back a little bit, I don't know, I could tell more from peoples' expressions when they didn't say anything...or when they were upset with what someone else was saying.I don't know, I didn't engage in any conflict.I try to stay away from them (Madison panelist 13).
Further, several Madison participants had either facilitation experience and/or past experiences in structured group-work, and comments indicate that they inherently understood deliberative "rules."In some cases, participants' awareness of the limited structure of the event led them not to bring up issues, including some they felt strongly about.For example, one person said: Because of lack of time…there were times at…the end of the last day or towards the middle of the last day that if things had come up at another time in the process I would have spoken up.Perhaps even vociferously about them but…there wasn't any point to it at that point…I mean people could have been convinced but not in the time available (Madison panelist 1).
While most participants who said they held back opinions seemed to think that holding back their opinions was good and helped facilitate a better deliberative process, there were also hints that this was at times frustrating, especially when they didn't say things they wanted to say: [Interviewer: So you kind of held back when you had really strong feelings?]Yes I did.If I had a really strong feeling, it's something I learned a long time ago… to keep back and see if it doesn't come up, if my point doesn't come up I would try and stick it in at the end…there are some things I did not bring up that I could have.But just to keep the overall pace and time to get things done I was satisfied that we were still moving along (Madison panelist 7).
The following participant explicitly mentioned not talking about certain issues he knew a lot about in order to not dominate-but he also felt frustrated by not sharing what he knew: I have done some reporting on it previously and it was just interesting to see what people had thought about it.[Interviewer: So in a sense you were sort of wearing three different hats, you are a citizen, you are a reporter, and an expert in a certain sense of having a lot of extra knowledge.Did you feel like one of those roles was more dominant during the process or at different times?]I like to think, I approached it as a citizen…I would try to keep the conflict of interest off and in perspective…I think I had to sit back and realize a lot of people really don't know a lot about this...I think the first thing I got was kind of interesting hearing where people were at.Because I mean, the last thing you want to do is sort of toot your own horn and sort of sound like a know-it-all or whatever….I guess it's just frustrating when you have a certain level of knowledge on something and have a discussion with people who don't have the same level of knowledge (Madison panelist 4).

Participants' perspectives on deliberative quality in online sessions (K2K)
In contrast to the in-person sessions, most participants were very critical, and in many cases explicitly negative, about the quality of the deliberation in the K2K sessions.In a more focused analysis of the K2K sessions, Delborne et al. (2011) found three troubling aspects of the online sessions: lack of coherence, limited participant autonomy, and low degree of engagement.Panelists felt that the online dialogues overall were chaotic and confusing (they were not coherent); they had limited voice in the structure or direction of the discussions (they had little autonomy); and they felt much less engaged in the K2K than they were in the F2F (degree of engagement)-which they attributed largely to the incoherence and lack of autonomy.
We focus in this paper on a particular subset of Madison participants' assessments of the online sessions, in line with our research questions: Did people feel anyone dominated in the online sessions, and did they feel heard and comfortable?Interestingly, despite their mostly negative assessments of the online sessions, over half of the participants said they felt heard and comfortable in these sessions (or were at least partially affirmative on this answer), and the majority said they didn't think anyone dominated.According to a couple of panelists, that was the case because the structure assured equal participation.For example, one panelist observed: "The six groups were divided…so they had almost equal participation from all the states….So I thought that was well done too, I mean the division" (Madison panelist 6).
On the surface, the perceptions that no one dominated in the K2K because of careful structuring could be seen as an indication that the discussions were nonpathological, in line with deliberative ideals that informed the event.However, looking at participants' full explanations for their answers, and extensive comments during F2F feedback sessions after the K2K portion, it is clear that most panelists did not feel that the online deliberations were healthy.Several noted, in much less positive tones than the panelist cited above (or in an ironic tone), that the highly segmented and yet chaotic structure made it impossible to dominate.One said "it would be hard to dominate when you're only chat-active one-sixth of the time" (Madison panelist 1), and another answered: "No because it just wasn't possible with the online part, there was no way you could…there was no conversation as far as I was concerned, there was no way to dominate" (emphasis added) (Madison panelist 5).In sum, according to most of the Madison participants, even though the highly structured online processes seemed to assure relatively equal participation in which no one dominated, this did not result in healthy deliberation (again, for a more detailed analysis, see Delborne et al., 2011).

Participants question-and at times resist-top-down facilitation
The majority of the Madison participants tended to attribute problems in the K2K sessions to the ways they were structured and moderated, rather than to other participants.While many panelists realized that the quality of the online discussions suffered in part from the awkward and often chaotic online format, as well as unexpected technical glitches, many associated the problems with the topdown decisions made by the national organizers.In line with the "lack of autonomy" noted in Delborne et al. (2011), they felt that they had little voice in the structure or direction of the online discussions.One noted, for example: "Since the plan was already in place there was no going back at that point.There was no saying, 'Let's consider other ways we might do this'" (Madison panelist 1).
Even more problematically, several Madison participants noted that at times moderators didn't show interest in their questions, didn't answer them, or tried to deflect them.Indeed, online transcripts include repeated instances in which citizen interests were pushed aside by online moderators.Our observations of the sessions confirm this.In particular, participants from nearly every site brought up issues related to environmental, public, and workplace risks.During online sessions two and three, participants were brainstorming key focal issues for experts to address, resulting in over 40 comments or questions from participants (at all sites) related to environmental, health, and/or safety issues, including the following questions: How might workers be exposed to nano-sized particles in manufacturing and how might these particles interact with our human bodies?(Tempe panelist) We already know that some are toxic, i.e., carbon nanotubes…can we not assume that other technologies will have the same toxicity?(Golden panelist) My focus is on environmental waste…where is it going?Off shore, foreign country, or our neighborhood?[And again later, to moderators]: Waste!!!!!!! [exclamation points in original].We are running out of places for our current waste.(Durham panelist) Yet, despite numerous other comments and/or questions about environmental health and safety issues, moderators explicitly stated many times that these issues, and especially those related to products on the market, were too broad or "off topic" from the focus on human enhancement NBIC technologies.
Interestingly, several participants from different sites actively resisted attempts to take environmental health and safety concerns off the table, drawing on their knowledge of the issues and outside reading about nanotechnology developments, as this excerpt from an online discussion illustrates: I still am concerned about environmental impact.Printers now spew nanoparticles; what else is known to produce nano?(Berkeley panelist) Moderator: Remember-we are specifically talking about nano for NBIC technologies, so things like stain-resistant pants are out of the discussion.
What are the current benefits and problems we are aware of with current nanotech?Products on the market?According to the nanoproject there are over 400 products using nanotech as of this date in the U.S. (Madison panelist) Following this exchange, other participants chimed in, agreeing that we should address current products, learn from negative effects so far, and "extrapolate to the future."In the next session, participants from all sites continued to raise questions about a variety of environmental and toxicity issues (e.g., "What safeguards are in place to protect humans and the environment?""What wastes are generated from NMs? Can they be recycled/removed from environment/body?What is lifecycle?") 14Regardless, moderators continued to persist in trying to deflect and/or eliminate these comments/questions.
Eventually, some participants began supporting each other in trying to raise these issues and related environmental justice issues.At one point, about midway through the session, a Native American participant raised the following question, which then led to a long interchange about who is affected by environmental health issues-and who gets to make decisions about them: Tempe panelist, D: As a member of a sovereign government that does not have adequate representation in Congress or at any governmental levels for that matter but holds over ¼ of the land mass of Arizona and only 1.5% of the US population and believes that nanotech is harmful to the planet…how do we get heard?How do we even get anyone to listen to us when we say we do not want this in our hospitals…many tribal clinics and hospitals rely on traditional medicines from the earth…what happens when we can no longer grow our medicines to treat our people because the land has been polluted?
Madison panelist, M: Interesting question, [D].I think experts can't help us on that one.Well, maybe a grassroots initiatives expert.
Tempe panelist, D: I just want ALL people of all races, religions, cultures, etc.…to have a say in what happens….It's always the same people that make the decisions and policy for everyone.
Madison panelist, M [many comments later, trying to bring environment up again]: The environmental issue isn't up there in any direct form yet.We could ask someone about the ways they know of so far that nanotechnology specifically and generally might harm the environment… Moderator: [M], the environmental impacts of nano are important, but isn't that a bit off the topic of human enhancement?
Participants then continued to resist the moderators' insistence that environmental issues were off topic.For example, after the moderator's comment above, the Tempe panelist continued to raise the environmental justice issues: "I think it is very important…without a healthy environment to live in…it doesn't matter how much nanotech there is if there's no one here to use it."(Tempe panelist, D).Supporting this perspective, another said, "Environment is off topic in a way.At the same time, I don't see how we can analyze the impact of enhancements only in relation to ourselves.That seems like it's been an error of the past all too often" (Madison panelist, M).
At many points in these interactions, frustrations and tensions among participants and moderators seemed high (evident to us as online observers, and in transcripts of online interactions).Eventually, some participants gave up trying to raise these points despite encouragement from other panelists.For example, later in session five, shortly after the exchanges above, the Tempe panelist who originally had raised the issues related to Native American sovereignty, said "I give up trying to explain…" to which another respondent said: "Don't give up, [D],…without a bed where do we sleep?…keep going girl" (Madison panelist, M).Just after that, the moderator said, "What you should be doing now is winnowing down to 5 Qs"-after which the panelist concerned about Native American sovereignty noted, "It's the same old stuff…the majority decides for everyone…" A Madison panelist finished her sentence, "…and the minority is screwed." Participants' Perceptions about NCTF Information Related to the issues above, a number of Madison panelists felt that the background materials framed issues in very narrow and futuristic ways that limited the range of discussions during the online sessions and/or contributed to the confusion about whether environmental health and safety issues were relevant to NBIC.
In one of the first few online sessions, for example, when participants were brainstorming and outlining priority questions for the experts, several participants repeatedly asked for a timeline of NBIC developments: I am finding the attempt to "prioritize" [questions to pose to experts] frustrating.The background material that we received deliberately, I think, blurred the lines between the immediately possible…and the piein-the-sky…I strongly endorse the positions of (names other participants), that we should be hearing sooner, rather than later, from experts who can put the developmental timeline of n[ano]-tech into perspective for us (Durham panelist).
Participants went back and forth in the online sessions about whether NBIC includes products currently on the market or not, and again stressed that the background materials didn't make this clear.For example: "Yes, I'm kind of frustrated that just getting this kind of background info on the current state of development and regulation…seems like a lot of these questions should have been covered in our initial background materials" (Madison panelist).In response to this comment the moderator said: "One problem with 'current state' discussions of NBIC is that most of the technologies of human enhancement simply don't yet exist, or are in laboratory stage development at most." Yet many Madison and other site participants (based on transcript comments) knew from personal experiences and/or research during the weeks of the NCTF that some NBIC applications were not entirely futuristic, and depending on how they were defined, a few were already on the market.A participant at one site claimed to have a nanotechnology-based implant.Others looked up information on their own and a few accessed existing databases about the hundreds of nanotechnology-based products on the market.One participant noted several times that he found it problematic that the background materials only mentioned one federal agency (the FDA), when many other agencies had been involved in nanotechnology research and/or environmental health and safety issues for years: "What about HHS? NSF, NIH, CDC, DOD…?" (Madison panelist).This participant also felt that NCTF participants should draw on the perspectives of the numerous federal agencies, think tanks, watchdog groups, etc., that had been debating nanotechnologies for years before the exercise, noting that "they might be our best resources for finding out about the most significant concerns that are out there now, and possibly also recommendations about how to limit or regulate NBIC…which seems to be a pretty significant concern of the group" (Madison panelist).
A number of Madison participants, moreover, expressed disappointment and/or confusion in follow-up interviews about the fact that environmental and health risk issues were barely mentioned in background materials.Some learned about them for the first time during the NCTF discussions, which motivated them to pursue more information: Through my own reading I learned about a lot of the safety concerns.I feel like a lot of the reading we had didn't cover in general what we talked about.…whenI looked at the websites and saw all these safety issues that affected my thinking on what policy recommendations…but that was only for our very last session.If I had been reading this stuff all along I think I would have had a different sort of mind set about nanotechnology I guess.Because it made me a lot more cautious about it.Like beforehand, when this expert was like, "Oh, don't worry about it, the scientists are taking care of all the safety issues," I was like, "Oh, okay" like I'll trust this one person.Then after reading this, oh, okay, we should be cautious here.I don't know, it would have been helpful to be reading the stuff all along (Madison panelist 13).
Another panelist who knew a lot about the environmental health and safety issues coming into the event felt that the futuristic approach was too abstract, and a distraction from more immediate safety concerns: It would be helpful for them to have some sort of concrete examples of things that are going on today [in the background materials]….You know, it is, it's really abstract for a lot of people….To me there are more immediate things that need to be addressed before we need to worry about if our brains are going to be hooked up to wires (Madison panelist 4).
Research Questions 2 & 3: Did participants expect the NCTF to have any impacts?Did they feel empowered by participating in the exercise?
We discuss our second and third research questions together because our data and experience suggest that they are interconnected, and in some cases complementary.Participants who expected the exercise to have impacts tended to feel more empowered by it, in part because of their confidence that their collective efforts might have some impacts.
Unfortunately, the majority of Madison participants either felt that the report would have no significant impacts or were ambivalent or uncertain on this issue (see Table 1).Most connected their cynicism or ambivalence to their understanding that the exercise was primarily for research, and several questioned what organizers would do with the research results.During interviews, nearly all of the Madison participants mentioned their awareness that the NCTF was part of a research project or "experiment."For instance, one asked, "What are you guys doing with all these things and the tapes and transcripts and recordings?" and another asked, jokingly, "Is this sort of like a Stanley Milgram experiment?With the electroshocks?"More seriously, another said, "I got a sense that we were more of an experiment than anything else.The moderators are taking more notes on us more than they were hearing our questions" (Madison panelist 4).
A few participants were more positive (albeit cautiously so) about the process overall having some impact: "I was really amazed to know such a scientific process is being conducted to extract the opinion of people at large.It's wonderful, I'm really fortunate to live in a society where people's opinions matter" (Madison panelist 6).However, when asked later whether she thought the report would have an impact, she seemed unsure: "I honestly hope so, I don't know about how much impact it's going to have…it would be a waste if it doesn't influence the policymakers, right?So I'm being optimistic…Because the policymakers are definitely not going to throw away people's opinion just like that." Other comments revealed nuanced understandings of the larger societal and political context of the exercise and the challenges of reaching broader audiences with citizen recommendations.For example: "I would hope more than people just interested in nanotechnology will read this...and think more about some of these issues being important to general citizens.It might affect how they develop nanotechnology.Maybe it won't" (Madison panelist 13).
A number of participants noted that what happened with the recommendations would depend on funding and organizers' follow-up.When asked if he thought the report would have an impact, one participant said, "I think if given to the right people and if people actually care about it...I think that if you guys get lots of support with doing this.Like I think it could, it just depends on what you do with it" (Madison panelist 9).Another noted, more pessimistically, that organizers didn't seem very committed to assuring that the report had any impact on policy: So do you think the report will have an impact?To be honest I don't think so….The producers of it have to decide what they want to do with it.Was the study on how people interact?Or is it how people come up with policy recommendations and I'm not clear that they...I'm not clear that they have decided they want it to be a policy recommendation.If they are going to study the process rather than the finished results.If they wanted to do the process and policy then I think it would be probably made clear in the budget and so forth (Madison panelist 11).
A few, like this participant, were less equivocal: "I know what they were trying to do, they were trying to categorize everyone so we could easily study and see how our thoughts changed.But it just wasn't useful at really truly generating a discussion.It's useful for you scientists study [ing] us, because we're all compartmentalized into these groups.But not useful for what you told us the project was for, which was to create a dialog and have this product" (Madison panelist 5).
Did participants feel empowered, or efficacious, after participating in the exercise (Research Question 3)?Some Madison participants felt more efficacious as a result of their experience (see Table 1), and most mentioned learning something about NBIC, about themselves, other participants, and/or deliberative processes.Not surprisingly, those who expected the exercise and/or report to have some impacts tended to feel more empowered.Still, the majority of comments in response to this question were mixed or negative.For example, one panelist reflected: "In agreeing with my fellow peers about a potential topic that will affect all of us.I feel good about that, about agreeing with the group…but I don't see how our agreement will make a big difference" (Madison panelist 8).
Nearly all of the participants said they would participate in another consensus conference, although, as with efficacy, many answers were qualified (e.g., "Yes, I would do it again, but would like to talk about ways it could be improved," or "I would do it again but I would like to try another model").One noted that she could not replicate a very complex and expensive experiment like the NCTF: I think that it was nice to get together with other people and a smaller group and say, "Oh look we can all talk about something."...I still think that there has to be a way for citizens to be empowered.I don't know what that way is….As far as, am I empowered?No. Do I have ways now that I think I can get into it?Yes.That I found a new way?No, because I can't replicate that.I mean I could...I could get neighbors together and write a letter.But most people don't have the time and the inclination to do that (Madison panelist 10).
In sum, reflections of Madison participants shortly after the event suggest that most did not feel very hopeful that the event would have impacts.While some participants felt increased efficacy from their deliberative experiences (especially those who were more confident it would have impacts), over half did not feel particularly empowered after the exercise and/or had mixed feelings.A few noted that the experience did not build their capacities or power as individual citizens to engage in NBIC governance issues beyond the exercise or to collectively organize with other citizens to do so.Others questioned whether policymakers and the general public would care about their opinions and recommendations if they received them.Most connected their lack of confidence about potential impacts with their perception that the exercise was primarily for research, noting that impacts were contingent on what organizers did with the results.

Discussion
National organizers of the NCTF judged the exercise a success in engaging ordinary citizens and avoiding pathological deliberation.Their final report concluded that "with the appropriate information and access to experts, citizens are capable of generating thoughtful, informed, and deliberative analyses that deserve the attention of decision makers" (Hamlett et al., 2008, p. 1).A previous analysis of the NCTF, based on reviews of final written reports produced by NCTF participants (written by facilitators at one local site) concluded that "NCTF discussions closely approximated deliberative norms of democratic discourse" (Philbrick and Barandiaran, 2009, p. 340).
Our observations and analyses suggest that, indeed, citizens are capable of thoughtful and informed deliberation.Overall, Madison participants felt good about deliberation in the F2F sessions.Most attributed smooth F2F discussions to good facilitation and a cooperative, respectful group of participants.Few participants felt that anyone "dominated" the deliberations in significant ways, and there were no indications that any "domination" that did occur led to polarization cascades in which certain participants unfairly swayed the opinions of the whole group.
At the same time, however, NCTF participants' assessments of the quality of the deliberations during the exercise were much more mixed, nuanced, and/or in some cases explicitly negative than organizers' assessments.Also, our data suggest that controlled deliberative structures and facilitation-along with participants' self-restraint at times-sometimes limited full expression of opinions and comprehensive discussion of issues in the F2F sessions.Participants were explicitly (and often strongly) critical of the quality of the deliberation in the K2K sessions.Many panelists perceived that organizers had an agenda that they did not want to diverge from, regardless of citizens' interests and questions, particularly regarding environmental health and safety issues.Overall, the majority of the Madison participants were not particularly empowered by the exercise or confident that it would have broader impacts-because they perceived it as a research project.
Perhaps most importantly, our findings highlight the important insights gained from using a combination of grounded, qualitative approaches focused on participants' perspectives.In particular, these methodologies reveal the ways participants were aware of conflicts and power dynamics in the event, as well as reflecting on its structure and goals and their relative power within this structure.Their awareness of-and more importantly, their active negotiation of-these dynamics shaped deliberative quality in a variety of ways, and also shaped their sense of empowerment and perceptions about the potential impacts of the event.
Our findings also raise a number of interesting issues that go beyond our focal questions.The remainder of our discussion elaborates on these issues.

Power Inequities, Conflicts and Emotions: Just beneath the Surface
While NCTF deliberations were carefully structured by organizers in an attempt to encourage calm, rational deliberations in which there were no power inequities among participants, there were, of course, inequities between participants, facilitators, moderators and organizers throughout all aspects of the NCTF processes.These kinds of inequities are inherent and to some extent unavoidable in structured, facilitated deliberations.As Harvey notes: "Facilitators and chairs…can not only enforce the rules of engagement (turn taking, length of each turn, actor speech rights, time keeping, and the like)," they also can "direct the substantive content of debate and discussion, determining what counts as relevant speech and opinion and maintaining control over the knowledge that is voiced and which issues are exposed and debated" (Harvey, 2009, p. 151).Again, comments throughout Madison participant interviews and during online interactions suggest that participants were aware of the power inequities and relational dynamics between themselves and facilitators/moderators, and they actively negotiated these dynamics in a variety of interesting ways.
Like most deliberative exercises, both the F2F and the K2K sessions in the NCTF were also structured to avoid uncomfortable conflicts and strong emotions among participants.Interestingly, our data suggest that participants also quietly self-censored at times to avoid conflicts.Interview comments suggest that many of the Madison participants were fairly sophisticated in their past deliberative experiences and understood the "rules" of group process.In other words, they had experience, and therefore "cultural comfort," with structured and facilitated deliberative processes.These factors likely made facilitation easier in some ways and also created the impression of relatively calm, conflict-free deliberation in which power was "equal" and everyone was satisfied.The NCTF final reports on the exercise, paralleling this view, highlight the consensus achieved about the recommendations among citizens.
Our data and participant observations of the exercise, however, reveal a more complex story.Madison participants, and all NCTF participants, not surprisingly, had different levels of knowledge, different deliberative styles, and contrasting opinions.In the Madison F2F sessions, the participants themselves noted, and we observed, that at times conflicts and emotions simmered under the surface, but were never openly on the table.Tensions and potential conflicts were much more explicit throughout K2K sessions, and comments throughout interviews and online session transcripts illustrate a range of emotions-both positive and negative.For example, participant comments reveal a range of positive emotions-e.g., related to learning about issues, connecting with other participants, hearing their perspectives, building a sense of solidarity with them, etc.At the same time, they also reveal annoyance with other participants, frustration about organizers' pre-set agenda, anger about not being heard, and a range of other negatively-valenced emotions.
While touches of conflicts and emotions were evident in both F2F and K2K sessions, for the most part, explicit conflicts and discussions of these emotions among NCTF participants were avoided by facilitation (in the case of the Madison F2F sessions) or the constraints of the medium and format (in the case of the K2K).Frustrations and critiques related to the online sessions were discussed with the Madison group in one of the last F2F sessions, but they were never directly discussed collectively in online sessions with the national organizers, moderators, and participants from other sites.
Careful facilitation to avoid participant conflicts and related emotions, while certainly well-intended and necessary to some extent, creates deliberative paradoxes.While Habermasian models imply that emotions have negative influences on the quality of the deliberations (e.g., anger, frustration, despair, and the like could result in disruption of deliberations or inability to reach consensus, etc.), emotions can also play many positive roles throughout deliberative processes, such as serving as a means of motivating participants to work together (e.g., feelings of solidarity, trust, compassion) and encouraging people to think more deeply about issues they are discussing (Mansbridge et al., 2006)."Negative" emotions such as anger and frustration are also necessary-and inevitable-in honest and productive deliberations.As Ryfe describes, because deliberation towards change of perspective involves "a disturbance of everyday reasoning habits," uncomfortable emotions are unavoidable-and necessary (Ryfe, 2005, p. 59).Through careful facilitation, potentially productive conflicts may be "masked by institutionalized 'comfort' among participants, apparently taking part equally" (Abelson et al., 2003, p. 246).
On a broader level, conflicts can bring critical but usually invisible power inequities to the surface, and actively negotiating these conflicts (as opposed to avoiding them) is inherent in reducing these inequities.As Abelson et al. (2003) argue, the comfort produced by careful facilitation in deliberation "is neither realistic nor worth pursuing as it masks inequalities that exist among participants and between participants and decision makers" (Abelson et al., 2003, pg. 246).Along these lines, others have strongly critiqued the equation of success with the generation of consensus, or the appearance of doing so, noting: In practice a consensus orientation tends to elide conflicts in such a way that the interests of the less powerful are rendered silent, invisible, and unthinkable.The habitual association of deliberation and consensus thus hampers our ability to cultivate and benefit from deliberative opportunities (Kadlec & Friedman, 2007, p. 13).
We argue that genuine consensus in deliberations is attainable, though extremely challenging to achieve, but we agree that it is highly problematic to prioritize consensus over meaningful and at times non-consensual engagement-especially among deliberators coming from very diverse racial, ethnic, and socioeconomic backgrounds and cultures, and with significantly different levels of societal power.
Further, the avoidance of conflict and emotions reflects the privileging of rationality and cognitive processes, and can serve to "foreclose dissent" in deliberative processes, especially those organized by powerful status quo actors and institutions (Elam & Bertilsson, 2002;Flyvbjerg, 2001;Kokotovich, 2008;Mouffe, 1999;Sanders, 1997;Young, 2003).The concept of rationality, of course, is itself value-laden, dependent on context and culture, and inextricable from power relations, since status quo societal actors and institutions get to define what is and is not rational, particularly in mainstream institutional arenas where public deliberation often occurs (Flyvbjerg, 2001).Dissent towards what is defined as rational by status quo actors is typically rendered as "irrational"-and therefore emotional-and hence not appropriate within deliberative processes that place the highest value on cognition (Elam & Bertilsson, 2002;Flyvbjerg, 2001;Kadlec & Friedman, 2007;Kokotovich, 2008;Mouffe, 1999;Nussbaum, 1995;Sanders, 1997;Young, 2003).Yet history amply demonstrates that citizen engagement that might lead to social change by definition involves dissent from the status quo, and strong emotions (e.g., in the movements for civil rights, women's suffrage, and the American independence).If dissent is deemed irrational and not appropriate in deliberative processes, then these processes are not likely to facilitate social change, and will be biased towards styles of deliberation that lean towards supporting the status quo.

Diversity, Deliberation, and Dissent
How might deliberative quality have been affected if the NCTF participant group had been as diverse demographically and culturally as the U.S. population?As discussed above, many deliberative processes are based on gendered (masculine) Western European cultural assumptions that place a high value on structure, control, abstraction/generalization, and that view strong emotions and communication styles not considered cognitive as inappropriate or detrimental.As Sanders (1997) notes, deliberative approaches are "fraught with connotations of rationality, reserve, cautiousness, quietude, community, selflessness, and universalism" which carry "conservative or antidemocratic connotations usually overlooked by well-intentioned theorists" (Sanders, 1997, p. 348).Yet people from many non-Western and/or non-white cultures may not place such high value on rationality and cognitive processes; though of course drawing on cognition and reason, they might also rely on unstructured conversation, sharing personal experiences and emotions, storytelling, rituals, music, dance, art, etc.-modes of communication that are often discounted in highly rational modes of discourse and decision-making (Burkhalter, et al., 2002;Nussbaum, 1995;Powell et al., 2011b;Sanders, 1997).Along these lines, we suspect that these groups might be alienated by highly structured deliberative processes such as those used in the NCTF, compared to more privileged and educated white people (Powell et al., 2011b).
Moreover, minorities, lower-income people, women, and other marginalized groups have substantial reasons to be upset about being disadvantaged and unheard in U.S society-and therefore may be more likely to distrust and question status quo actors and institutions that typically sponsor deliberative exercises.In fact, this is likely one explanation (among many) for the difficulty in engaging less privileged and marginalized people in these exercises in the first place. 15As Kadlec and Freidman note, quoting the arguments of Young (2000) and Sanders (1997), "Deliberation is a luxury to which only political elites have access and, given the structural inequalities, why should disempowered individuals and groups be asked to trust so-called 'rational dialogue' with those in whose interest it is to perpetuate unjust economic and political arrangements?"(Kadlec and Freidman, 2007, p. 3).
In the case of the NCTF, if more lower-income, minority and marginalized people had been included in the exercise, we suspect that anger, tension, and questioning of both process and content of the F2F and K2K discussions would have been more prevalent.In the online sessions, for example, these issues were played out when a Native American participant raised questions about how NBIC might affect the people of her nation and whether they would have any say in the matter.Her anger and despair about the marginalization of Native Americans in our society-and frustration when she didn't feel heard in the sessions-were evident in her comments.While other participants tried to support her, comments in the online sessions indicate that she and they felt it was futile (e.g., "I give up"; "The majority decides for everyone"; "…the minority is screwed").
If people in even this relatively homogeneous NCTF group had been allowed to deliberate for longer periods of time, especially in less structured ways, we expect that more explicit conflicts would have arisen, as they do in deliberations in real-world contexts with actual stakes and stakeholders.Eventually, as participants gained more knowledge, efficacy, and collective capacities, they would have begun to openly question organizers' decisions, their agendas, and many other aspects of the deliberative processes (Powell & Colin, 2009).We would welcome such "rebellion"; engagement tensions reflect healthy attempts to reduce power differentials and a shift to more authentic and democratic deliberation (Powell & Colin, 2008;Kadlec & Friedman, 2007).

Exclusion of Environmental Health Issues Reveals Organizers' Top-Down Power-and Participants' Bottom-Up Resistance
Oddly, environmental health and safety issues were the only areas repeatedly raised by participants that were deemed not relevant to NBIC by moderators, even though heated debates had been ongoing by the time of this exercise among governments, scientists, NGOs, citizens, and others worldwide about the environmental, health, and workplace risks related to nanotechnology developments, including NBIC technologies.Unfortunately, the reasons environmental and health issues were deemed "off topic" by national organizers and moderators were never resolved.We (authors of this paper) contacted national organizers by email and asked for their rationale in excluding environmental health and safety issues from NCTF deliberations, but did not receive a response.We speculate, in part, that organizers' reluctance to discuss these issues could have been related to the framing of the exercise as "upstream" engagement (e.g., analysis and consideration before rather than after technologies come to market).Perhaps organizers perceived the discussion of environmental health and safety issues related to current NBIC developments as downstream engagement, and thereby outside of their preferred frame for deliberation.Also, trying to exclude these issues from discussion was likely in part an attempt to make the online deliberations easier to manage.
While we sympathize with the difficult task for online moderators to keep discussion focused-indeed, at times such facilitation is absolutely critical to a productive and enjoyable deliberation on a complex topic-we are troubled by what seemed to be an arbitrary but significant determination of the scope of permitted questions and discussion.For one, and in line with our methodological focus on participants' perspectives, participants experienced the narrowing as forceful rather than facilitative.Secondly, this narrowing is clearly not in line with key criteria of good deliberation, in which all participants have equal power to set the agenda and to have their voices heard; in this case, it appears that organizers and moderators had deemed environmental and health issues as outside of the scope of deliberations before the event began and were unwilling to change that even when participants asked repeatedly.In several critical ways, it seems that the NCTF organizers and online moderators fell short of "master[ing] the give-andtake of deliberation" (Hamlett, 2007, p. 2).This is ironic, given that the NCTF background materials described the goal of testing participants' ability to "master the give-and-take of deliberation" as a key rationale for the NCTF.Also, the exclusion of health and environmental aspects of NBIC technologies carries distinctly political weight, as these types of questions are central to considering unintended negative consequences of NBIC technologies.
Further, these issues reflect significant power differentials inherent in the structure of the exercise and ways that they shape deliberations and exercise outcomes.Clearly, online moderators and organizers had much more power to frame the agenda for deliberations, background information, and deliberative content than site facilitators and participants did.The framing of critical and politically important issues as outside of the appropriate range of deliberation is an example of the ways engagement exercise organizers can use their power over the framing of the exercise to shape the content of the deliberations and the range of potential outcomes for the exercise (Toker, 2005).

Citizen Participants are not Ordinary, Naive or Passive
Ironically, organizers' attempts to exclude discussions about environmental health and safety issues backfired because participants were aware of these attempts, and some were frustrated and even angered by them.All participants, regardless of whether or not they had scientific backgrounds, brought important experiences, opinions, and knowledge into the process-which they gleaned from their work, media, books, magazines, friends and family, and a variety of other sources.Also, many NCTF participants knew quite a bit about the content and contexts of NBIC developments-indeed, our interviews and interactions with Madison participants, and comments and questions during online sessions, suggest that several panelists knew more than NCTF organizers, moderators, and/or experts.Apparently, this was something the organizers had not anticipated-perhaps in part because they had conceptualized their ideal "ordinary" participants as lacking science and technology backgrounds or knowledge about NBIC.Instead, participants' background and knowledge coming into the exercise gave them some power to question and resist the organizers' futuristic and relatively narrow framing; and regardless of moderators' attempts to exclude these issues, recommendations related to environmental health and safety issues were included in all of the final citizen reports.
These findings also raise questions about attempts to avoid "polarization cascades" that framed the design of this exercise, and in particular, the assumptions that underpin it-e.g., that expert facilitation will guarantee healthy deliberative processes, and the parallel (but implicit) assumption that citizen participants in deliberative processes are passive and/or naive.Not only did NCTF participants bring knowledge relevant to NBIC into the deliberations, some Madison participants also had experience in group work and/or expertise facilitating deliberative processes.All Madison participants were thoughtful about the quality of these processes and their roles in them.They reflected on their own deliberative styles and how they influenced deliberations, as well as observing the styles and perspectives of other participants.Contrary to the expectation that some participants would try to dominate discussions, several participants explicitly mentioned their attempts not to dominate others, while very few mentioned concern about others dominating.
Madison participants were also highly cognizant of the hierarchical structure and relational power dynamics in which they were engaged-both internal and external to the exercise.Comments throughout interviews indicate that they were aware of the different roles of local site facilitators and national organizers and their varying degrees of power to set the exercise's agenda, frame the content of the discussions, and shape the outcomes.They were reflective about how the decisions, actions, and goals of national organizers and online moderators, as well as site facilitators, shaped the quality of deliberations in the exercise as well as its potential impacts.They actively negotiated these dynamics in a number of ways-self-censoring, withdrawing when they felt they had no voice or they felt it would be pointless, and/or resisting organizers' attempts to silence their questions.
Moreover, the substantive differences in relational dynamics involved in face-to-face versus online text-based formats likely played roles in participants' deliberative behaviors and perceptions about deliberations and the exercise overall.For example, we suspect that in the K2K sessions, some people may have been more comfortable violating deliberative rules-explicitly questioning power and/or authority, expressing strong opinions and emotions to other participants and moderators (including national organizers)-since they had never met them, weren't facing them in person, and knew they would never interact with them in the future.Such comfort might have encouraged "frank speech," leading to more open, honest deliberation (via text).On the other hand, online deliberators could also have easily "invisibly" and quietly disengaged from deliberative processes when they felt uncomfortable with tensions or became too frustrated with not being heard, or thought their questions were being unfairly shut down by other participants and/or moderators.Our evidence suggests that both happened-some online participants at times engaged in what appeared to be "frank speech," while at other times participants "invisibly" retreated from online discussions (or dropped out altogether), especially in the later K2K sessions (see Delborne et al., 2011).Further research might explore the prevalence and impacts of these kinds of interactions in face-to-face versus online deliberative formats.

Participants' Perceptions about Exercise Outcomes Affect Deliberative Quality
Madison interviews and survey results show that experiences in the exercise tended to increase participants' sense of personal efficacy, while at the same time decreasing their confidence that societal decision-makers such as government and policymakers would be responsive to their recommendations (external efficacy).Paralleling the drop in external efficacy in the survey results, the majority of Madison panelists were not particularly hopeful that their recommendations would have any societal impacts.Several noted that they didn't think government and policymakers were likely to be very influenced by their recommendations, for a variety of reasons.In other words, according to participants' assessments, the exercise clearly did not rate highly on "influence" criteria listed by Rowe and Frewer (2000).These results parallel evaluations of GM Nation!, in which the majority of participants agreed (when surveyed after the event) that event sponsors and organizers would not act on their recommendations and that feedback from the events would not influence the future of genetically modified (GM) crops in the UK (Rowe et al., 2008, p. 429).
These findings, and Madison interview comments in particular, reflect participants' awareness of the broader institutional and political contexts in which these exercises take place-and their awareness that NCTF leaders and organizers had the most power, resources, and capacities to facilitate any potential political or societal impacts following the exercise.Several Madison NCTF participants, for example, said explicitly in interviews and/or during F2F feedback sessions that potential impacts of the recommendations were contingent on organizers' level of commitment to doing something with the final reports.In an attempt to respond to these comments, during the first and last Madison F2F sessions, we encouraged participants to brainstorm potential audiences for the report-and they suggested local and state government agencies, local companies, university nanotechnology centers, trade journals, regulatory agencies, congressional committees, NGOs, CEOs of large corporations doing nanotechnology research, NSF, conferences, citizen watchdog groups, and many more.Panelists also suggested several times that their recommendations be shared with the broader public via news media and the Internet.However, in follow-up interviews, most didn't feel very confident that much would be done with their recommendations, because they perceived the organizers to be primarily interested in research.Unfortunately, confirming participants' expectations, other than presenting some points from citizen reports in a briefing to the U.S. Congressional Nanotechnology Caucus several months after the event, there is limited evidence that national NCTF organizers and/or local site facilitators made significant attempts to actively share participants' recommendations with policymakers, media, NGOs, or other potential audiences after the exercise.
Interestingly, participants' perceptions that the exercise would have little policy or other kinds of impacts affected their deliberative behaviors in a variety of ways-at times, encouraging "active passivity" (our own term)-passivity that is consciously chosen rather than unconscious or naïve.For instance, some Madison participants said (or implied) that they held back opinions because they felt there was no point in raising potentially contentious arguments that could take significant amounts of time to comprehensively discuss and debate when it was a short-term research exercise with people they would likely never see again.Perceiving themselves to be primarily research subjects, participants lacked motivation to argue for their perspectives or risk experiencing tension and conflict with other participants and/or with facilitators and moderators.
It is logical that participants would be less likely to energetically or passionately engage in short-term constructed exercises that they perceive as research projects and in which they believe themselves to have limited power or responsibility.Deliberation is both cognitively and emotionally difficult, and "people will be more likely to engage in a probing, difficult deliberation if they are motivated by accountability, high stakes, and the diversity of the deliberators" (Ryfe, 2005, p. 57).As we learned in this exercise, and know from our long-term engagement with citizens (Powell et al., 2011b;Powell & Colin, 2008, 2009) engaged people are cognizant and reflective about the goals of the people and institutions that engage them, and understandably skeptical that completely research-oriented projects will have any societal outcomes.As Kyle and Dodds note, "Public engagement that seeks to meet the demands of democracy needs to occur under conditions of respectful deliberation based on well-placed public trust that public engagement processes will affect policy outcomes, rather than serving as a perfunctory consultation process that at best allows for the venting of opinion" (Kyle & Dodds, 2008, 17, p. 318).

Conclusion
In this study, we privileged the experiences of citizen participants in the National Citizens' Technology Forum (NCTF), a large engagement exercise on emerging technologies-nanotechnology, biotechnology, information technology, and cognitive science (NBIC).Using a mixture of qualitative methods, we explored participants' perspectives on deliberative quality, their sense of empowerment, and their expectation that the exercise would have policy impacts.While our data-primarily focused on Madison participants-give us confidence that the NCTF was experienced as largely positive, the foregrounding of participants' reflections challenges some assumptions held by organizers of large-scale deliberative exercises designed to affect the governance of emerging technologies.
First and foremost, in contrast to idealistic visions of equal power among deliberators, there were clearly knowledge and status disparities among NCTF participants, which they were cognizant of and actively negotiated throughout the exercise.Such dynamics should not surprise or discourage organizers of deliberations.Conflicts-key ingredients of meaningful deliberation when participants bring diverse perspectives into conversation and are seeking consensus-often stem from cultural differences as well as hierarchies of expertise and societal power, and can involve intense emotions.These hierarchies may ebb and flow, and citizen participants-who bring their own prior knowledge and experiences of civic discussion to a deliberation-can weather such challenges.In fact, suppression of conflict by facilitators, however wellintentioned to foster the meaningful and respectful exchange of ideas, may tamp down the diversity of viewpoints allowed within the deliberation and implicitly discourage dissent.This limitation may paradoxically support status quo arrangements of power and resources within a process often imagined to challenge the status quo by bringing "citizen voices" into policy debates.
Second, we suggest that to adequately understand the power dynamics and conflicts inherent throughout structured deliberative exercises, evaluators should widen their analytical frameworks to also consider the goals of scholars, experts, and other professionals who organize, facilitate, and evaluate these exercises.In leaving these powerful actors out of deliberative analyses, critical power dynamics that shape deliberations, and in addition, relevant outcomes external to the deliberations (or lack thereof), are rendered invisible.In the NCTF, for example, the exclusion of concerns about environmental and health safety impacts of nanotechnology must be understood not simply as a "tough call" in facilitating complex dialogue, but as an imposition of a set of assumptions and goals held by national organizers in the face of competing interests of participants.Such critical deliberative dynamics are very difficult to bring to light in evaluations based only on top-down quantitative survey methodologies.
Third, our findings point to questions about the roles and impacts that constructed deliberative exercises can play within broader political and societal contexts.In the NCTF, participants knew they were part of what was primarily a research project (as opposed to real-world political discussions), and based on their comments, this awareness not only led to more passive deliberative styles, but also diminished their sense of empowerment and their expectations that the exercise would have meaningful impacts.This sense of "subjectivity" raises insights and questions about the complex ways that citizen participants experience short-term deliberative research exercises.To what extent do they encourage authentic deliberation and empower people for participation in the real worldwhen participants "are acting as research subjects rather than as citizens…paid to attend…not as an act of civic will and commitment" (Friedman, 2006, p. 9)?
Given the NCTF participants' low expectations of their deliberations having a meaningful policy impact, how do policymakers and other important decision-makers interpret and respond to consensus reports and other outcomes of organized deliberation?A stated goal of the NCTF was to "provide decision makers-in the government, in business, and in society generally-with the informed, deliberative opinions of ordinary people who have taken the time and effort to study the issues carefully" (Hamlett, 2007, p. 2).Yet as Friedman (2006) notes: Deliberative research talks about public opinion under ideal circumstances, but it does little, by itself, to create those circumstances in the wider polity.If leaders see a scientific sample of the public changing in a kind of laboratory setting, in isolation from the political life of the community, there is no particular incentive for them to change how they interact with the public....Having a knowledge of what people are likely to think if they have good opportunities to deliberate will do little if the "real" public is in the same old place, expecting the same old thing from leaders" (Friedman, 2006, p. 9, italics in original).
Here we call not simply for further research into the "actual" policy impacts of citizen deliberations, but for a brainstorming of ways to organize dialogues on emerging technologies that inspire a cascade of meaningful dialogues within the larger polity.As a long-term goal, deliberative organizers should find ways to "embed" engagement mechanisms throughout community and decision-making institutions (Cunningham & McKinney, 2010;Fagotto & Fung, 2009).In addition to increased political commitment and funding, this change will require significant paradigm shifts in deliberative approaches and goals-from attempting to create "ideal speech" through highly structured dialogues in research settings, to organizing more dynamic and complex (and at times conflictual and emotional) discussions among diverse laypeople, experts, and policymakers within real-world science and technology decisionmaking contexts.