Measuring Deliberation's Content: A Coding Scheme

This paper details a content analysis scheme to measure the quality of political deliberation in face-to-face and online groups. Much of deliberation research studies the outcomes of deliberation, but there has been a lack of analysis of what groups actually do when tasked with deliberating. The coding scheme was developed out of the theoretical literature on deliberation and further enhanced by the empirical literature on small groups, deliberation, online political talk, and conversation analysis. Strict standards for creating coding schemes were followed to ensure a valid and reliable coding process. Results of the coding of deliberations on the topic of public schools suggest that participants produced a fairly high level of reasoned opinion expression, but not necessarily on the topic which they were asked to deliberate. It is hoped that the code scheme can be utilized by practitioners and researchers of political and social deliberations.

The literature on deliberation has grown at a rapid pace in the past fifteen years.A number of creative and influential experiments have found that deliberation affects political knowledge (Fishkin, 1991(Fishkin, , 1995)), consideredness of opinion (Luskin et al., 2004), and attitude and opinion change (Cappella at al., 2002;Gastil & Dillard, 1999;Price et al., 2002), to name a few effects.A comprehensive review of the deliberation literature by Ryfe (2005) indicates, however, a large gap in the research concerning what actually transpires when people deliberate.Few studies measure the content and process of intra-group interaction.
The purpose of this paper is to offer a systematic way to measure what transpires during deliberations.It is hoped that academics and practitioners of deliberation will find this coding procedure useful, because there is a need to understand how groups deliberate, what they say to each other, what the process of deliberation looks like, and how the content and process affect outcomes of deliberation, such as opinion change or satisfaction with the deliberation experience.
The coding scheme described in this paper was developed with the guidance of existing empirical, normative, and theoretical studies of deliberation.The data used to test the coding scheme came from the Virtual Agora Project, in which Pittsburgh, PA residents deliberated school policy (Muhlberger, 2005).The coding procedure followed rigorous standards for content analysis as described by Neuendorf (2002) and Krippendorff (2003), including establishing a code book, training coders, practicing coding, and establishing intercoder agreement.The results of this project suggest that the coding scheme is a valid and reliable measure for assessing the qualities of deliberativeness in group, political deliberation.
After defining deliberation and discussing the relevant literature, a description is provided of six elements necessary for measuring the quality of deliberation.A description of the deliberation data used to test the coding is offered, followed by details of the coding process and reliability measures.Then, a brief discussion of the results of the coding is offered as well as implications of this coding scheme on assessing the content and process of deliberation.The paper concludes with a note on implementation of this coding scheme to assess other deliberative projects.

Defining Deliberation
In order to assess deliberative experiments to identify their strengths and weaknesses, the first order of business is to be clear about what is meant by the term deliberation.That term has been applied to many different types of conversation; similarly, other words have been used synonymously with deliberation.In the existing literature there is no consensus about the concept being investigated.Such inconsistency is problematic, because then it is difficult to come to a clear conclusion on the process and effects of deliberation.If some research projects analyze casual political conversations while others analyze experiments in which groups are tasked with coming to consensus, the effects of those projects are likely to be different.Moreover, the characteristics to be analyzed in dialogue might be quite different from those of deliberation.The elements to be attended to or counted in dialogic projects are likely to be different from those of deliberative projects.Hence, becoming conceptually clearer on the phenomenon of study is an essential first step to identifying the utility of deliberation in a democratic society.
Although deliberation and conversation are sometimes used synonymously, for conceptual clarity they should not be considered the same phenomenon.Schudson (1997), for example, makes a distinction between political deliberation and sociable interaction, arguing that deliberation is essential to democracy; although, it is rarely enacted in practice.Deliberation for him requires that people discuss a common problem and reach consensus on how to solve that problem.It is difficult discussion that requires people to disagree with each other and to assess the clarity and strength of competing arguments.
Sociable interaction, by contrast, is conversation between people.The goal of these casual conversations, even when the topic is political, is to build social relations or simply to get along.Conversation, by this definition, is not the same as deliberation, in part because political conversations are not a means to an end of solving a social or political problem.
Another term that often is used synonymously with deliberation is dialogue.One critical distinction between these terms is that dialogue underscores understanding as central outcome to the process.Shields and Edwards (2005) write that dialogue is "a way for us to understand something or someone who is in some way different from ourselves, who has a different perspective, alternative lens, varied history, and so forth" (p.15).Another way that dialogue is conceptualized emphasizes ethical and value judgments (Meyer, 2004).Thus, the development of a shared ethic or a set of moral standards also should be an outcome of dialogue.The early Socratic dialogues written by Plato serve as an example.Such dialogues emphasized virtue and attention to what is right and just (Zappen, 2004).
If we start with the definition offered by Schudson (1997), then dialogue and deliberation are distinct in that dialogue emphasizes mutual understanding as a means to an end of establishing a shared ethic or moral foundation for a just society, whereas deliberation is focused on addressing a shared problem and arriving at solutions to that problem.Schudson's (1997) definition is not the only understanding of deliberation in the many studies that claim to analyze this phenomenon.Perhaps the most widely referenced conceptualization is that offered by Habermas (1984).He defined deliberation as requiring a group of individuals to exchange rationalcritical arguments on a problem and work towards a solution that can be acceptable to all who have a stake in the issue.Rational-critical arguments are grounded in truth or at least a shared understanding of objective reality, that are open for judgment and for critique, and that can be further argued or defended.In keeping with that notion, Gastil (2000) defines deliberation as "discussion that involves judicious argument, critical listening, and earnest decision making" (p.22).These two definitions resonate most with the kinds of experiments academic researchers have been conducting of late, and are relevant to many of the projects and programs by practitioners of deliberation in the United States. 1  For the purpose of conceptual clarity, however, and to differentiate deliberation from dialogue or casual political conversation, deliberation is defined here as a process whereby groups of people, often ordinary citizens, engage in reasoned opinion expression on a social or political issue in an attempt to identify solutions to a common problem and to evaluate those solutions.This definition aligns most closely with that of Schudson (1997), Habermas (1984), as well as Gastil (2000).

Elements of Deliberation
Now that a definition of deliberation has been established, it is essential to figure out what elements are required in order to operationalize deliberation.Relevant literature from several areas of empirical study, including deliberation, political talk, small group, and online interaction were analyzed prior to this study for guidance on what elements are important for promoting deliberation.The theoretical and normative literature on political talk also was consulted.The aim was to identify shared or agreed-upon characteristics that comprise political deliberation.
There are a handful of studies that operationalize deliberation for content analysis.These studies have primarily coded discussions that occur through the Internet.To date, only a few studies that systematically code face-to-face group political deliberation have been published (see, for example, the dissertation of 1 Some theorists offer different ways of thinking about the phenomenon being labeled "deliberation."Young (2003) defines it as "both a normative account of the bases of democratic legitimacy, and a prescription for how citizens ought to be politically engaged" (p.103).In this definition, deliberation is not a process, but rather a theory that justifies decision-making in a democratically organized government.For Gutmann andThompson (1998, 2004), deliberation is the procedural way to engage moral disagreement in a society, not necessarily to come to consensus, but at least to enable the disagreeing parties to better understand each other, understand their own positions, and to continue to work on the areas upon which they agree.For Page (1996), deliberation happens primarily between elites and is channeled through the mass media, because face-to-face exchange among ordinary citizens is impractical.These three definitions offer other ways of thinking of deliberation.Dutwin, 2002).Both means for deliberation are worth attending to, since although deliberative projects have historically been face-to-face, the expansion of access to the Internet has led to an increase in the number of projects that use it to bring together geographically dispersed people.Thus, regardless of whether the content analysis projects were conducted on face-to-face or online groups, six elements appear to be essential for political deliberation: reasoned opinion expression, references to external sources when articulating opinions, expressions of disagreement and hence exposure to diverse perspectives, equal levels of participation during the deliberation, coherence with regard to the structure and topic of deliberation, and engagement among participants with each other.Each element is discussed next.

Reasoned Opinion Expression
Rational argument is one of the cornerstones of deliberative and political theory.Many of the studies that analyze political deliberation reference the Habermasian public sphere (1962/1989), which requires arguments supported by reasons in order for public opinion to form and to serve as a check on government.Rational argument requires individuals to advance positions that can be defended against critique.In The Theory of Communicative Action, Habermas (1984) argues that an expression is rational if the claim provides evidence that can be observably confirmed or empirically denied or appeals to a shared normative ground.Thus, in a deliberative context, a reasoned argument would be one in which assertions are grounded in empirically verifiable evidence or in shared understanding of moral or normative behavior.
In studies of deliberation, reasoned argument generally is defined as opinion claims supported by evidence for those claims.Dahlberg (2001), for example, in his model of ideal deliberation requires "exchange and critique of reasoned moral-practical validity claims" (p.623).In other words, "deliberation involve[s] engaging in reciprocal critique of normative positions that are provided with reasons rather than simply asserted" (p.623).
Similarly, Graham and Witschge (2003) advocate for a coding scheme that includes rationality.They coded a set of political conversations on UK Online, a publicly available, asynchronous, threaded, group discussion.Each message was coded for the type of message in the sequence (an initial message, a response to a prior message, or an irrelevant message not part of the discussion topic).Then, the message was coded as either "rational" or "irrational."A rational message was one that provided justification or a reason for a "validity claim" (p.182), which is a claim offering reasons or evidence.

Sourcing
Sourcing is related to reasoned opinion expression, but it is often treated as being a category in its own right.Existing coding projects generally include an external source code (Davis, 1999;Graham & Witschge, 2003;Hill & Hughes, 1998).Attending to the source used to support a claim is another way to measure whether deliberation is rational.Davis, for example, conducted a content analysis of political threads on Usenet, which included a code for "evidence" that included "inclusion of other materials" defined as "references to, quotations from, or discussion of materials such as specific books, articles, government reports, speeches, etc." (p .162).The importance of coding for a source, beyond just assessing whether a claim offers evidence or reasons, is that the types of sources people might use to support their claim matter.For example, Dutwin (2002) finds in his coding project that citizens reason and offer evidence based primarily on personal narratives rather than on external sources, such as books, newspapers, or speeches.
For deliberative projects, determining what sources participants are using to support their claims might be an important piece of information.For example, many deliberation projects provide documents for participants to read in order to help them become familiar with the problem they are to discuss.It is important to track whether participants are referencing such materials to support their claims, or if they default back to personal narratives and experiences.The benefit of background documents is that they are a shared resource for discussants.When there is disagreement, claims based on background documents can be more readily understood and contested than claims based on personal experiences, which are harder to verify and to contest.

Disagreement
Disagreement is an important marker of deliberation.Disagreement is a sign that there is a problem in need of a solution, a conflict in need of consideration and resolution.It also is a sign that there are participants in the dialogue with distinct views on a particular issue.This difference suggests heterogeneity of perspectives.
The question of whether there is disagreement in a deliberation matters for three reasons.The first is that disagreement is an indication that there are diverse viewpoints in the group.The participants are not homogenous in their viewpoints-a concern that has become important to research on online political discussion (Doheny-Farina, 1996;Sunstein, 2001).The second is that there is a concern that people who share similar perspectives are more likely to polarize in their beliefs; that is, they are more likely to develop more extreme attitudes as a result of their interactions with like-minded others (Sunstein, 2003).When there are participants in the dialogue with alternative perspectives, this can mitigate the polarization effect (Sunstein, 2003).Third, people who differ on a position are more likely to have their own views further examined and strengthened in a more rational way when they are exposed to disagreement and the articulated perspectives disagreement invites (Cappella et al., 2002).Thus, measuring whether disagreement is occurring in political discussion can serve as an indicator of the level of heterogeneity of the participants and can provide some indication of whether participants are experiencing the benefits found when disagreement is expressed (Cappella et al., 2002).

Equality
Along with reasoned opinion expression, equality is theorized as an essential element of deliberation.Each participant must be able to participate on equal footing with every other participant (Dahlberg, 2001;Graham & Witschge, 2003;Habermas, 1984).No participant should dominate the conversation or silence others.Contributions by participants to the discussion entail the possibility that additional information will be contributed, which can allow the group to examine the topic more completely.Put another way, equal speaking opportunities increase the likelihood that a diversity of perspectives can be heard (Burkhalter et al., 2002).

Topic
Topic is also an important element of deliberation.If the discussion is off topic, then the deliberation cannot meet its objective of deep consideration of an issue.Thus, attending to the topic of the deliberation matters for deliberation.Stromer-Galley and Martinson (2005) identify two conceptualizations of topic in discussion.The first conceptualization is the structuring topic, which is the topic established prior to or outside of the immediate interaction.In a deliberation, participants may have been brought together to discuss a current spike in school violence.The problem of school violence would be the structuring topic.The second conceptualization is the interactional topic.This is the subject of discourse established through the interaction.In the case of a deliberation on school violence, the conversation itself might focus on several subjects as it unfolds, including availability of guns in the district, absent fathers in children's lives, and the causes and consequences of single parenthood.Each of these subjects of discussion would be an interactional topic.
Both structuring topic and interactional topic likely matter to the quality of deliberation.Participants who are brought together to discuss one topic, the structuring topic, but spend most of their time discussing unrelated topics, are not addressing the fundamental problem they were brought together to address.Hence, it is difficult for such a group to be able to identify the moral disagreements that underlie the problem or identify solutions to the problem.It is also unlikely that anticipated outcomes such as opinion change will take place, because it is unlikely opinion will change if it is not discussed.With regard to interactional topics, participants that drift quickly from one topic to another in the deliberation may not be fully considering the problem or identifying solutions to the problem.Hence, groups that exhibit many interactional topics are likely not as focused on the problem as groups that stay focused on a small set of such topics (Stromer-Galley & Martinson, 2005).

Engagement
Another element of deliberation is engagement-both with the topic under discussion and between participants.Similar to concerns with equality, it is important to note whether participants are actually engaging each other, or if they are simply engaging in monologues in the presence of an audience, which in turn fails to respond.Graham and Witschge (2003) in their coding refer to this as "reciprocity," where people take turns speaking and respond to the claims of others.
Ideally, then, for a discussion to be considered deliberation, it should be characterized by these six elements.Thus, the research questions that guide this research are twofold.First, can a coding scheme be developed to measure these elements, and is it reliable?Second, what descriptive results come out of such a coding scheme?

Data Gathering Method
To answer these questions, this project conducted a secondary analysis of discussions in an experiment conducted at Carnegie Mellon University in Pittsburgh, Pennsylvania, called The Virtual Agora Project, which brought together 568 Pittsburgh residents, who approximately matched the demographics of the city (Muhlberger & Weber, 2006).
The experiment was conducted over three weeks in July 2004.Up to 60 participants arrived at a given time and were broken into smaller groups with a range of 5 to 12 participants (Mean = 8, SD = 2).The participants were given the task of discussing the problem of the city's public schools.Specifically, Pittsburgh has seen a marked decline in its population.The result is that many schools are under-capacity.In the 1980s, the city's school board closed schools and moved affected students into other buildings, but such changes were met with protests.A plan in 2004 proposed closing several schools by August of that year.
Participants were given the task of discussing the closing of schools in general and were given four solutions to the problem of underutilized schools in the city.Participants were asked to discuss the viability, likely success, and problems with the four solutions.The first solution was the closing of middle schools and creating schools that house kindergarten through 8 th grade.The second solution was to merge high school students from smaller school buildings into a larger school building but maintain the identity and grouping of the smaller high school community as a "learning community."The third solution was to allow parents to choose where to send their children to school.Rather than being forced to send children to assigned schools, parents could "vote with their feet," and the mostly empty schools then would be closed.The fourth solution was to maintain the status quo-to keep all the schools open, because it allows for a small classroom/small population learning environment.
Participants were randomly selected into one of three conditions: face-toface, online, or individual contemplation.The third condition was a nodeliberation condition whereby after participants read the briefing materials they were asked to reflect on them.The two deliberation conditions were characterized by the moderators as a "discussion" on the problem of underutilized schools.All discussions were moderated by one of four trained facilitators who were graduate students at the university.Facilitators were instructed to establish ground rules at the start of the discussion.Ground rules included staying on the topic of discussing the four solutions, and discussing the topic among themselves and not with the moderator.Participants were told that good discussion can occur in any number of ways, including following a process of brainstorming, analysis, and synthesis.Participants were not instructed to find common ground, but simply to discuss the four solutions.They were told that each would privately vote on the solution they preferred at the end of the discussion.Moderators did not start the discussion with a question, but rather instructed the participants to begin the discussion.With the exception of one moderator, participants were not asked to engage in any ice-breaker or introduction exercises.
The procedure for the online discussion participants went as follows.Participants arrived at Carnegie Mellon in the morning.After completing a consent form and receiving a brief training session on the software,2 each participant was situated in a dormitory room equipped with a computer, headphones, and microphone.The computer was internet-enabled.Participants filled out a short survey, and were then asked to spend 40 minutes in a "library session" where they were asked to review documents on the four solutions and other materials related to the problem of underutilized schools.After the library session, participants discussed for 90 minutes.After a lunch break, participants were given another 40 minutes to read through the documents, and then discussed for another 90 minutes.Participants concluded their participation by filling out a questionnaire.Unfortunately, due to equipment malfunction, the face-to-face discussions were not recorded.Thus, the focus of this research is only on the online discussions.Twenty-three groups participated in the online condition.

The Coding Scheme
The discussions were systematically content analyzed following guidelines by Neuendorff (2002).A content analysis codebook was developed to categorize several elements of the discussion, specifically the elements discussed in the previous section.Before coding the categories, units for coding were identified at the level of the thought, (see, for example, Meyers et al., 2000).Because participants could talk up to three minutes per turn in the online condition, there was a high likelihood that they would introduce multiple thoughts in their turn. 3 thought is defined as an utterance (from a single sentence to multiple sentences) that expresses an idea on a topic.A change in topic signaled a change in thought.A second indicator of a change in thought was a change in the type of talk.The distinct types of talk that this coding captured were the following: talk about the problem of public schools, talk about the process of the talk, talk about the process of the deliberation, and social talk.Talk about the problem is talk that focuses on the issue under consideration in these deliberations: school consolidation.Opinions, agreements, disagreements, facts, and questions about that problem were all considered talk on the problem.Metatalk is talk about the talk.It is talk that assesses what has transpired or is transpiring in the interaction, either as a group, or between individuals or to clarify meaning-one's own or someone else's.Process talk is talk about the technical and deliberation process (both immediate to the discussion, as well as the context for discussion).Process talk may include questions, agreement, disagreement, metatalk, or facts, but in this category if the topic is about process, then it is coded as process talk.Social talk is talk that brings the strangers together by building social bonds, including salutations, praise, and apologies.
To provide an example of the coding, consider this turn: "Hello, I'm Sarah.I think that the K-8 is a good solution.I also like the small learning communities.I'm having some trouble hearing others, so maybe the moderator could help me?Thanks."In this coding, this turn would consist of four thoughts: (a) Hello, I'm Sarah.(b) I think that the k-8 is a good solution.I also like the small learning communities.(c) I'm having some trouble hearing others, so maybe the moderator could help me?(d) Thanks.The first thought (a) "Hello, I'm Sarah" is coded as social talk, as it is a greeting.Thought (b) is problem talk.Here, she is expressing opinions on two of the choices.Thought (c) is process talk, specifically a statement of a technical problem.Thought (d) is social talk in the form of praise.[See Appendix A for description of codebook rules].

Reasoned Opinion Expression
Reasoned opinion expression is a combined measure that includes the codes for statements of opinion, agreement, or disagreement and whether there is elaboration on that opinion expression.Opinion expression was defined as an expression of the speaker's belief about how the world is.Opinions are expressed judgments the speaker has made on a person, an event, a social problem, a state of affairs, a crisis, values, and the like.Expressions of agreement and disagreement are combined into the measure of opinion expression.Agreement was defined as a signal of support with an opinion expressed by a prior speaker.Disagreement was defined as a statement that signals opposition with an opinion a prior speaker stated.Disagreement is often done in more subtle ways than agreement (Kuo, 1994;Pomerantz, 1984).Disagreement was coded if a speaker said "I sort of disagree," preceding the opinion claim with "well," or statements that start with suggestions of agreement and then "but," which signals disagreement.
Elaboration can be in the form of further justification (as simple as: I'm for k-8, because I think it solves the problems we face), a definition, a reason for holding the opinion, an example, a story, a statistic, or fact, a hypothetical example, a solution to the problem, further explanation for why the problem is a problem, a definition, an analogy, a consequence to the problem or solution, a sign that something exists or does not exist, or any further attempt to say what they mean or why they have taken the position that they have.
The elaboration measure did not categorize the types of reasons offered, the quality of the reasons, nor the accuracy or factual nature of the reasons.This project also did not count the number of reasons a participant might provide for an expressed opinion.Identifying the number of reasons in open-ended conversation is exceedingly difficult, and measuring the factuality or quality of reasoning was deemed unnecessary for this project to get a sense of whether participants are backing up their opinions with justification or further explanation.

Sourcing
Sourcing was operationalized as any references to mass media (including newspapers, television, and books), the documents participants were given for the deliberations, referencing other participants and using ideas from them in their own reasoning, or offering personal narratives as part of their elaboration.There are likely many other sources from which people draw to support their opinion claims.For this particular project, the four sources were identified for a few different reasons.It was especially important for this project to see if participants were using and referencing the briefing documents to support their positions.If participants were using those materials, then that would suggest a fairly informed conversation was occurring that used a shared resource to support claims.Referencing the documents means that participants who disagreed with a claim could challenge the interpretation, since all participants had equal access to that resource.This project also focused on references to the mass media, in part, to continue the work by Gamson (1992).The mass media is often a source for information, and it seemed important to track this common source.Given Dutwin's (2002) research on deliberations, it also seemed important to attend to personal narratives and anecdotes that were used to support positions.In part, the project sought to confirm Dutwin's observations and to see what proportion of the discussions referenced personal experiences.Finally, this project tracked whether participants were using evidence or claims made from other participants to support their own positions.This serves two functions in the deliberations.First, other participants sometimes use what others say to support their own positions, and this project wanted to track how common this was.This element might also be considered a measure of engagement, in that by using as support what prior speakers have said, it signals that that participants are listening to each other, which indicates reciprocity.

Equality
Measuring equality is a difficult matter.Should equality be measured as a function of how much time each person takes to ensure that people are speaking for approximately the same duration?Should it be measured more leniently by only assessing how many people spoke and how frequently each of them spoke, so that rather than how much time each person took to speak, only whether or not they spoke is assessed?Should there be some measure of the substance or force of their contribution?Some participants might speak frequently but contribute little to the conversation, whereas others might speak infrequently but offer critical insights that really help further the discussion.Or, perhaps, instead of counting frequency or amount of time speaking, assessing whether participants had equal opportunity to speak is enough.
For this project, equality was measured by counting the frequency of participation, and by volume-measured by number of words-also was noted to see if there is domination over others in the discussion.Based on how the deliberations were structured for the project, it is reasonable to assume that participants had equal opportunity to speak.Specifically, participants could get "in line" to speak at any point, could not be interrupted by other speakers, but could "jump in" if they had something urgent and relevant to say once another speaker had finished his or her turn.Given that, it seemed necessary to look further to see whether people actually took those opportunities to speak.Thus, for this project, how much people spoke was assessed to determine whether people dominated the discussion or if participation was roughly equal.

Engagement
Engagement was measured in several ways.First, genuine engagement in a discussion requires that participants talk to each other.It is conceivable, especially in online environments, that participants in the discussion talk past each other rather than genuinely engage each other.
There are several markers of engagement with what others are saying.The first is whether participants start a new topic when it is their turn, respond to what another speaker has said, or continue their own thought from a prior turn.The more frequently participants start new topics of discussion rather than responding to what others are saying signals a lack of engagement in the topic at hand.If participants are continuing their thoughts from turn to turn, rather than responding to what others are saying, then there is a likelihood that people are talking past each other-in effect getting on soap boxes to declaim their views rather than talking with others.
The second marker of engagement in this research, which has not been discussed in the deliberation literature, is question asking.Questioning indicates engagement either with the topic or with fellow participants.Genuine questions attempt to illicit information from others, and hence as a process invite engagement with others.From a measurement perspective, it can be quite difficult to tease out genuine questions from rhetorical questions.This project sought to measure genuine questions meant to seek information from others (Pomerantz, 1988).Genuine questions were operationalized as statements that place the operator (who, what, where, how, etc.) before the subject (including statements like: is that a fair assessment?), statements with intonation rising at the end of the sentence, and directive questions, such as "please give me an example." Engagement also was measured by attending to metatalk, which is defined here as talk about the talk.Instead of advancing an opinion claim, this is talk that expresses what the speaker thinks has happened or is happening and why it is happening in the discussion.Metatalk in this coding has four instantiations: 1) metatalk that identifies some consensus; 2) some conflict; 3) that clarifies the speaker's prior opinion expressions; or 4) that clarifies some other speakers' prior opinion expressions.Metatalk was viewed as indicating engagement, because meta-talk cannot occur without some reflection on what others have been saying or on what a speaker has said before and which now appears to be misunderstood.Meta-talk that clarifies one's own position, for example, occurs only when a participant believes that his prior opinion or question has been misunderstood by at least one other participant.Given such a perceived misunderstanding, clarifying one's position is a signal that there is engagement occurring such that participants are hearing what others are saying and wishing to correct misperceptions of views.

Topic
This project coded whether participants were talking on the structuring topic, or on other topics that arose through the interaction (i.e. the interactional topic).Topics that were on the structuring topic (the topic established by the experiment, which was the problem of under-utilized schools) included talk about the regional choice plan, the move to K-8 schools, the small learning community approach to larger consolidated schools, and not consolidating any schools.Talk that moved away from those four topics was coded as "other" and considered to be off the structuring topic.The interactional topic was tracked in this project, but not systematically.Coders wrote a brief phrase that captured the topic of the turn.

Additional Measures
In addition to the six elements just described, some additional elements of the discussion were tracked.One of those elements captured thoughts about the technology or about the process of the deliberation.Thoughts were coded if they expressed concerns with the technology or had concerns or questions about the deliberation process.Both of these elements were important to track in order to see if participants had difficulty using the technology or understanding their task in the deliberations.Participants who cannot make the technology work, for example, will likely not be able to have a quality discussion.
Second, the moderator's thoughts were tracked and coded.Although the theoretical literature has little to say about the role or importance of a moderator, it stands to reason that good facilitation might be helpful to promoting good deliberation.Hence, this project tracked the moderator's behavior in the deliberation.Moderator thoughts were coded into one of 14 categories.The categories reflected the elements of the discussion captured in the participants' talk: process, problem, and social.Process talk tracked if the moderators asked participants to introduce themselves, statements of the process of the deliberations, or statements about the technology used to channel the deliberations.In addition the following were tracked: summaries of the discussion, statements or questions about who agrees or disagrees with a perspective, attempts to bring participants back to the topic if they are on a tangent, intervening if there is conflict between participants, and inviting quiet speakers to speak.Problem talk tracked the questions moderators asked.Social talk included salutations, praise, or any sociable chat.A few elements were quite rare, specifically intervention and bringing participants back to the topic who were off topic, and so were dropped from further analysis.

Determining Reliability
Two coders spent nearly two months developing and training with the coding scheme.Several amendments to the code book were made during those training sessions that further improved and clarified the codes.Several elements of the coding scheme proved especially difficult, including the meta-talk, questions, and disagreement categories.Coders were instructed to think of themselves as participants in the discussion, and to interpret and code each thought according to the code book.In instances where the codebook did not clarify how to code a particular thought's element, such as on whether it was a genuine or a rhetorical question, to code it based on the coders interpretation of the thought.The reason for this is that humans interpret meaning in interaction.If coders consider themselves to be part of the interaction, the hope was that such a framing would encourage them to be "natives" with the participants, and therefore to better understand what was being expressed.Moreover, no codebook can articulate every rule for every conceivable type of thought expressed.Human speech acts, while generally patterned, can and often do deviate from those patterns.In such instances, relying on the context and the interpretive abilities of the coders as meaning-makers helped to increase the level of reliability in the coding.Thus, when a coder came across an utterance that did not have a clear rule to help guide the assignment of code, the coder could reflect on the conversation and his or her interpretation of the utterance to assign a code.
After a final codebook was developed, coders practiced nine additional rounds of coding before taking a measure of intercoder agreement to assess reliability.The coding was done in two stages: the first stage unitized and the second stage coded the thoughts.In both stages, coders were instructed to listen to the audio file of the turn while also reading a transcript that was made for each group.Often, vocalic cues provided important information as to how a thought should be coded that a transcript alone could not reveal.
The intercoder agreement measures, which are detailed next, were established from coding 3 of the 23 groups, which were randomly selected.The first measure of agreement that needed to be established was the unitizing process (unitizing speaking turns into thoughts).The coders of the unitizing process achieved a statistically significant correlation of .86 (p < .001).
Cohen's Kappas of the coding elements described above are as follows: thought statements on the problem of the public schools, .95;elaboration, .86; the sources of support including mass media, .86,briefing documents, .82,other participants, .83,personal narratives, .87;turn type (new topic, continuing self, responding to others) .97;meta-talk, 1.0; structuring topics of regional choice, .85,K-8 .92,small learning communities, .93,status quo, .89,and other, .75;deliberation and technical problems .93;and finally, moderator thoughts .97.Thus, for the first research question, a coding scheme could be developed that had a sufficient level of reliability.

Descriptive Results
The results that follow focus only on describing the results at the level of the thought, since that was the level at which the coding was done.Additional analysis at the individual and group level is outside of the scope of this paper.

Reasoned Opinion Expression
With regard to reasoned opinion expression, overall, 3,482 or 55% of the total thoughts were expressions of opinion (including expressions of agreement and disagreement) (M=.84,SD=.37).Of those 3,482 opinion expressions, 571 or 16% had no elaboration.The rest, 84%, did.

Sourcing
Although participants might make use of other sources to support their opinions, this project coded for four specific sources: the mass media (N=88, M=.03, SD=.17), other participants who had spoken before them (N=194, M=.06, SD=.24), the briefing documents they were asked to read on the topic (N=613, M=.20, SD=.40), and personal stories or anecdotes (N=1020, M=.33, SD=.47).Personal anecdotes were used most frequently (33.46%), followed by the briefing documents (20.29%), then other participants (6.49%), and finally the mass media (3.01%).

Disagreement
Disagreement occurred in 5.6% of the thoughts that focused on the problem of underutilized public schools (N=351; M=.07; SD=.3).

Equality
The number of speakers within a group ranged from 5 to 12 (M=8, SD=2).The thoughts were also counted for the number of words per thought.Number of words ranged from 1 to 613 (N=6310, M=83.77,SD=85.71).The total number of thoughts spoken in a given group ranged from 159 to 353 (N= 6310, M=229, SD=48).A correlation analysis suggests that there is a strong relationship between number of speakers in a group and the number of words spoken.The higher the number of speakers the lower the number of words (R (6317) = -.06,p < .01).

Engagement
Analysis of turn-type indicates that participants primarily responded to what other participants said or responded to the moderator.Participants responded to a prior speaker in 83% of the turns (N=4339, M=.83, SD=.38), and responded to the moderator in 10% of the turns (N=547, M=.10, SD=.31).Participants introduced a new topic on their own and without prompting from the moderator in 6% of the turns (N=312, M=.6, SD=.24).The coding also captured whether a participant continued a thought from a prior turn.For example, a participant who takes multiple turns to continue their thought-regardless of what other participants might have said in the intervening turns.This was a relatively rare phenomenon.Only 1% of the turns were continuations of a speaker's thoughts from a prior turn (N=52, M=.01, SD=.10).
Questioning occupies a small number of thoughts in the overall thought expressions across the groups.Eight percent of thoughts (N=424, M=.08, SD=.27) were genuine questions seeking to solicit information from fellow participants.

Topic
Participant thoughts that focused on the problem of underutilized public schools tended to focus on topics other than the four choices participants were offered.Sixty-six percent of the thoughts were on some topic other than on the four choices (N=2658, M=.66, SD=47).With respect to the four choices, the most frequently discussed topic was the consolidation of elementary and junior high schools into a unified K-8 school (16.8%,N=676, M=.17, SD= .37).The other three topics were discussed almost equally.The philosophy of creating small learning communities in consolidated high schools was discussed in 10% of the problem thoughts (N=406, M=.10, SD=.30).The regional choice option, which allows parents to choose the school to send their children in the district, was discussed 9.7% of the problem thoughts (N=390, M=.1, SD=.30).The status quo option, which suggested that the current situation of under-utilized schools remain, was discussed 8.8% of the problem thoughts (N=352, M=.09, SD=.28). 4he "other" topics were broken into 112 categories ranging from discussions of the declining population in the city, to the school board, to the need for school uniforms.These 112 categories were then collapsed into 22 categories (N=2654, M=9.74, SD=6.2).Six categories comprised over half of the "other" problem thoughts.Discussion about school buildings comprised 12.1% of the problem talk (N=320).This talk included discussion about what to do with empty school buildings, the problem with over-capacity schools, the need to renovate old buildings, and class size.Issues of public education was the second most common "other" topic (9.3%).This topic included thoughts on the quality of education and its goals, the problems with students who cannot learn, grading and standardized testing, and the consequences of poor quality education.Talk about teachers was third most common (7.9%).This talk included complaints about the abilities of teachers, teacher salaries, teacher unions, and teacher's aids.The fourth most common other topic was discussion about school administration, including the school board and the superintendent (7.8%).Thoughts about students and about student behavior were fifth (7.1%).Sixth was discussion about parents, including the problems of absentee parents, the need for parental involvement, the reality of working parents, and the faults of bad parents (6.6%).These first six topics comprised half of the talk in the "other" category.The next five categories comprise an additional 25% of the "other" problem talk.These categories include an "other" category (5.8%), funding for education (5.4%), school programs, such as gifted programs, after school programs, and sports (4.8%), magnet schools (4.7%), and discussions about racial inequalities in the schools and in Pittsburgh (4.3%).
Some of these 22 "other" categories could still be considered "on topic" even if they were not on the topic of the four choices.The "other" category of magnet schools, for example, is still on the topic of the problem of underutilized public schools.The regional choice option in the briefing materials references magnet schools.Generally the participants would start a conversation on regional choice, and then talk about the current magnet school program and its pros and cons.The "on topic" categories included: issues of public education, busing, neighborhood schools, school buildings, funding of education, magnet schools, specific schools, declining population, and multiple solutions.The "off topic" categories included discussion about: teachers, parents, student's behavior, community involvement, activism, school programs, non-public schools, administration of schools, racial inequities, external factors, participant resources, other, and questions and answers between participants.Off-topic talk occurred in 57.6% of the thoughts (N=1528), and on-topic talk occurred in 43.4% of the talk (N=1126) in the "other" category.
An additional measure was created that combined the on-topic "other" categorized talk with the talk on the four choices to create an overall measure of on-versus off-topic thoughts.Results suggest that there was more off-topic (59%) than on-topic (41%) talk (N=6318, SD=49).

Problems with the Deliberation
Some participants experienced technical difficulties during their deliberations.Others had questions or concerns about how the deliberation was supposed to work.Of the total number of thoughts, the amount of talk about technical or deliberation problems was small.There were 143 thoughts that expressed problems with the technology (2.7%, M=.03, SD=.16).There were 152 questions or comments about the deliberation process (2.9%, M=.03, SD=.17).

Moderator Behavior
The bulk of the moderator's contributions to the discussions occurred at the beginning of the deliberation.The moderator delivered a somewhat scripted introduction to the participants that described the discussion process, as described in the methods section.Unsurprisingly, then, moderator talk about the deliberation process comprised 31.6% of the total number of thoughts from the moderators (N=324).What is revealing in this analysis, however, is that the next largest category of thoughts was questions on topics other than the four choices.This category comprised 21.2% of the thoughts (N=217).Social talk, which includes greetings, praise, and well wishes for a good lunch comprised 14.4 % of the thoughts (N=148).Thoughts about the technical aspects of the deliberation was the next largest category and comprised 13.8% of the thoughts (N=141).Technical aspects included instructing the participants to use the "ask to speak" button when they wanted to get into line to speak, and that participants could use the "jump" button if they wanted to move to the front of the speaking line.Of questions on the four solutions, specific questions on them comprised 8.7% of the thoughts (N=90).Of the four topics, questions about regional choice were the most frequent (2.7%, N=28), followed by questions about k-8 schools (2.3%, N=24), then questions about the status quo (2.1%, N=22), and questions about the small learning communities (1.6%, N=16).

Discussion
The purpose of this project was to offer a clear definition and operationalization of deliberation, and to develop a valid and reliable coding scheme that would help describe the process and content of deliberation.The code book developed to assess the quality of deliberation is grounded in empirical literature from the domains of conversation analysis, group studies, and in deliberation research.It also is guided by the theoretical and normative literature of deliberation, political conversation, and dialogue.The code scheme was relatively reliable, with all measures scoring .7 or higher using the Cohen's Kappa measure of intercoder agreement for nominal level data.
The results of the coding of the deliberation content suggest a mixed tale.The participants generally produced fairly a high volume of reasoned opinion.From the angle of the reasoned opinion expression measure, it appears that the groups were deeply engaged in the problem of under-utilized schools in their city.There also were some expressions of disagreement, although not very high.The presence of disagreement suggests that there was some heterogeneity in the views expressed, and that participants were hearing divergent perspectives.In the future, it might be of use, in addition to studying the measure of expressed disagreement, fairly engaged with participants attempting to clarify their views when they perceived others did not understand.The consensus measure also suggests that participants were taking the initiative to summarize the positions in the discussion.
Although it was noted whether participants were using what prior participants said as sources for their own viewpoints, this measure may also be considered a measure of engagement and reciprocity.When participants reference what prior speakers said as support for their own views, this additionally signals that participants are listening to each other and incorporating those perspectives into their own thinking.
The moderator measures suggest some problems.It is not a surprising result that participants were more likely to talk off topic than on topic given that the moderators were more likely to ask questions that were not about underutilized public schools.There also was a lack of uniformity in whether participants engaged in a round of introductions before deliberating.Only one of the four moderators had participants engage in an ice-breaker round so that participants could introduce themselves.Further analysis is warranted, but on the face of it, it is possible that such lack of uniformity could have consequences for other aspects of the deliberation.
Thus, in total, these deliberations seemed engaged, with a good volume of reasoned opinion expression grounded in the briefing documents and in personal experiences.The one serious note of caution is that the deliberations were less likely to be focused on the problem that brought them to the discussions-the problem of underutilized schools in their city.
The development of this coding scheme has important implications for research on deliberation.As noted earlier, there has been a lack of attention paid to what groups actually do when they deliberate.There is an assumption that groups are in fact deliberating, but that assumption requires testing.This code scheme provides a step in that direction.This manuscript did not delve into the analysis of the variance of groups.Further assessment is needed to determine whether groups deliberate equally on these measures of quality or if there is variance.Research by Gastil and Sawyer (2004) suggests that there is variance, although they did not measure content in order to arrive at that conclusion.Instead, they collected feedback from participants as well as rating assessments from external observers.Participant responses are possible in structured experiments, but for naturally occurring deliberations online or offline, collecting that sort of feedback can be difficult if not impossible.
Additionally, research is needed that analyzes the relationship between measures on the survey and elements in the deliberations.This data set, specifically, gives rise to the possibility to track individuals from their contributions in the deliberation to their responses in the survey.As a result, a next step in this project is to assess, for example, how expressed agreement and disagreement by the individual and by others in the group affects individual-level satisfaction with the deliberation experiment.
The coding scheme developed also aims to balance the need to capture the complexities of group interaction while being simple enough to achieve acceptable levels of intercoder agreement.Prior measures of deliberation quality have been rich but overly complicated (see, for example, Graham and Witschge, 2003), making it highly unlikely that coders could identify and accurately code at such a complex level.
Although this project only had two coders, it is quite possible to increase the number of coders, which would shorten the length of time needed to do the coding.Additional steps would need to be taken to ensure good intercoder reliability scores, such as taking additional reliability scores during the coding, to ensure that a coder is not straying from the established rules and producing results inconsistent with fellow coders.Training for multiple coders is somewhat longer and requires more practice sessions.
It is hoped that this coding scheme could be used by both practitioners and researchers of political deliberations.The goal of this paper is to help researchers consider elements of deliberation, their definitions, and ways to measure those elements.If one wants a full picture of whether deliberation took place, then all elements identified here should be considered.As Appendix B describes, the coding scheme highlights a set of core issues that require capturing to assess the quality of the deliberation.However, there are likely projects in which only certain elements, such as moderator behavior or whether participants are on topic, are most essential for assessment.This project, it is hoped, will provide a foundation for thinking about these elements and incorporating those that seem most relevant for a given research project.
Having noted that, it is hoped that the development of a coding scheme of the sort proposed in this paper could be applied in large part to other deliberation projects.This then would allow the scholarly and practitioner community to further assess how different types of moderation, different group sizes, the presence and absence of briefing documents or supporting materials, and the medium through which deliberation occurs, to name a few examples, affects the process and quality of the deliberations.Such a corpus of data would help us develop optimum deliberation processes, which needs to be on the agenda for deliberation researchers.
Appendix A: Codebook for Measuring Quality of Deliberation [This codebook is a shortened version of the codebook used for the Virtual Agora Project.It includes the descriptions for each of the codes, but does not include the caveats and rules that were implemented to analyze the Virtual Agora data.The purpose of this Appendix is to provide researchers with a general idea of the coding implemented.For further details or information about this coding, please contact the author.]

Unitizing Thoughts
Step 1: Categorizing Participants may speak for several minutes.During that time they may do many things, and the heart of the coding is to categorize what they are doing.In order to do that, their turn needs to be segmented into categories for further coding.This segmenting is the first stage of the coding process (after transcribing).
Each turn (that is each unique speaker), must be broken into the categories exhibited in the turn.The categories correspond with the categories in the codebook for coding thoughts.The four categories are problem, metatalk, process, or social.Talk about the problem is talk that focuses on the issue under consideration in these deliberations: school consolidation.Opinions, agreements, disagreements, facts, and questions all deal with the problem they are discussing.Metatalk is talk about the talk.It attempts to step back and assess what has transpired or is transpiring in the interaction, either as a group, or between individuals or to clarify meaning-one's own or someone else's.Process talk is talk about the technical and deliberation process (both immediate to the discussion, as well as the context for discussion).(Process talk may include questions, agreement, disagreement, metatalk, or facts, but pay attention to whether the question, for example, is about the process or about the problem.If the question is about the process, then it is categorized as process talk.)Social talk is talk that brings the strangers together by building (or harming) social bonds.
Once the segmenting is done, then the category segments must be broken further into thoughts.The "thought" is the unit of analysis for which the deliberations are coded.
offering a new thought.If they answer a question, and then continue to offer an opinion based on that answer, it should be 1 thought.
Question -a genuine question directed to another speaker that is trying to seek information or an opinion from others.Genuine question forms: • place the operator before the subject: How much does the school board make?Are you sure?Or a shortcut of that: Fair?For "Is that a fair assessment?" • assertion-questions: appears to be a statement with rising intonation at the end indicating a question: The school board doesn't approve of that?• Wh questions: who what where when why how.
• Directive-questions: "Give me an example" or "Ask me again."Rhetorical questions are not genuine questions, but instead seek to advance an opinion or argument in the form of a question.For example, "I don't understand why the school district built the new school.It was a waste of money.What was their reasoning behind that?"These are often questions that the participants could not answer or are not really meant to be answered.For example, "How do we get parents involved in their children's education?I mean, it seems to me that this is one of the most important elements of education.If the parents aren't involved, then how will that child succeed?"This set of questions is not attempting to invite other participants into a brainstorming session about how to get parents involved.If the participant was trying for this, she would likely cue other participants that she wants to have such a brainstorming session: for example "I've been trying to think about how to get parents involved who don't want to be.I am having trouble coming up with ways.Does anyone have any ideas?"It's important to note that sometimes these rhetorical questions will be answered or treated as genuine questions.That is a participant might reply to the rhetorical question: "How do we get parents involved" by saying "I don't have an answer to that question" or "maybe requiring parents who receive federal assistance to attend parenting classes."The uptake treats the question as genuine, but because of the way it was phrased, for the purpose of this coding, it should be coded as an opinion, not a question.

Metatalk
Metatalk is talk about the talk.Instead of advancing an opinion claim, this is talk that attempts to step back and observe what the participant thinks has happened or is happening and why it's happening.The thrust of the thought must be talk about the talk in order for the thought to be coded as metatalk.Consensus -.Consensus metatalk is talk about the speaker's sense of consensus of the group ("I think we all agree that . . .."), including an explanation for the collective's opinions or the collective's behavior ( We're asking you these questions because ..).Conflict -highlighting some disagreement or conflict in the group ("I sense some disagreement around . .

Process
Technical Problems -a question or statement about problems with the technical features of the system, statements of confusion about the system, the software, whether anyone can hear them, or that they can't hear, or don't know what they're doing with the software.Technical Benefits -a statement about the positive aspects of the technical features of the system, praise for how good the software works.Deliberation Process -a question or a statement about the process of the deliberation to moderator or other discussants (without any valence), questions about the moderator (such as his or her absence) or to the moderator about what they are supposed to be doing, or statements about the surveys before or after the discussion or about any element of the process before or after their conversation.Questions or statements about what they should read in the briefing documents.Deliberation Problems -frustration about the process of what they are supposed to be doing, expressions or questions of confusion about the task or the procedure, suggestions that the participants have strayed off the topic and the participant is trying to get them back on the topic.Deliberation Positive-a statement about the participants' belief that the discussion has been good for them, good for the group, or potentially good for Pittsburgh, the school board, the mayor, the students, the parents, and the like.

Social
Salutations -statements of welcome, greeting, hello, and good bye, see you later, and the like.Apologies -statements of apology: I'm sorry, and 'I hope' statements, such as "I hope I haven't been too obnoxious."Includes statements of reflection of how the participant performed in the group (likely comes at the end of the group's discussion): "I hope my few ideas did get across."Praise -includes thank you, you're welcome, as well as praise for other individuals or the group ("you've been a good group."I've really enjoyed myself," "this has been fun").Praise in the service of an argument about the problem is coded as a problem thought ("I want to commend Sally for volunteering at her school.We need more people to be volunteers").ChitChat -thought statements that are not on topic relative to the deliberation.These could be jokes or puns (but not as they relate to the problem of schools), social chit chat about the weather, and the like.
Topic: all thought type: problem and metatalk require a topic code.Determine the main claim of the thought.If it's on one of the choices, indicate yes.Otherwise, select yes for "other" if the main claim is not about one of the choices.Notes on Topic: Topic, loosely understood as the subject of the thought, is something that interactants coordinate together.People, in general, abide by rules of interaction, which include trying to do what they can to help the person(s) they are speaking with understand them.This includes making sure that the listener understands what the topic is.Now, when people talk, they don't continually talk on the same thing; the topic shifts over the course of an interaction.People may start by talking about the weather, and then move to what they are doing this weekend, to the best movie they've seen in awhile.Weather, activities over the weekend, and movies, would be considered distinct topics, although they likely related in some way.The weather affects the activities one can do on the weekend, and the weekend is a good time for movies.Thus, there is often some relationship between the old topic and the new topic.This makes differentiating new topics oftentimes a bit tricky in this coding.For the purpose of this coding, topic will be differentiated in three ways: 1) by policy choice -participants were asked to speak on four policy choices, status quo, regional choice, k-8, and small learning communities.They also tend to discuss a number of other topics: busing, what to do with old buildings, quality of teachers, parental involvement, problems at the school board, declining population in Pittsburgh, cost of education, the quality of education, and segregation in the schools.2) by re-introducing topics that occurred in a prior turn or in a thought not immediately preceding the current thought.3) by introducing topics that have not yet arisen in the discussion Regional choice -the option to send children to any school in the region.K-8 grade structure -remove middle schools, and shift 7 th and 8 th grades to be housed in the same building as K-6.Small learning communities -reducing the size of the overall population of the schools to house less than 800 students total, or create an environment within large schools that feel "small."Status quo -Keep all the schools open-don't consolidate them in any wayand let them be half empty.Conversations that are about being in favor of closing schools in general, without reference to the three options (unless it's in passing), is a statement against status quo.So, if they talk about closing schools because of the k-8 option, then the topic would be k-8.If they say "no matter which option we choose, k-8, slc, regional choice, we need to close schools" then the focus is on closing schools.
If thought is one of the four choices, then select the valence: For -arguing for the choice or offering a for argument without explicitly stating for For-but -arguing for a choice but offering some quibble or hesitation with the choice or how the choice would be enacted.Includes "I think it's good, but . .." or "I think small learning communities are an ideal solution.Whether or not we'll see them, I don't know" or "I think the cost of the K-8 system is retrofitting the schools.What that cost would be, I'm not sure, but it seems like it's minor."Against -arguing against the choice or offering an argument against the choice without explicitly stating against.
Against-but -arguing against a choice but offering some positive of the choice or how the choice would be enacted, or a counterargument to their against argument (i.e.making a pro/con argument).Includes "I think it's not a good solution, unless they. .." or "I think small learning communities are a bad idea if there are several administrations in them.But, my children benefited from the magnet school" Unsure/None/Both equally -expressing hesitancy with the option (I read that there's a problem, but I'm not sure whether it's true or not; need more information).Includes questions about the topic that are not valenced.Includes pro con argument where it's unclear if they are for or against the choice.Includes, if they make a claim that does not indicate their valence.Other -anything that isn't about the four choices (includes talk about the school board, teachers, parents, costs, test scores, students with disabilities, demographics, declining population, what to do with empty buildings, quality of education, school reputation, empty buildings or old buildings, political activism, voting, and getting involved, school uniforms, magnet schools, prior school closings).They may mention one of the choice options, but if the focus of the thought is not on one of the choices, select this.For example, if a participant's opinion claim is that the school board is incompetent, and says that the school board will never be able to get k-8 enacted, the thought should be coded as "other." Topic -write a brief phrase of the topic Is there elaboration of the problem or metatalk thought?An elaboration is a statement (a claim) with some additional elaboration.Elaboration can be in the form of further justification (as simple as: I'm for k-8, a given deliberation, or might reflect some additional interest the research might have.

3c. Tracking Elaboration for Problem Talk
Talk that is about the problem can also be assessed for whether there is elaboration that supports the opinion being expressed.This is the second component to the "reasoned opinion expression" measure.The type of elaboration can also be tracked.If the deliberations include background documents or a stimulus of some exposure to a news story, then the researcher might wish to track when such was referred to.

Coding Moderator Thoughts
If there is a moderator, the moderator's thoughts should be tracked.If the moderators were given explicit instructions on how to steer the discussion, those should be tracked.Researchers may wish to mirror the problem, process, social types of thoughts used to track the participants' thoughts.
. .")Clarify own -clarify the speaker's own opinion or fact statement ("what I'm trying to say is").It's an attempt to clarify what the speaker means.This will arise ONLY after they've provided an opinion, NOT a question, and are now trying to clarify their original opinion on the problem, likely because they believe someone has misunderstood them.Clarify other -clarify someone else's argument/opinion or fact statement ("Sally, so, what you're saying is . .").It's an attempt to clarify what someone else means.Pay attention to the use of another participants' name.That can be a sign of metatalk of another's position 3b. Tracking Topic for Problem Talk Topic categories can be constructed in advance if the researchers have knowledge about what topics they wish to track in the discussions or what topics are likely to arise during the conversations.It likely will be necessary to have an "other" code, which after the initial round of coding might require further analysis.Researchers may also track valence of the topic, that is what side of a position the participant takes on the topic being discussed.