The Magic Sauce: Practices of Facilitation in Online Policy Deliberation

Online engagement in policy deliberation is one of the more complex aspects of open government. Previous research on human facilitation of policy deliberation has focused primarily on the citizens who need facilitation. In this paper we unpack the facilitation practices from the perspective of the moderator. We present an interview study of facilitators in RegulationRoom – an online policy deliberation platform. Our findings reveal that facilitators focus primarily on two broad activities: managing the stream of comments and interacting with comments and commenters – both aimed at obtaining high quality public input into the particular policymaking process. Managing the immediate goals of online policy deliberations, however, might overshadow long-term goals pf public deliberation, i.e. helping individuals develop participatory literacy beyond a single policy engagement. Our contribution is twofold: we unpack the practice of human facilitation in online policy deliberation, and suggest both design and process implications for sustainable growth of civic engagement environments beyond the individual case we analyze.

Civic engagement in policymaking carries a great promise for increased legitimacy and acceptance of policymaker decisions, greater accountability of and trust in government institutions, and overall stronger democracy (Cramton, 1972;. In the U.S., Web 2.0 and government enthusiasm brought a strong promise of more open and citizen-centered policymaking processes enabled through technology (Shirky, 2008(Shirky, , 2011Towne & Herbsleb, 2011). In practice, however, participation of individual citizens in online policy deliberation has been limited in both scope and quality. For example, regulations.gov, a U.S. government website established to enable public comments on rules proposed by federal agencies, is used primarily by professional policymakers, as opposed to the general public (Bryer, 2013;Coglianese, 2006). One reason is that effective policy deliberation requires a certain kind of participation literacy that includes reasoned arguments and provision of facts, consideration of social, political, legal and organizational constraints, and understanding of what is workable versus what is ideal (Carcasson & Sprain, 2016;Epstein, Newhart, & Vernon, 2014). Instead, popular mass online civic participation often takes the form of a large volume of short, conclusory, formulaic comments that have limited utility in the policymaking process (Shulman, 2009). Lacking depth and reasoning, mass civic participation is often viewed as disappointing in its usefulness to decision-makers and the polity (Buckingham Shum, 2008).
One way to address this gap is having moderators facilitate public deliberations. In face-to-face deliberations, human facilitation plays a critical role in helping citizens participate meaningfully despite being inexperienced in the established, technocratic, procedurally complex, and legally constrained policymaking process (Kaner, 2007;Levine, Fung, & Gastil, 2005;Quick & Sandfort, 2014). Similarly, in online situations, active and thoughtful human facilitation may help improve the overall quality of deliberation (Albrecht, 2006;Rhee & Kim, 2009) and make the process more inclusive (Albrecht, 2006;Trénel, 2009).
While earlier studies focused primarily on the citizens who need facilitation in policy deliberations (e.g. Trénel, 2009), in this paper we unpack the facilitation practices from the perspective of the moderator. We interviewed a cohort of moderators working with RegulationRooman experimental civic engagement platform that hosts public consultations about federal rulemaking proposals. We probed into the moderators' practices and their perceptions of the community of participants.
Our findings depict two primary kinds of activities moderators practice in facilitating online policy deliberations. First, they manage the stream of comments to keep the discussion in good order. Second, they directly interact with commenters to facilitate the solicitation of high quality public input into the policymaking process. Both activities are aimed at achieving more and better public comments in the short term, which potentially creates a tension with broader, long-term goal of helping individuals develop participatory literacy beyond a single policy engagement. We discuss our findings through the lens of a dual challenge of keeping content quality high in the short term while scaling online policy deliberations in the long-term. We highlight tensions in accomplishing these goals under constraints of limited moderation resources and suggest design avenues to automate some maintenance tasks, potentially freeing resources for building a community of civic-minded individualstensions and aspirations pertinent more broadly to online civic engagement and deliberation environments.

Moderation and Facilitation in Online Policy Deliberation
Literature on public deliberation acknowledges that quality deliberation is not naturally occurring or self-generated (Carcasson & Sprain, 2016;Lee, 2011;Levine et al., 2005;Zhang, Cao, & Tran, 2013). The literature has addressed the constructive role of facilitation, particularly in the physical settings (Levine et al., 2005). In the words of Dillard (2013), "It is the work of facilitators that turns 'everyday political talk' into rigorous deliberative exchanges" (p.218). The role of the facilitator has been discussed as that of an "empathetic critic" (Carcasson & Sprain, 2016, p. 51) engaged in the search for common ground and conflict resolution, as well as one who promotes "processes of problematization, reflexive learning, and citizen empowerment" (Fischer, 2004, p. 26). Facilitators have also been discussed as enablers of engagement for inexperienced participants (Carcasson & Sprain, 2016;Ryfe, 2006), as educators of participatory expertise or "the capacity for personal autonomy and constructive interpersonal relations that the governance of the lives of real people require" (Rosenberg, 2003, p. 15).
In the world of face-to-face deliberation, the styles of in-room facilitation, coupled with elements of process design, have been discussed in the context of assessing deliberative outcomes (Russon Gilman, 2012). There is a general consensus about the benefits of facilitation to deliberative processes, and about it being both form and context dependent. Passive facilitators, for example, can expose the deliberative process to takeover by vocal interest groups (Dillard, 2013), while involved facilitators run the danger of introducing bias (Spada & Vreeland, 2013) or limiting the individual autonomy of participants (Levine et al., 2005). As to the facilitators themselves, research on face-to-face facilitation has looked into how facilitators acquire their skills (Quick & Sandfort, 2014) and what practices are used in specific deliberative situations (Black & Wiederhold, 2014). While rich, this body of research is confined primarily to the physical, typically small-group deliberative settings. The question thus is what happens when deliberation moves online?
The internet allows scaling deliberative processes, but it also introduces a layer of complexity absent in traditional, physical settings. Thus, studies of online deliberation have been mainly concerned with effects of technological mediation, or that of affordances and design of online spaces, on deliberative outcomes (Friess & Eilders, 2015). Some earlier research compared deliberative outcomes of face-to-face and online deliberation, showing that both have comparable impact on issue knowledge, political efficacy, and willingness to participate in politics (Min, 2007). Later research asked more nuanced questions about the affordances of deliberative platforms (e.g. interactivity as in Brinker, Gastil, & Richards, 2015) and offered generic platform design recommendations (Towne & Herbsleb, 2011). When facilitation was present in some of those platform-centric studies (e.g. Min, 2007), it was not interrogated or unpacked, but treated as a single variable or part of the experimental design.
Online policy deliberation spaces have higher barriers to entry for the public than the popular discourse would like us to believe. Policy deliberations are unique in a sense that they constitute deliberative events that are "typically one-off experiments that occur within the confines of a single issue over a short period of time" (Nabatchi, 2014, p. 1). Further, they follow linear processes, and result in an output that reflects the conclusions or a consensus of the deliberating group (Carcasson & Sprain, 2016). Interfacing with bureaucratic policymaking institutions, policy deliberation is constrained by structures, regulations, procedures, practices, and processes that require a very particular kind of participatory literacy. Effective participation in policy deliberations requires a certain level of subject matter expertise as well as reason-giving and, where possible, evidence-based substantiation of one's opinions on specific issues (Farina, Epstein, Heidt, & Newhart, 2014;Parker, 2002). Novice participants, however, often default to voting-like behaviors of registering their preferences or unsubstantiated sentiment expression. Lacking depth and reasoning, mass online civic participation (e.g., online petitions) is often viewed as disappointing in its usefulness to decision-makers (Shulman, 2009). As such, there is a growing consensus that technical solutions alone cannot adequately address challenges of effective online deliberation. Not only, online communication requires a level of technical expertise, it has limited affordances for non-verbal cues and interactivity Friess & Eilders, 2015;Zhang et al., 2013).
Human moderation in online spaces is one way to facilitate better and more inclusive deliberative engagement (Coleman & Gotze, 2001;Farina, Epstein, Heidt, & Newhart, 2013;Manosevitch, 2014;Wright, 2009). At the basic level, such moderation includes mostly administrative tasks related to content, maintenance, and technicalities: providing content, setting the agenda and the schedule, curating and sometimes policing messages for lack of adherence to site use guidelines, and removing duplicate or repetitive posts (Edwards, 2002;Trénel, 2009;Wright, 2009). Focusing exclusively on this level of moderation, however, can be potentially counterproductive. Some studies showed such moderation leading to higher levels of participants' suspicion towards the process (Wright, 2009) and exclusion of traditionally underrepresented populations (Trénel, 2009). Other studies demonstrated that different levels of moderation can lead to various levels of politeness and quality of reasoning, as a function of group homogeneity (Zhang et al., 2013).
At a more sophisticated level, moderation involves advanced (Trénel, 2009) and interactive (Wright, 2009) facilitation activities. These activities help create a respectful climate, encourage interactivity among the participants and between the participants and decision-makers or subject-experts, balance participation, situate the discussion within the political and legal context, and elicit relevant information (Edwards, 2002;Rowe & Frewer, 2005;Trénel, 2009;Wright, 2009). In these activities, moderators serve as third party intermediaries. This kind of facilitative moderation was found to lead to greater engagement (Wright, 2009), higher inclusion of traditionally excluded populations (Trénel, 2009), more relevant information for the decision-makers (Rowe & Frewer, 2005), and improved dialogue between members of the public and policymakers (Edwards, 2002).
Previous work on moderation in online policy deliberations looked at moderation from the perspective of the participating citizens (Trénel, 2009), the decisionmakers initiating an engagement (Rowe & Frewer, 2005), or the deliberative process as a whole (Edwards, 2002;Manosevitch, 2014;Wright, 2009). Lacking is the perspective of the moderators themselvestheir experiences and practicesin moderating online policy deliberations. Inserting this perspective, however, is necessary in designing online deliberative systems that aspire to meet the needs of all involved parties. We add to existing literature by focusing explicitly on facilitative practices and on policy deliberation that takes place online.
In completing their tasks, moderators of online policy deliberations are directed toward goals at three levels. At the immediate level, the goal is to keep the discussion in good order. The mid-level, short-term goal is collecting high-quality content to be relayed to policymakers. And in the long run, they work toward building a community of civic-minded individuals. What do moderation practices look like in trying to effectively moderate online policy deliberations and achieve all these goals? What are the tensions moderators face in balancing various moderation goals and activities? What strategies do they apply to overcome these tensions? In answering these questions, our goal is to identify opportunities for achieving the three goals of moderation through moderation processes, technology design, and community engagement.

Research Site: RegulationRoom
RegulationRoom 1 was developed by CeRIthe interdisciplinary Cornell eRulemaking Initiative. Working with federal agency partners, RegulationRoom is an online platform that solicits public participation in commenting on and discussing live federal proposed rulemakings. Since its launch in 2009, the site hosted seven live public consultations on proposed rulemakings.
RegulationRoom is intentionally designed to support and encourage deliberative participation by lay citizens (Farina et al., 2014). Lengthy policy proposals written in legal/professional language ("legalese") are translated to short topic posts in plain English (with links to original documents), and presented side-by-side with the comment stream ( Figure 1). This makes it easier for participants to learn about the policy under discussion and highlights questions where public input would be particularly useful. It also signals the centrality of the proposal content to the discussion.
At the heart of RegulationRoom efforts is human facilitation. Graduate (mostly law) students serve as moderators on the site, as part of an e-government course clinic. They learn about conflict resolution, interpersonal communication, social psychology, and plain-language writing. They are trained in active listening, neutral and open-ended questioning, and other facilitation techniques. During live consultations, moderators work in shifts of 4-12 hours. Shift length is determined by collective discussion among the moderators and the fellow, and may change over the course of the discussion depending on comment volume and external events such as academic breaks or exams. A second, back-up moderator may be scheduled if very heavy comment volume is anticipated. A moderator is responsible for all the comments posted during his/her shift. If a moderator experiences a large volume of comments and needs assistance, he or she is to contact the clinic instructors.
Moderators use a protocol that outlines their responsibilities and they receive ongoing support and supervision from clinic instructors. Until a student demonstrates an acceptable moderation proficiency level, they submit draft responses to instructors for discussion and approval. Even when moderating independently, moderators consult with instructors about challenging comments or when encountering situations they need help with. This process of drafting, discussing, and revising responses is considered a key part of learning effective facilitative moderation in the clinic.
To help manage the process, the team uses a moderator interface (Figure 2), which allows them to sort comments chronologically, by topic, by user, or by status. Moderators use the interface to assign each comment a status (unread, in progress, done), indicate the nature of the moderation response (replied, no reply, recommended) and, in very rare cases of site use violations, redact or quarantine comments. They can also add notes on a comment, to be viewed by the instructors and other moderators. Responding to a comment is typically done in the main interface ( Figure 1), from which a moderator can also take actions such as changing a comment status. During most of RegulationRoom's operation (including the discussion upon which this study is based), moderators logged in the system with their individual accounts but their responses appeared publicly on the site signed with a single "moderator" persona 2 .

Figure 2: RegulationRoom moderator interface
Earlier CeRI studies focused primarily on the main web interface design and participants' experiences on the platform (Farina et al., 2013(Farina et al., , 2014Farina, Newhart, & Heidt, 2012). Although moderators are a key part of RegulationRoom and a great deal of effort is dedicated to their training, guidance, and design support, their experiences and practices have not been systematically examined. To better understand the role of the moderators in facilitating productive online policy deliberations, we carried out a study that explores the perspective of the human moderators in the system.

Moderators Interview Study
The Consumer Debt Collection Practices Proposal From November 6, 2013 to February 28, 2014, people could use RegulationRoom to learn about and discuss a proposal by the U.S. federal Consumer Financial Protection Bureau for new regulations on consumer debt collection practices. Supervised and guided by an administrative law professor, a subject matter expert (law professor), a senior researcher, and an e-government fellow (a law school graduate who had taken the clinic for several semesters), the moderation team participated in a set of activities leading to the discussion: they helped prepare plain English versions of the proposal materials, assisted with setting up the site, and conducted outreach activities to raise awareness and reach individuals who are unlikely to participate meaningfully in the conventional process, but can contribute to the discussion.
During the 115 days the discussion was open, 8,480 unique visitors came to the site. There were 12,629 total visits. 377 people registered as users. Moderators posted a total of 250 responses to 956 comments by 224 commenters.

Interviewees
During the Fall 2013 academic semester when the discussion started, 17 graduate students (16 law; 1 masters of public administration) were involved in moderation. One of them was an experienced moderator who had participated in the prior year's clinic. During the Winter break when students were away, the senior researcher and the e-government fellow moderated the discussion. Eight students, including the more experienced moderator, continued in the clinic in the Spring 2014 semester and moderated during the final part of the discussion.
In April 2014 we interviewed the eight students (2 female, 6 male) who had moderated the Consumer Debt Collection Practices discussion in both semesters. We also interviewed the e-government fellow as a pilot run. These interviews add to our own ongoing experience as participant observers of CeRI design, training, and consultation management processes in the past three years.

Procedure
The interview was carried out by a graduate student who did not have previous relationships with the interviewees. The interviews followed a semi-structured protocol, and took place in one of the CeRI offices. After obtaining the interviewee's informed consent, the interviewer opened on a desktop browser the moderator interface as well as the main interface (Figure 2 and 1, respectively) in a non-live version of the site that had been populated with selected comments from earlier RegulationRoom discussions. All interviewees except the experienced moderator were unfamiliar with these discussions and comments.
The interviewer asked questions about the practical process of moderating, such as: in what order the moderators read and respond to comments, how they track and organize comments, their thought processes and actions in making decisions about responding and crafting replies. Using a "think aloud" technique, the moderators answered the questions while walking the interviewer through the interfaces they use to demonstrate their moderation actions and decisions. This technique, common in user studies (Ericsson & Simon, 1984), helps substantiate responses to questions with real examples of actions, decisions, and thought processes. Interviews lasted 52 minutes on average (min 31 minutes, max 79 minutes).

Data Analysis
Interviews were recorded using Snagit, software that synchronously captures the computer screen and the interview audio, resulting in almost 8 hours of recordings. Audio recordings were fully transcribed using a transcription service.
We used an open-coding technique, iteratively reading the transcripts, highlighting excerpts, and identifying key insights, themes, and reoccurring patterns in the data. In our report on the findings below, all names are pseudonyms.

Findings
The interview started by asking interviewees to show, step-by-step, and talk through what they do when they facilitate discussions on RegulationRoom. Moderators described logging into RegulationRoom soon after their shift started, going first to the moderator interface ( Figure 2). In this interface, they see all the comments that were posted on RegulationRoom. After skimming through the most recent comments, moderators described clicking on a permalink from the comment, which takes them to the main interface ( Figure 1). Moving between interfaces and activities, moderators described two main aspects of their role. First, they triaged the comments as a way to manage the comment flow. Second, they interacted with comments and commenters in the effort to promote participatory literacy and an effective deliberative discussion.

Comment Flow Management
Every comment added to the site requires the moderators' attention, which included reading the comment and the surrounding context, deciding whether and how to respond, drafting a response, and emailing back and forth with supervisors until their response was approved and could be posted to the site. Moderators recognized that per each comment this was a labor-intensive and time-consuming process, and they developed strategies and mechanisms to effectively manage the moderation process as a whole.
Moderators made decisions on the order in which to take care of comments during their shift. When the number of comments was small, which was often the case, moderators told us that they worked on comments in the order in which they were posted: "My shifts never get more than two to eight comments. I'd start with the ones that were chronologically earliest on my shift" (David). However, especially at the beginning of the commenting period of the consumer debt collection practices consultation, the site received a lot of attention from the public, with a few hundreds of comments posted within the first few days. As a result, moderators had to develop strategies for sorting, prioritizing, and triaging comments, deciding which to pay attention to first, and which to leave for later. While this was an unusual case in the history of RegulationRoom, these strategies may be particularly interesting as they highlight the challenges of scaling human facilitation of effective online civic engagement with limited moderation resources.
One strategy moderators applied was to skim all the new comments and search for uncivil comments. Such behavior in RegulationRoom is extremely rare, with only a couple of comments that included inappropriate language or personal attacks in a history of seven consultations with thousands of comments over five years. Yet, such behavior is considered harmful to the participating commenters and to the process as a whole, because it can limit civil and productive discussion. The moderator training therefore emphasized civility policing as a priority especially at times of high volumes of commenting activity: I Another strategy moderators developed was to prioritize welcoming newcomers, and only then attend to comments from returning commenters. Moderators checked to "see if this was a first-time commenter, because our protocol was always if it was a first-time commenter, to always respond" (David). Past RegulationRoom discussions demonstrated that although most commenters leave only one comment and do not come back to engage further in the discussion, responding to these first timers increased the likelihood that they will return. When a commenter receives a response to their comment, an automatic email is sent to them with the content of the response and a link to the discussion.
Moderators' training and practice to prioritize responding to new commenters was therefore rooted in the typically episodic nature of participation in deliberative events and consequently heavy emphasis on initial engagement.
A more complicated strategy some moderators applied when faced with high volumes of comments was first responding to comments that contributed to the current policy deliberation in a straight-forward fashion, and later responding to more ambiguous comments in which the contribution was not as clear. To apply this strategy, moderators described judging each comment within the context of the proposal and the discussion around it: whether it only expresses an opinion or also includes support for the opinion in the form of one's personal experiences or verifiable facts; whether it sheds new light on issues related to the policy proposal; brings in new ideas, opinions, or experiences; is directly related to the proposal being discussed; and, offers constructive feedback. This required delving into the comment and context of each comment, and the writing style was sometimes used as a heuristic. Moderation of verbose or poorly written comments ("when a comment starts with random capitalization and misspelled words it's going to put off the reader" (David)) was sometimes postponed, after triaging and taking care of comments that were quicker to understand and respond to: Moderators are expected to pay full attention, consider, and respond to comments regardless of their workloads and comments' contribution or writing style. What we see here is the development of practical heuristics on top of these expectations of ideal moderation based on formal training and protocols.

Interacting with Comments and Commenters
Establishing context. Using the moderator interface primarily to manage the flow of comments, moderators typically moved to the main interface to read and handle specific comments. In the main interface, they could gather information that contextualized the comment in relation to the policy section it was attached to, other comments in the conversation, and the commenter who posted it. This allowed the moderator to get a better sense of what the comment was about and decide what to do with it: I will read the comment, I'll go to the permalink. This will give me the context of where that comment is so I can see the whole comment thread, because it's tough to read a comment on its own and know exactly what they're talking about (Gavin).
A commenter's record of activity on the site was retrieved by clicking the commenter's username, which opened a new page with the user profile that listed other comments written by that commenter. From there, moderators further opened additional pages for each comment written by that commenter in the context of the proposal section on which it was posted:

Then, I also open the commenter, double [check] to see if this is a firsttime commenter, or if this is a person who's commented multiple times. If they've commented multiple times, I go back and read their comments again, and see if they […] need a response (Andy).
In this process, the moderator interface was useful as a starting point of the moderation session and to manage comment flow. However, it did not provide enough information for getting a full view on each comment and crafting an appropriate response. Instead, to get all the information they needed, moderators opened multiple browser tabs, one or more for each comment that came in during their shift and for other relevant pieces of information for each comment they moderated. To establish the context for each comment, they alternated between the open tabs, closing them one-by-one as they received the appropriate context they needed to moderate each comment and write the appropriate response.
Crafting a Response. At the heart of the facilitative moderation activity is responding to comments. The main purpose of this activity is to solicit more, better participation through informed and substantiated arguments that are specifically related to the topic being discussed and to other comments in the discussion. Through their responses, moderators facilitate the deliberative process, model community norms and commenting behavior, and educate the public toward meaningful and effective engagement in the policymaking process.
To achieve these ambitious goals, the moderators echoed the emphasis from their training and protocols on active listeningtrying to understand what the commenter is saying and finding the relevance in their comment: This task was not always easy. Moderators considered some comments difficult to respond to because their contribution to the policy proposal was not immediately evident. These include, for example, comments that do not explicitly address the policy ("One that has absolutely nothing to do with the proposed rule. Someone who just saw 'debt collection' and wanted to talk about a debt they had" (Andy)), have unrealistic suggestions for the agency ("if an idea is so farfetched, and it's not grounded in a reality at all, that can be a very tricky comment" (Jake)), or do not provide any grounding to the expressed opinion ("just conclusory statements that have no support" (Tyler)).
Working with the more challenging comments required additional time and effort on the part of the moderators: moderators had to read them multiple times, look for comment context and commenter history to find out what the comment was about, and carefully craft the best response. To streamline response formulation during periods of high commenting activity, moderators described getting a quick understanding of what type of comment this was, and following heuristics that match appropriate responses to the type of comment: "generally," David explained, "you can tell what type of comment it's going to be from the first few lines." Some of the heuristics described earlier for managing comment flow were also applied in composing responses, such as civility policing and welcoming newcomers. Moderators described a variety of additional heuristics for categorizing comments and composing an appropriate response. For example, they responded to misplaced comments, attached to one section of the proposal but referring to a different section, by directing the commenter to the relevant section: "ones that are off topic, that's going to be just a quick, "Hey, this was in the wrong topic. Maybe you should check out over here"" (Jake). Another heuristic involved asking for clarifications or requesting additional information to complete one's comment: If I find a comment that has a statement that has no foundation I would probably respond to that comment first because it would be a quicker response to say, "Hey, why do you think this?" or "Could you tell us a little bit more?" (Tyler).
Following heuristics in identifying comment types and formulating responses accordingly may be useful to moderators who are trying to respond to many comments in a limited amount of time. However, this sometimes led to what Jake called a "stock reply," and Jeff called "canned responses, where if they mentioned this specific thing, this is what you say to them." In their training and ongoing guidance, moderators were encouraged to formulate new, personalized responses to each comment they replied to as much as they could and were warned against providing "robotic" sounding or repeating responses. Moderators found this to be a challenge, especially when encountering comments that moderators have seen similar ones before, and was seen as a challenge against the role of moderators as attentive, active listeners.
Dealing with commenters. As seen above, the person behind a commentand especially his or her history and interactions on the siteplayed an important role in making decisions about managing comment flow and responding to comments. In their interviews, moderators used language that suggested an understanding that they were not simply reading comments and responding to them, but that they were directly communicating with the people who wrote the comments and they were genuinely invested in building relationships with them. For example, Carol explained how her interactions with commenters could have a positive effect on building participatory skills: "Responding to people the sooner the better helps them kind of change their behavior and see that somebody is actually reading what they have said. It might encourage them to come back and engage more productively." On one end of the spectrum were the first timers. Like many other sites with a "long tail" distribution of contributions, first timers were the majority. As discussed above, moderators prioritized responding to first timers under the assumption that a response, coupled with automated notification, increases the chance that commenters will come back to engage in the conversation: We talked yesterday in class [about]  In between first timers and serial commenters, moderators identified the commenters who come back and provide more information about a previous comment they made as a result of a moderator's response. Seeing the returning commenter's comment in the context of the conversation helped moderators decide if further intervention was required: "I replied to someone, basically asking them to clarify one detail. They responded back [...]. I see that and know, that's not a pressing thing. I don't need to reply to that" (Gavin).
A small number of returning commenters appear to understand well effective commenting and meaningful participation in RegulationRoom. They often come back and provide thoughtful comments on multiple sections of the proposal. Moderators can recommend these comments which then appear to other site visitors as "Recommended by the moderator for demonstrating effective commenting skills," thus modeling good commenting practices to other commenters. However, because these comments do not require intervention, and their authors are perceived as already engaged in the process, often moderators chose not to reply: I'll look for the good comments, the best comments from returning commenters. I just will not reply. If they're a returning commenter, they've been here for a long time. We know that they are going to come back. They don't need a whole lot of moderation, and they've shown a willingness to go elsewhere on the site (Jake).
Given the goal of RegulationRoom to solicit more, better public feedback that is later relayed to the proposing agency, moderators focus on encouraging new commenters to come back and improve the quality of their comments. The moderators pay less attention to commenters who post repetitive low quality comments with the purpose of not encouraging this behavior. Further, they also pay less attention to those who already demonstrate good participatory skills through responsive, thoughtful, and recurring comments. We identify the latter as a potentially underutilized opportunity to develop a civic-minded community with these returning, engaged commenters, who understand effective public participation in policymaking and who may potentially care about the quality of participation on RegulationRoom beyond a single engagement.

Discussion
Our findings demonstrate the complexity of the moderators' practices in RegulationRoom. The bulk of this practices centered around managing the stream of comments and responding to commenters. Managing the stream of comments included detecting incivility, identifying misplaced and duplicate comments, making sure newcomers are welcomed, and responding to the more straight forward comments first, before spending more time and effort on the more complex comments. These were directed toward achieving one immediate goal of the RegulationRoom: to keep the discussion in good order. Responding to comments was directed toward a second, more primary goal of RegulationRoom: to facilitate the solicitation of high quality public input into the policymaking process. To achieve the second goal, RegulationRoom moderators undergo intensive training, receive ongoing guidance and mentorship, and are equipped with a detailed protocol and a custom-built moderator web interface. Active listening, neutral and open-ended questioning, and careful attention are seen as necessary in order to facilitate a fully deliberative environment, in order to help the agency in gathering quality feedback about a policy proposal from the public.
One factor that is crucial to constructing and maintaining effective online policy deliberation environments is managing the tension between dedicating resources to tasks related to these two goals. Maintenance tasks are required to keep the deliberation environment in good order, but they take away moderation resources from facilitation tasks that help elicit quality discussions and useful input for policymakers. And while achieving high quality public input is what policymakers ultimately need, it cannot be achieved without a well-organized discussion (Quick & Sandfort, 2014). Although some of the specific practices we observed in this study could be unique to RegulationRoom moderators, we argue that the tension we identify may exist in other online policy deliberation environments.
Based on our findings, we further argue that focusing on managing the tension between the immediate goals of online policy deliberations might overshadow a broader long-term goal: helping individuals develop participatory literacy beyond a single policy engagement. When facilitating the discussion through guiding participants to post their comments in the relevant policy section, asking them for clarifications, or requesting additional information, the moderators indeed achieve the immediate goal of soliciting high quality comments. It is also plausible that in the process, participants develop a set of participatory skills that help them craft more effective comments. Yet, it is unclear if the participants gain an understanding of why particular elements of the comment are needed or why a particular form of argumentation is expected in policy deliberation. Such understanding is necessary for the goal of helping citizens acquire participatory literacy and become civic-minded citizens who can fully participate in policy deliberations beyond a single engagement.
One hurdle in the way of achieving this long-term goal is the ad-hoc nature of policy deliberation events. As Nabatchi (2014) suggests, policy proposals are typically bound in time and scope, and are primarily information eliciting endeavors. For example, the Department of Transportation may be interested in what truckers have to say about a new policy proposal to install electronic onboard recorders in trucks. Within broad, long-term political deliberations (e.g., highway safety), a single instance of policy deliberation will often addresses one specific aspect (e.g., electronic on-board recorders in commercial trucks), and will last a limited period of time. Policy deliberations target specific stakeholders who may be affected by the proposed policy (e.g., truckers and truck company owners), and those need to get up to speed with their participation skills quickly, even if this is going to be their first and last time engaging in this kind of consultation. Given this ad-hoc nature of specific policy deliberations, the task of building a community of well-informed citizens seems less relevant, compared to the task of increasing comment quality through moderation.
In the long run, however, fostering an independent community of civic-minded and participatory-literate individuals might be necessary to sustain citizen participation in policy deliberations. In the next section we offer implications for design of online environments that may support such an ambitious endeavor.

Implications for Online Deliberation Environments
Automating Maintenance As discussed above, maintenance tasks keep the online deliberation environment organized, but when carried out by human moderators, may pull resources away from facilitation and community-building activities. Some maintenance tasksdetecting and addressing incivility, checking topic compatibility and relevance, and prioritizing comment based on urgency of responseare mostly invisible to participants (Trénel, 2009;Wright, 2009), and are therefore clear candidates for automation. This may be especially important when scaling up the online deliberation environment. With its current setup that requires manual attention to each comment, the RegulationRoom process can handle no more than one or two consultations simultaneously, whereas the federal government alone produces around four thousand rulemakings a year.
Automation could include, for example, analyzing comments using machine learning techniques to detect arguments that lack substantiating support (Park & Cardie, 2014), and flag those to moderators to check in with the commenter and intervene. Other tools could identify excessive repetitions or duplicates in one's repeating comments and as such detect "serial commenters" (similar to Shulman, 2009).
Another interesting avenue to explore combines crowdsourcing for some maintenance moderation tasks. Maintenance tasks could be broken into small assignments such as identifying the match between a comment and the policy section it is posted on, and marking those that need moderator attention. Such peripheral activity could reduce the efforts of moderators toward menial tasks. Further, when combined with appropriate interfaces and tools for both the crowd participants and moderators, crowd moderation does not necessarily have to be focused on assembly-line piecework (Kittur et al., 2013). Instead, this could provide participants with additional ways to participate in the discussion and understand what high quality policy deliberation looks like, and encourage positive norms of participation (Lampe, Zube, Lee, Park, & Johnston, 2014).
Not all maintenance tasks should be automated or crowdsourced. For example, similar to efforts in Wikipedia to retain new editors (Morgan, Bouterse, Stierch, & Walls, 2013), automation could be used to identify and flag newcomers, but it is important that human moderators engage in the act of personally welcoming them (Choi, Alexander, Kraut, & Levine, 2010). Care must be taken in the interplay between automation and human activities to foster a robust environment that runs smoothly at scale (Geiger & Halfaker, 2013).

Building a Civic-Minded Community
As we identified above, one path toward sustainability of an online platform for effective public engagement in policy deliberation may be supporting a community of civic-minded individuals. These individuals would have the necessary participatory literacy and share the ethos of public engagement in policymaking as a fundamental component of democracy. Similar to Wikipedians, who care about the overall quality of Wikipedia regardless of the specific topic of an individual entry (Bryant, Forte, & Bruckman, 2005;Panciera, Halfaker, & Terveen, 2009), these could be individuals who care about the quality of public participation in policymaking regardless of the specific topic of an individual policy consultation. Building such community is particularly challenging under the constraints of formal policymaking processes such as rulemaking, the ad-hoc nature of these processes, and the current culture of online civic engagement that focuses on single policy feedback solicitation.
According to the RegulationRoom moderator protocol, moderators serve a pivotal role in "the formation of a community that supports commenters' access to, participation in, and learning about the policymaking process." Freeing moderators from some maintenance tasks through automation, is necessary, but not sufficient, to help them fulfill this role.
One potential course of action is engaging effective returning participants in community-building activities. In RegulationRoom, currently these returning commenters are left to their own devices because their high quality comments do not require a moderator's intervention. However, these participants could potentially be further engaged, for example, by gradually promoting them to community leadership positions in which they care not only about the quality of their specific comments, but also about the online community as a whole (Preece & Shneiderman, 2009). These participants could be provided with tools for welcoming newcomers and helping detect comments that require attention from moderators. Moderators should play an important role in identifying these individuals, building relationships with them, mentoring them, and over time providing such leaders with more community responsibilities.
The task of growing bottom-up leaders from within the community may be difficult when interfacing with bureaucratic policymaking institutions. Such institutions assume a certain level of participatory expertise, which may not necessarily be present among novice participants (Fischer, 2004). At the same time, decision-makers expect the deliberation platform and its leaders to be neutral with respect to the proposed policy, allowing all voices from multiple sides to be heard (Kerwin, 2003). This means that moderators should be especially attentive in identifying, connecting with, and training potential community leaders. In addition to nurturing participatory expertise, moderators would need to monitor and deal with potential biases to avoid discouraging or excluding other participants, These community-building efforts are likely make moderators' job more demanding, but the process as a whole could be personally fulfilling to commenters and moderators, with more meaningful ways to participate. Creating new ways for participants to engage, and offering bottom-up efforts to increase trust in the policymaking deliberation process may be overall beneficial in nurturing a civic-minded community, expanding the purpose of online deliberation platforms from soliciting quality policy input to civic education and expanding legitimacy of deliberation as a form of democratic participation.

Limitations
As an experimental program run out of academia, RegulationRoom has a unique set of features that may explain some of our findings. First, having students as moderators, means that they are involved for a limited period of time (typically 1-2 semesters), and acquiring facilitation skills is only one of their educational goals in the clinic (others include learning to prepare materials, outreach activities, summarizing the discussion, etc.). The students' formal training, class discussions, faculty guidance, and the protocol, are all designed to support beginner facilitators, and despite ongoing support and supervision, the learning curve is still steep. Second, in preparing for and running consultations, there is often misalignment between the university academic year schedule and the agencies' schedules for public consultations (Jackson, Ribes, Buyuktur, & Bowker, 2011). As a result, students often begin moderating with compressed training programs, and need to learn how to moderate on the go. Other organizations running policy deliberations may not face these constraints. They may benefit from long-term professional facilitators, who are also engaged in the community beyond a single consultation, and they may be able to better align their schedules with those of the agencies they are working with.

Conclusion
Arguably, effective online civic engagement in policymaking is still in its infancy.
Attempts to have large-scale citizen participation in online policy deliberation have yielded mixed, but mostly unsatisfactory, results. Moving beyond merely providing access to policy deliberation (as in regulations.gov), the current challenge is addressing the participatory literacy barrier. In unpacking the practices of moderators in RegulationRoom, we demonstrate an inherent tension between the short-term and the long-term goals of any online civic engagement toward addressing this challenge.
In the short term, the goal is to provide decision-makers with the best and the most insightful comments from the members of the public. Focusing on that goal, RegulationRoom moderators take on the entire burden of bridging the participatory literacy gap starting with the preparation of materials, through live moderation of ongoing discussions. The moderators developed heuristics that in fact reflect different levels of participatory literacy and they use those heuristics to craft responses to improve the overall quality of the comments in the moment. Based on the feedback RegulationRoom received from policymakers, this approach works. It is a costly endeavor, but RegulationRoom-facilitated engagement yields information that is valuable, but otherwise inaccessible to the policymakers. This is particularly interesting given that 60-95% of RegulationRoom participants are new to the federal rulemaking process.
In the long term, the goal is to nurture a more civically-minded community that would help bridge the participatory literacy gap and expand effective public engagement beyond a one-time encounter around a single issue. This study suggests that some practices of RegulationRoom moderators may indeed contribute to such long-term goal. The writing style the moderators use and highlighting recommended comments both act as practice proxies for desired participation norms (Mugar, Østerlund, Hassman, Crowston, & Jackson, 2014). Yet, most moderation efforts are aimed towards the short-term goal, leaving the moderators with fewer resources to nurture relationships with individual participants and foster a sense of community.
As a long-term goal, a community that can take on the responsibilities of participatory literacy education is perhaps the most promising route toward sustainable effective online civic engagement. As RegulationRoom demonstrates, effective online civic engagement in policymaking is a resource-intensive endeavor, particularly when it involves human facilitative moderation. Sustaining such enterprise could be achieved through institutionalization within policymaking bodies, commercialization, or bottom-up growth à la Wikipedia. In any case, it would be beneficial to the moderators to be able to dedicate resources to community-building activities.
Ultimately, the administrative policymaking process should be revised to accommodate participation by inexperienced citizens. For example, it should allow more time and enable (e.g. through the use of plain language, presentation of materials, etc.) learning of both the substance of the consultation and modes of effective participation. Until that happens, designers and managers of online civic participation processes and platforms will act as mediators between the public and the policymakers. When doing so, they will inevitable have to balance the need for immediate results and long-term nurturing of a civic society.