The Effects of Peer-Controlled or Moderated Online Collaboration on Group Problem Solving and Related Attitudes

Abstract. This study investigated the relative benefits of peer-controlled and moderated online collaboration during group problem solving. Thirty-five self-selected groups of four or five students were randomly assigned to the two conditions, which used the same online collaborative tool to solve twelve problem scenarios in an undergraduate statistics course. A score for the correctness of the solutions and a reasoning score were analyzed. A survey was administered to reveal differences in students' related attitudes. Three conclusions were reached: This study investigated the relative benefits of peer-controlled and moderated online collaboration during group problem solving. Thirty-five self-selected groups of four or five students were randomly assigned to the two conditions, which used the same online collaborative tool to solve twelve problem scenarios in an undergraduate statistics course. A score for the correctness of the solutions and a reasoning score were analyzed. A survey was administered to reveal differences in students' related attitudes. Three conclusions were reached: • Groups assigned to moderated forums displayed significantly higher reasoning scores than those in the peer-controlled condition, but the moderation did not affect correctness of solutions. • Students in the moderated forums reported being more likely to choose to use an optional online forum for future collaborations. • Students who reported having no difficulty during collaboration reported being more likely to choose to use an optional online forum in the future. Résumé. Cette étude a examiné les avantages relatifs d'une collaboration en ligne contrôlée par des pairs et animée, dans le cadre d'une résolution de problèmes en groupe. Trente-cinq groupes autosélectionnés de quatre à cinq étudiants chacun ont été affectés au hasard aux deux types de coopération, qui utilisaient le même outil de collaboration en ligne pour résoudre douze scénarios de problèmes dans un cours de statistiques de premier cycle. Les notes relatives à l'exactitude des solutions et au raisonnement ont été analysées. Un sondage a été mené pour révéler les différences d'attitudes des étudiants pour les deux types de collaboration. Nous en avons tiré trois conclusions: • Chez les groupes affectés au forum animé, les notes relatives au raisonnement étaient plus élevées que celles du groupe contrôlé par des pairs, mais l'animation n'a pas affecté l'exactitude des solutions. • Les étudiants affectés aux forums animés ont indiqué qu'ils étaient plus susceptibles de choisir un forum en ligne facultatif pour des collaborations futures. • Les étudiants ayant indiqué qu'ils n'avaient éprouvé aucune difficulté pendant la coopération, ont également indiqué qu'ils étaient plus susceptibles d'utiliser un forum en ligne facultatif à l'avenir.


Introduction
proposed that social interactions play a very important role in learning, and recent research demonstrates that working in small groups can facilitate group learning (Bruffee, 1999;Hamm & Adams, 1992;Johnson & Johnson, 1975. As a result, collaborative learning has become increasingly popular in both face-to-face and distance education settings. Computer technologies, as demonstrated in research and practice, can support collaborative learning and be beneficial to learners in different ways (Bonk & Cunningham, 1998;Harasim, 1990;Scardamalia & Bereiter, 1994). For example, computer technologies make collaborative learning achievable even when face-to-face communications are difficult or impossible, and they can be employed to support reflection and improve the quality of the learning-related conversation (Lin, Hmelo, Kinzer, & Secules, 1999;Schwartz, Lin, Brophy, & Bransford, 1999). This study investigated the relative effectiveness of two approaches to the management and execution of online forums during group problem solving.

Theoretical Rationale for the Study
One of Vygotsky's most influential contributions was the analysis of human learning from a social-cultural history perspective (Gredler, 1997). Vygotsky (1978) proposed that learning is a developmental process occurring in social activities, and that higher-order, complex thinking is gradually developed through social interactions with others in the culture (Driscoll, 1994). As Vygotsky and other socio-cultural theorists point out, people learn from mediations and scaffoldings, which are offered within one's "zone of proximal development" (ZPD) from more capable peers or experts (Bonk & Cunningham, 1998;Gredler, 1997;Wertsch, 1985). Vygotsky defines the ZPD, one of the key concepts in socio-cultural theory, as the distance between one's independent competency and the competency obtained with expert assistance or in collaboration with peers (Wertsch, 1985). In collaborative learning environments, a learner's ZPD can be reached and extended with scaffolds through different agents, such as instructors, peer learners and supporting materials (Bonk & Cunningham, 1998;Gredler, 1997;Wertsch, 1985). Hannafin, Land, and Oliver (1999) classify scaffolding into four types: conceptual, metacognitive, procedural and strategic. Conceptual scaffolds guide students in what to consider; metacognitive scaffolds are about how to think; procedural scaffolds guide students how in to utilize the available features for learning purposes; and strategic scaffolds provide macro-strategy as needed (Hannafin et al, 1999). Prompts, guidelines, and questions can also serve as scaffolds (Bonk & Cunningham, 1998).
Some studies indicate that a facilitator or moderator can smooth the process of collaboration (Hooper, 1992a(Hooper, , 1992bJohnson, Johnson, & Stanne, 1986;Ruberg, Moore, & Taylor, 1996). Research also shows that successful group learning promotes higher order thinking (Blumenfeld, Marx, Soloway, & Krajcik, 1996). As educators and researchers stress the importance of problem solving (e.g., Bransford, Brown, & Cocking, 2000;Bransford, Sherwood, & Sturdevant, 1987;Bransford & Stein, 1993;Jonassen, 1997;Schmidt, 1989), problem-based learning (PBL) has been increasingly practiced in a variety of fields, such as medicine, law, engineering, social work, and agriculture (Barrows & Tamblyn, 1980;Boud, 1985;Bouhuijs, Schmidt, & Berkel, 1993). It is often emphasized that the instructor plays a critical role in the problem-based learning process, as a moderator or facilitator to foster active and effective learning (e.g., Barrows & Tamblyn, 1980;Barrows, 1985Barrows, , 1996Coles, 1991). As researchers report the various challenges faced by learners in online PBL (e.g., Lohman & Finkelstein, 2000;Orrill, 2000;Sage, 2000) and online collaborative learning (e.g., Zhang & Harkness, 2002), a skillful moderator may help address some of the issues and challenges in the group learning process. Research suggests, therefore, that if learning is largely a social activity, if learning in groups generally improves student learning, if successful group work can improve higher-order thinking, and if a facilitator/moderator can improve collaboration, then facilitation of online collaboration might be expected to improve performance in a group problem-solving task.

Research Questions
This study set out to answer four questions: • Are two types of collaboration, peer-controlled and externally structured and moderated, equally effective in terms of determining correct answers and developing reasoning to support the chosen solution?
• Do the two types of collaboration lead to different amounts of time spent in collaboration or the use of different media for collaboration (e.g. email, telephone, face-to-face, etc.)?
• Does the type of moderation influence students' perceptions of the difficulty or value of online collaboration during group problem solving?
• Do student perceptions of the difficulty or the value of online collaboration influence the probability that they would use an optional online collaboration tool in the future?

Structuring and Moderating Online Forums
Some studies report that when participants have more active control over the learning process, learning increases (Jensen, 1996). Therefore less instructor participation may foster more active learning. Another study (Aviv & Golan, 1998), however, indicates that students' participation in electronic discussion is often minimal without an instructor's participation. Researchers (Aviv & Golan, 1998) report that in electronic discussions most students participate passively and only respond to a few messages, and very few students ask questions.
They also propose that electronic discussions can result in a successful learning experience when they are planned, focused, and guided throughout the process.
Research indicates that in online learning environments, moderation and structuring of the process can be important contributors to a successful learning experience (Flannery, 1994). When changed from the "center of authority" to "co-conspirator," instructors assume substantial responsibilities to cultivate and promote a learner-centered collaborative learning environment. "Free riding" and "social loafing" in teamwork have negative impacts on group dynamics and team performance.
However, little research has formally investigated alternative methods of structuring peer online collaboration.
In practice, peer collaborations are often structured by learners when they split the group task into smaller pieces and each take charge of part of the task (Althauser & Matuga, 1998). In order to reduce "free riding" in group work, instructors then often try to structure the online peer collaborations by assigning a set of questions to each member of a group. In such cases, every group member must participate in the part of the task to which they are assigned. Such structuring, however, tends to lead students to work on the group assignment in a cooperative way instead of collaboratively -individuals can complete their own part of the assignment without participation or contribution from peers, and they may, but usually do not contribute to parts of the assignment directed toward others. Bosworth and Hamilton (1994) suggest that instructors direct both the structure of the teamwork and collaboration process to improve collaborative learning. Based on this advice, two strategies are employed to structure the online collaboration in the treatment group in this study: (a) Two moderators, through joint efforts, structured the group work by requesting each member's original contributions and meaningful, constructive responses to contributions of others; and (b) they also jointly moderated the collaboration process.
Based on a review of the literature on moderating online collaboration (e.g., Flannery, 1994;Hamm & Adams, 1992;Harasim, 1990), the moderators in this study, in addition to the structuring efforts, were asked to: (a) help the teams keep the discussion on the topic; (b) make sure everyone knew the purpose of the forum and used it appropriately; (c) check group progress and remind users of the timetable; and (d) encourage and support individuals for equal participation. The moderators provided the four types of scaffoldings, as classified by Hannafin et al (1999), as needed. The two moderators worked together, deciding when and how to respond.

Method
The study was a randomized, post-test only, control group design. The independent variable manipulated was the method of peer online collaboration (peer-controlled or structured and moderated). Thirty-five self-selected groups of four or five peers were randomly assigned to either the control (peer-controlled) or the treatment (structured and moderated). To further promote collaboration outside the private groups, a public forum was set up for each condition so that students could seek assistance on issues and problems that were challenging for their private team. The dependent variables were the group achievement score on the problem solving assignment (the answer score for the correctness of solutions), the quality of the argumentation provided with the solutions, which was measured by a "reasoning score", and students' perceived difficulty of collaboration, perceived value of collaboration, probability of future use of an online collaborative forum, and time spent using each collaboration medium. An open-ended survey (Appendix A) was conducted upon completion of the assignment to obtain the data for the last four variables.

Participants
The study was implemented in a large undergraduate introductory statistics course at a major university in the northeastern United States. This course, required by approximately sixty academic programs in eleven colleges and schools at the university, contained students from all classes (freshman to senior) and included students with varied academic backgrounds. One hundred and forty eight students volunteered to participate in this study. The participants had experienced other tasks requiring teamwork in this course prior to this study.
Before the study began, the instructor asked the students to re-organize their existing groups into groups of four or five students, choosing their own team members. These self-selected groups were then randomly assigned to either the peer controlled online collaboration or structured and moderated online collaboration condition.
Approximately five weeks before the study started, students were given hands-on instruction on the use of the online forums. Detailed job-aids on the use of the forums were available in printed form and online. Students were given opportunities in the computer lab sessions to practice accessing the forums and posting and replying to messages on the private and the public forums. Two weeks after the initial introduction of the forums, students were encouraged to use the online forum as an option for team collaboration and communications with the instructor on a group project. Some groups started using the private forum to communicate with their teammates then, even though it was not required.

Materials and Procedures
Because the academic content of this study consisted of the topics of inference and regression, which were considered the most difficult topics in this course, students were offered a rich context about which to engage in peer collaboration. The problems used in this study were twelve scenarios in which one statistical technique can be used most appropriately to address the problem or answer the question raised in the scenario. Because the cases left students a great deal of freedom in determining the most appropriate methods to solve the problem, they warranted team collaboration to reach consensus. As an example, one of the problem scenarios is provided below: "A politician asserted that only 40% of the American public supported the use of ground troops in Kosovo.
Secretary of Defense Cohen disputed this, saying it was certainly more than 40%. A CNN poll of 850 people showed that 442, or 52%, did favor the use of ground troops. Who do you think is right-the politician or Cohen?" Each scenario was presented as a two-part assignment. First, students were to recommend an appropriate statistical technique to address the problem, then they were to briefly report their reasoning for recommending that technique. The correctness of the recommendation of a statistical technique was scored and reported as the "answer score" in this study. In order to align the score produced with the grading system of this course, three points were awarded for a correct answer to each of the 12 problem scenarios. No partial credit was given when an inappropriate statistical technique was recommended, so the answer score could range from 0 to 36. The reasoning score was established by assessing the quality of the explanation for the selection of each statistical technique they recommended. Three points were also possible for the reasoning score for each scenario, but partial credit could be assigned according to the match between the response and the categories identified on a scoring rubric (See Appendix B). Like the answer score, the reasoning score could range from 0 to 36.
When the problem solving assignment was completed, an open-ended, individual survey (Appendix A) was conducted to gather additional data on perceived collaboration difficulty, perceived value of collaboration and probability of future use of a collaborative forum.

Assessment Reliability
All group assignments were blind reviewed and scored for correctness by two raters. The correlation of the two rater's answer scores was 1.000 (the answer scores were identical for both raters). The raters also worked independently to develop reasoning scores. The correlation of the two rater's independent assessments of reasoning scores was 0.990. In their ratings of the twelve problem scenarios for the thirty-five groups (420 scores in total), the raters disagreed on five, differing by only one point on four items and by two points on one.
The two raters then worked together, jointly re-scoring these five items, and the negotiated scores were reported and analyzed in this study.
As the odd and even halves of the test were considered parallel and the assessment was not timed, an odd-even split-halves procedure was executed to examine the reliability of the assessment (Nitko, 1996). For both answer scores and reasoning scores, the even and odd halves of the assessment were significantly correlated (p values were 0.007 and 0.000 respectively).
Data collected through the three open-ended questions in the survey were classified by the lead researcher into the following four categories: "yes"; "no"; "maybe"; and "maybe not". "Yes" or "no" responses to these open-ended questions were reported as they were. Responses like "very likely", "probably", "maybe", "perhaps", and "possibly" were reported as "maybe", and responses like "maybe not", "not very likely", "probably not" were reported as "maybe not."

Treatments
The groups in this study were from three different sections of the same course with the same instructor.
Because it was possible that the three sections were different in terms of students' academic performance, motivation, and other important factors, computer software was used to perform a block randomization so that each treatment was randomly assigned with the same number of groups from each section.
Peer controlled treatment. Each group in this treatment was required to collaborate in a private, peer-controlled online forum, to which only members of the group had access. No moderator was assigned. A public forum was also made available to all members of all groups in this treatment. Also a peer-controlled public forum, this tool was made available so that students could choose to publish problems and issues they were not able to solve within a group to the public forum, to reach a broader audience. No moderation, scaffolding, or structuring efforts were implemented in any of the forums available to these groups.
Structured and moderated treatment. Each group in this treatment was provided a private structured and moderated online forum, to which only group members had access. In addition, a structured and moderated public forum was made available to all of the subjects in all of the groups receiving this treatment, so that students could choose to publish problems and issues they were not able to solve within a group to the public forum ,in orderto reach a broader audience.
For each group assigned to this condition, collaborations in both the private and public forums were structured and moderated with scaffolding, as needed, by two moderators, the instructor and a graduate assistant, to promote learning and thinking. Before the study was implemented, both moderators were trained on how to support peer online collaborations through structuring, moderating and scaffolding efforts. The lead researcher introduced the moderators to relevant literature and provided examples of structuring, moderating and scaffolding efforts in similar online collaborative learning environments. Using online collaboration transcripts from other courses as examples, the lead researcher and the moderators worked together to identify possible interventions for this particular course. A worksheet was provided to the moderators on which to record the interventions, media used, and group reactions, as applicable.
Two strategies were employed to structure the online collaboration for groups assigned to this condition: (a) Two moderators, through joint efforts, structured the group work by requesting each member's original contributions and meaningful, constructive responses to contributions of others; and (b) they also jointly moderated the collaboration process. The moderators were also asked to: (a) help the teams keep the discussion on the topic; (b) make sure everyone knew the purpose of the forum and could use it appropriately; (c) check group progress and remind users of the timetable; and (d) encourage and support individuals for equal participation. In addition, the moderators provided the four types of scaffolding, as classified by Hannafin et al (1999), as needed. Throughout the course of online collaboration, the two moderators worked together and provided all the structuring, moderating and scaffolding through online forums and private email.
The moderators never gave any direct answers to specific statistics-related questions. Most moderating and structuring efforts were delivered through online forum messages, with some provided through private emails to individuals as appropriate. All messages from the moderators were written in conversational style in order to foster student participation (Ahern, Peck, & Laycock, 1992). With scaffolding and moderating efforts, the moderators assisted with non-statistics issues such as team dynamics and team processing , and helped with the problem-solving task only indirectly.
The same two moderators jointly served all the treatment groups, behaving in accordance with the recommendations, which were described earlier. Examples of the types of moderation included: Conceptual moderation (22 times, all through online forums). The moderators provided clarification on the usage of the public and private forums when students seemed to be uncertain of the difference between the two, and encouraged participation on both types of forums as appropriate.
Moderators posted conversational prompts such as: " Seems that you have a little trouble with #6? What did you do in the last project? Is there anything you learned there that might be helpful?" When a group seemed unclear about where to start with the problems, the moderators posted the following message: "You may want to start by asking yourself: What type of data is sought in the scenario? How many samples are there? Is there a comparison? Also, think about what we talked about before, and what you did in previous projects. Is there anything that might be helpful? By working with these small questions, I believe you'll soon be able to figure it out. Good luck!" Procedural moderation (17 times, all through online forums). The moderators reminded the groups of the built-in calculator and other simulation functions in the course website.
Strategic moderation (266 times, through a combination of the online forums and private emails). Two types of strategic scaffolds were employed: one was related to strategic team work, the other was related to solving the problems themselves, as stated in the opening posting in every group private forum. The moderators helped the groups to think about group work strategy as well as online collaboration strategy. For example, when a group was planning on a real-time collaboration through the online forum, the moderators suggested that they set an agenda first, and discuss how they would like to proceed in detail before the real-time online conference. Table 1 below shows the number of groups and students in each treatment group. Table 1.

Problem Solving Results
The answer scores and reasoning scores in the peer-controlled online peer collaboration groups ranged respectively from 12 to 36, and from 24 to 36; and those scores in the structured and moderated peer online collaboration forums ranged from 24 to 36, and 32 to 36 respectively. Means and standard deviations are shown in Table 2 below. Table 2.

Performance Data Achieved by Treatment Group
Treatment and correctness of solutions.As indicated in Table 3 below, there is no statistically significant difference (p = .096) between the means for "answer score" exhibited by the peer-controlled online collaboration groups and the structured and moderated peer online collaboration groups. Table 3.

One-Way Analysis of Variance for Answer Score
Treatment and reasoning during problem solving .As shown in Table 4 below, there is a statistically significant difference (p = 0.032) between the means of reasoning scores exhibited by the peer-controlled groups and the structured and moderated groups. The structured and moderated forums appear to be more effective in developing reasoning during group problem solving tasks. Table 4.

Survey Results
When students had completed all of the group assignments, they were asked to fill out a survey (see Appendix A), individually. Their perceptions about the media used for team collaboration on this assignment, the time spent on the task with each method of collaboration, and preference for future use of an online forum for peer collaboration were investigated.
Treatment and time spent on communication.The data related to the amount of time spent in different types of interaction are exhibited in Table 5. None of the differences in time spent by the two treatment groups were statistically significant. Although the groups in the structured and moderated forums reported making significantly more phone calls, the total time spent on the phone was not significantly different between the two treatments. Table 5.

Survey Data on Time Spent in Collaboration
Treatment and the perceived difficulty and value of collaboration .The survey data also probed students' perceptions regarding the effort required and the value of teamwork, peer collaboration, and the online forums, as illustrated in Table 6. Table 6.

Survey Data Regarding Teamwork and Online Forum
More than 90% of students in both the peer-controlled and the structured and moderated conditions found that online peer collaboration was valuable, and approximately 3/4 of the students in both groups reported that it was not difficult. The self-reported differences were not statistically significantly different for the two treatment groups, neither for perceived difficulty (F = 1.36; MSE = 0.193; p = 0.246) nor for the perceived value of collaboration (F = 0.00; MSE = 0.071; p = 0.976). In other words, the different approaches to online collaboration did not lead to differences in individual learner's perceptions of the difficulty of group work or the perceived value of collaboration.
Treatment and probability of future use.The data in Table 7 below represent a significant relationship between the type of forum used in this study and students' attitudes toward the use of optional online forums in the future, with students from the moderated treatment significantly more likely to report the desire to use online collaboration in the future. Table 7.

Online Collaboration Approach And Preference For Future Use Of Online Forums
Difficulty of group collaboration and future preference for online forums.The data displayed as Table 8 show a significant and perhaps predictable relationship between the perceived difficulty of collaboration and students' attitudes toward use of online forums in the future. Students who found such communication difficult reported being less likely to use such systems in the future. Table 8.

Collaboration Difficulty and Preference for Future Use of Online Forums
Perceived value of online collaboration and future preference for online forums. Table 9 shows students' perceptions of the value of collaboration and students' possible use of online forum in the future. Although the students' perceptions of the benefits of peer collaboration do not appear to significantly affect their future preference for online collaboration forum usage, it seems worthwhile to note that almost 90% of the students reported that they found peer collaboration valuable, leaving very few subjects to distribute across cells, which reduced statistical power. Table 9 Value of Collaboration and Preference for Future Use of Online Forums

Discussion
This study provides evidence that structured and moderated online peer collaboration during group problem solving may result in superior reasoning and better attitudes toward collaboration, without significantly increasing the time students must invest in the learning activity.
This finding may actually underestimate the importance of moderation, because the differences between the peer-controlled and moderated conditions also favored the structured and moderated condition and approached statistical significance for quality of solution (p = 0.096). The ceiling effects may also have limited the power of the treatment. Perhaps studies looking at different content, employing different problem types (ill-structured problems, for example), using subjects of different ages or academic abilities, or even using more than 18 groups per condition would identify differences in the quality of the solution that are statistically significant.
Another area worthy of further investigation is the effect of peer online collaboration on individual achievement and the relationships between learner characteristics and the type of online collaboration on performance during problem solving.
These findings are consistent with socio-cultural learning theories and illustrate the critical importance of the instructor's changed role in online collaborative learning. Researchers and scholars believe that collaborative learning is a dynamic process, in which learners are more actively involved (Flannery, 1994;Hamm & Adams, 1992), and learners' active control over the learning activities is reported to enhance learning (Jensen, 1996).
The moderator's role, as defined in this study, can enhance the value of the online collaborative learning experience.