Skip to main content
SpringerLink
Log in

A rating scheme for assessing the quality of computer-supported collaboration processes

  • Published:
International Journal of Computer-Supported Collaborative Learning Aims and scope Submit manuscript

Abstract

The analysis of the process of collaboration is a central topic in current CSCL research. However, defining process characteristics relevant for collaboration quality and developing instruments capable of assessing these characteristics are no trivial tasks. In the assessment method presented in this paper, nine qualitatively defined dimensions of collaboration are rated quantitatively: sustaining mutual understanding, dialogue management, information pooling, reaching consensus, task division, time management, technical coordination, reciprocal interaction, and individual task orientation. The data basis for the development of these dimensions was taken from a study in which students of psychology and medicine collaborated on a complex patient case via a desktop-videoconferencing system. A qualitative content analysis was performed on a sample of transcribed collaboration dialogue. The insights from this analysis were then integrated with theoretical considerations about the roles of communication, joint information processing, coordination, interpersonal relationship, and motivation in the collaboration process. The resulting rating scheme was applied to process data from a new sample of 40 collaborating dyads. Based on positive findings on inter-rater reliability, consistency, and validity from this evaluation, we argue that the new method can be recommended for use in different areas of CSCL.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  • Anderson, A. H., O’Malley, C., Doherty-Sneddon, G., Langton, S., Newlands, A., Mullin, J., et al. (1997). The impact of VMC on collaborative problem solving: An analysis of task performance, communicative process, and user satisfaction. In K. E. Finn, A. J. Sellen, & S. B. Wilbur (Eds.), Video-mediated communication (pp. 133–156). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Angiolillo, J. S., Blanchard, H. E., Israelski, E. W., & Mané, A. (1997). Technology constraints of video-mediated communication. In K. E. Finn, A. J. Sellen, & S. B. Wilbur (Eds.), Video-mediated communication (pp. 51–74). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Barron, B. (2000). Achieving coordination in collaborative problem-solving groups. Journal of the Learning Sciences, 9, 403–436.

    Article  Google Scholar 

  • Carell, A., Herrman, T., Kienle, A., & Menold, N. (2005). Improving the coordination of collaborative learning with process models. In T. Koschmann, D. Suthers, & T. W. Chan (Eds.), Proceedings of the CSCL 2005 (pp. 18–27). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Clark, H. H. (1996). Using language. Cambridge MA: Cambridge University Press.

    Google Scholar 

  • Clark, H. H., & Brennan, S. E. (1991). Grounding in communication. In L. B. Resnick, J. M. Levine, & S. D. Teasley (Eds.), Perspectives on socially shared cognition (pp. 127–148). Washington, DC: American Psychological Association.

    Chapter  Google Scholar 

  • Clark, D., & Sampson, V.D. (2005). Analyzing the quality of argumentation supported by personally-seeded discussions. In T. Koschmann, D. Suthers, & T. W. Chan (Eds.), Proceedings of the CSCL 2005 (pp. 76–85). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • De Wever, B., Schellens, T., Valcke, M., & Van Keer, H. (2006). Content analysis schemes to analyze transcripts of online asynchronous discussion groups: A review. Computers and Education, 46, 6–28.

    Article  Google Scholar 

  • Deutsch, M. (2003). Cooperation and conflict. A personal perspective on the history of the social psychological study of conflict resolution. In M. A. West, D. Tjosvold, & K. G. Smith (Eds.), International handbook of organizational teamwork and cooperative working (pp. 9–43). Chichester, UK: Wiley.

    Google Scholar 

  • Dillenbourg, P. (1999). Introduction: What do you mean by “collaborative learning”? In P. Dillenbourg (Ed.), Collaborative learning. Cognitive and computational approaches (pp. 1–19). Amsterdam: Pergamon.

    Google Scholar 

  • Dillenbourg, P., Baker, M., Blaye, A., & O’Malley, C. (1995). The evolution of research on collaborative learning. In P. Reimann & H. Spada (Eds.), Learning in humans and machines: Towards an interdisciplinary learning science (pp. 189–211). Oxford: Pergamon.

    Google Scholar 

  • Dönmez, P., Rose, C., Stegmann, K., Weinberger, A., & Fischer, F. (2005). Supporting CSCL with automated corpus analysis technology. In T. Koschmann, D. Suthers, & T. W. Chan (Eds.), Proceedings of the CSCL 2005 (pp. 125–134). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Erkens, G., Jaspers, J., Prangsma, M., & Kanselaar, G. (2005). Coordination processes in computer supported collaborative writing. Computers in Human Behavior, 21, 463–486.

    Article  Google Scholar 

  • Fischer, F., & Mandl, H. (2003). Being there or being where? Videoconferencing and cooperative learning. In H. van Oostendorp (Ed.), Cognition in a digital world (pp. 205–223). Mahwah, NJ: Lawrence Erlbaum Associates.

  • Heckhausen, H. (1989). Motivation und Handeln. [Motivation and behavior]. Berlin Heidelberg New York: Springer.

    Google Scholar 

  • Hermann, F., Rummel, N., & Spada, H. (2001). Solving the case together: The challenge of net-based interdisciplinary collaboration. In P. Dillenbourg, A. Eurelings, & K. Hakkarainen (Eds.), Proceedings of the first European conference on computer-supported collaborative learning (pp. 293–300). Maastricht: McLuhan Institute.

    Google Scholar 

  • Hinsz, V. B., Tindale, R. S., & Vollrath, D. A. (1997). The emerging conceptualization of groups as information processors. Psychological Bulletin, 121(1), 43–64.

    Article  Google Scholar 

  • Janis, I. L. (1982). Groupthink. Boston: Houghton Mifflin.

    Google Scholar 

  • Järvelä, S., & Häkkinen, P. (2003). The levels of web-based discussions: Using perspective-taking theory as an analytical tool. In H. van Oostendorp (Ed.), Cognition in a digital world (pp. 77–95). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Johnson, D. W., & Johnson, R. T. (2003). Training for cooperative group work. In M. A. West, D. Tjosvold, & K. G. Smith (Eds.), International handbook of organizational teamwork and cooperative working (pp. 167–183). Chichester, UK: Wiley.

    Google Scholar 

  • Jucks, R., Bromme, R., & Runde, A. (2003). Audience Design von Experten in der netzgestützten Kommunikation: Die Rolle von Heuristiken über das geteilte Vorwissen. (Audience design of experts in net-based communication: The role of heuristics about shared knowledge). Zeitschrift für Psychologie, 211(2), 60–74.

    Article  Google Scholar 

  • Kapur, M., Voiklis, J., & Kinzer, C. K. (2005). Problem solving as a complex, evolutionary activity: A methodological framework for analyzing problem solving processes in a computer-supported collaborative environment. In T. Koschmann, D. Suthers, & T. W. Chan (Eds.), Proceedings of the CSCL 2005 (pp. 252–261). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Kerlinger, F. N., & Lee, H. B. (2000). Foundations of behavioral research. Fort Worth: Harcourt College.

    Google Scholar 

  • Kneser, C., & Ploetzner, R. (2001). Collaboration on the basis of complementary domain knowledge: Observed dialogue structures and their relation to learning success. Learning and Instruction, 11(1), 53–83.

    Article  Google Scholar 

  • Koschmann, T., Zemel, A., Conlee-Stevens, M., Young, N., Robbs, J., & Barnhart, A. (2003). Problematizing the problem. In B. Wasson, S. Ludvigsen, & U. Hoppe (Eds.), Designing for change (pp. 37–46). Dordrecht: Kluwer.

    Google Scholar 

  • Larson, J. R., & Christensen, C. (1993). Groups as problem-solving units: Toward a new meaning of social cognition. British Journal of Social Psychology, 32, 5–30.

    Google Scholar 

  • Lee, E. Y. C., Chan, C. K. K., & van Aalst, J. (2006). Students assessing their own collaborative knowledge building. International Journal of Computer-supported Collaborative Learning, 1, 277–307.

    Article  Google Scholar 

  • Malone, T. W., & Crowston, K. (1990). What is coordination theory and how can it help design cooperative work systems? Proceedings of the Conference on Computer-Supported Cooperative Work (pp. 357–370). Los Angeles, CA.

  • Malone, T. W., & Crowston, K. (1994). The interdisciplinary study of coordination. ACM Computing Surveys, 26(1), 87–119.

    Article  Google Scholar 

  • Mayring, P. (2003). Qualitative Inhaltsanalyse. Grundlagen und Techniken (Qualitative content analysis. Foundations and techniques). Weinheim: Beltz.

    Google Scholar 

  • Moreland, R. L., & Myaskovsky, L. (2000). Exploring the performance benefits of group training: Transactive memory or improved communication? Organizational Behavior and Human Decision Processes, 82(1), 117–133.

    Article  Google Scholar 

  • Nickerson, R. S. (1999). How we know—and sometimes misjudge—what others know: Imputing one’s own knowledge to others. Psychological Bulletin, 125(6), 737–759.

    Article  Google Scholar 

  • Nurmela, K., Palonen, T., Lehtinen, E., & Hakkarainen, K. (2003). Developing tools for analyzing CSCL process. In B. Wasson, S. Ludvigsen, & U. Hoppe (Eds.), Designing for change (pp. 333–342). Dordrecht: Kluwer.

    Google Scholar 

  • O’Conaill, B., & Whittaker, S. (1997). Characterizing, predicting, and measuring video-mediated communication: A conversational approach. In K. E. Finn, A. J. Sellen, & S. B. Wilbur (Eds.), Video-mediated communication (pp. 107–132). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Postmes, T., Spears, R., & Cihangir, S. (2001). Quality of decision making and group norms. Journal of Personality and Social Psychology, 80(6), 918–930.

    Article  Google Scholar 

  • Prins, F. J., Sluijsmans, D. M. A., Kirschner, P. A., & Strijbos, J. -W. (2005). Formative peer assessment in a CSCL environment: A case study. Assessment and Evaluation in Higher Education, 30(4), 417–444.

    Google Scholar 

  • Rummel, N., & Spada, H. (2005a). Instructional support for collaboration in desktop videoconferencing settings. How it can be achieved and assessed. In R. Bromme, F. W. Hesse, & H. Spada (Eds.), Barriers and biases in computer-mediated knowledge communication— and how they may be overcome (pp. 59–88). Berlin Heidelberg New York: Springer.

    Chapter  Google Scholar 

  • Rummel, N., & Spada, H. (2005b). Learning to collaborate: An instructional approach to promoting problem-solving in computer-mediated settings. Journal of the Learning Sciences, 14(2), 201–241.

    Article  Google Scholar 

  • Rummel, N., Spada, H., & Hauser, S. (2006). Learning to collaborate in a computer-mediated setting: Observing a model beats learning from being scripted. In: S. A. Barab, K. E. Hay, & D. T. Hickey (Eds.), Proceedings of the International Conference of the Learning Sciences 2006 (pp. 634–640). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Sacks, H., Schegloff, E., & Jefferson, G. (1974). A simplest systematic for the organization of turn-taking in conversation. Language, 50, 696–753.

    Article  Google Scholar 

  • Sosa y Fink, S. (2003). Merkmale gelungener Kooperation. Eine qualitative Analyse netzgestützter Zusammenarbeit. (Characteristics of successful cooperation. A qualitative analysis of net-based collaboration.) Diploma thesis, Albert-Ludwigs-Universität, Freiburg.

  • Spada, H., Meier, A., Rummel, N., & Hauser, S. (2005). A new method to assess the quality of collaborative process in CSCL. In T. Koschmann, D. Suthers, & Chan, T.W. (Eds.), Proceedings of the CSCL 2005 (pp. 622–631). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Stasser, G., Stewart, D., & Wittenbaum, G. (1995). Expert roles and information exchange during discussion: The importance of knowing who knows what. Journal of Experimental Social Psychology, 31, 244–265.

    Article  Google Scholar 

  • Stasser, G., & Titus, W. (1985). Pooling of unshared information in group decision making: Biased information sampling during group discussion. Journal of Personality and Social Psychology, 48, 1467–1478.

    Article  Google Scholar 

  • Steiner, I. D. (1972). Group process and productivity. New York: Academic.

    Google Scholar 

  • Tindale, R. S., Kameda, T., & Hinsz, V. B. (2003). Group decision making. In M. A. Hogg & J. Cooper (Eds.), Sage handbook of social psychology (pp. 381–403). London: Sage.

    Google Scholar 

  • Webb, N. M. (1989). Peer interaction and learning in small groups. International Journal of Education Research, 13, 21–39.

    Article  Google Scholar 

  • Wegner, D. M. (1987). Transactive memory: A contemporary analysis of the group mind. In B. Mullen & G. R. Goethals (Eds.), Theories of group behavior (pp. 185–208). Berlin Heidelberg New York: Springer.

    Google Scholar 

  • Weinberger, A., & Fischer, F. (2006). A framework to analyze argumentative knowledge construction in computer-supported collaborative learning. Computers and Education, 46, 71–95.

    Article  Google Scholar 

  • Whittaker, S., & O’Conaill, B. (1997). The role of vision in face-to-face and mediated communication. In K. E. Finn, A. J. Sellen, & S. B. Wilbur (Eds.), Video-mediated communication (pp. 23–50). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Wirtz, M., & Caspar, F. (2002). Beurteilerübereinstimmung und Beurteilerreliabilität. (Inter-rater agreement and inter-rater reliability). Göttingen: Verlag für Psychologie.

    Google Scholar 

  • Wittenbaum, G. M., Vaughan, S. I., & Stasser, G. (1998). Coordination in task performing groups. In R. S. Tindale et al. (Eds.), Theory and research on small groups (pp. 177–204). New York: Plenum.

    Google Scholar 

  • Zumbach, J., Schönemann, J., & Reimann, P. (2005). Analyzing and supporting cooperative computer-mediated communication. In T. Koschmann, D. Suthers, & Chan, T.W. (Eds.), Proceedings of the CSCL 2005 (pp. 758–767). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychology, University of Freiburg, Engelbergerstr. 41, 79085, Freiburg, Germany

    Anne Meier, Hans Spada & Nikol Rummel

Authors
  1. Anne Meier
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hans Spada
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nikol Rummel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anne Meier.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Meier, A., Spada, H. & Rummel, N. A rating scheme for assessing the quality of computer-supported collaboration processes. Computer Supported Learning 2, 63–86 (2007). https://doi.org/10.1007/s11412-006-9005-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11412-006-9005-x

Keywords

Search

Navigation