Angela E.B. Stewart
Hana Vrzakova
Valerie Shute
Skip Abstract Section
Abstract
Collaborative problem solving (CPS) is a crucial 21st century skill; however, current technologies fall short of effectively supporting CPS processes, especially for remote, computer-enabled interactions. In order to develop next-generation computer-supported collaborative systems that enhance CPS processes and outcomes by monitoring and responding to the unfolding collaboration, we investigate automated detection of three critical CPS process ? construction of shared knowledge, negotiation/coordination, and maintaining team function ? derived from a validated CPS framework. Our data consists of 32 triads who were tasked with collaboratively solving a challenging visual computer programming task for 20 minutes using commercial videoconferencing software. We used automatic speech recognition to generate transcripts of 11,163 utterances, which trained humans coded for evidence of the above three CPS processes using a set of behavioral indicators. We aimed to automate the trained human-raters' codes in a team-independent fashion (current study) in order to provide automatic real-time or offline feedback (future work). We used Random Forest classifiers trained on the words themselves (bag of n-grams) or with word categories (e.g., emotions, thinking styles, social constructs) from the Linguistic Inquiry Word Count (LIWC) tool. Despite imperfect automatic speech recognition, the n-gram models achieved AUROC (area under the receiver operating characteristic curve) scores of .85, .77, and .77 for construction of shared knowledge, negotiation/coordination, and maintaining team function, respectively; these reflect 70%, 54%, and 54% improvements over chance. The LIWC-category models achieved similar scores of .82, .74, and .73 (64%, 48%, and 46% improvement over chance). Further, the LIWC model-derived scores predicted CPS outcomes more similar to human codes, demonstrating predictive validity. We discuss embedding our models in collaborative interfaces for assessment and dynamic intervention aimed at improving CPS outcomes.
- Richard Alterman and Kendall Harsch. 2017. A more reflective form of joint problem solving. Int. J. Comput. Collab. Learn. 12, 1 (March 2017), 9--33. DOI:https://doi.org/10.1007/s11412-017--9250--1Google Scholar
Cross Ref
- Kathleen T Ashenfelter. 2007. Simultaneous analysis of verbal and nonverbal data during conversation: symmetry and turn-taking. University of Notre Dame.Google Scholar
- Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2014. Fitting linear mixed-effects models using lme4. arXiv Prepr. arXiv1406.5823 67, 1 (2014), 1--48. DOI:https://doi.org/10.18637/jss.v067.i01Google Scholar
- Steven Bird and Edward Loper. 2004. NLTK: The Natural Language Toolkit. In Proceedings of the Association for Computational Linguistics 2004 on Interactive Poster and Demonstration Sessions, 31-es. DOI:https://doi.org/10.3115/1219044.1219075Google ScholarDigital Library
- Emily A. Butler and Ashley K. Randall. 2013. Emotional Coregulation in Close Relationships. Emot. Rev. 5, 2 (April 2013), 202--210. DOI:https://doi.org/10.1177/1754073912451630Google ScholarCross Ref
- Dan Calacci, Oren Lederman, David Shrier, and Alex ?Sandy" Pentland. 2016. Breakout: An Open Measurement and Intervention Tool for Distributed Peer Learning Groups. CoRR abs/1607.0, (2016). Retrieved from http://arxiv.org/abs/1607.01443Google Scholar
- Prerna Chikersal, Maria Tomprou, Young Ji Kim, Anita Williams Woolley, and Laura Dabbish. 2017. Deep Structures of Collaboration: Physiological Correlates of Collective Intelligence and Group Satisfaction. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17), 873--888. DOI:https://doi.org/10.1145/2998181.2998250Google ScholarDigital Library
- Kenneth Ward Church and Patrick Hanks. 1990. Word Association Norms, Mutual Information, and Lexicography. Comput. Linguist. 16, 1 (March 1990), 22--29. Retrieved from http://dl.acm.org/citation.cfm?id=89086.89095Google ScholarDigital Library
- Jacob Cohen. 2003. A power primer. American Psychological Association, Washington, DC, US.Google Scholar
- Sarah D'Angelo and Andrew Begel. 2017. Improving Communication Between Pair Programmers Using Shared Gaze Awareness. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 6245--6290. DOI:https://doi.org/10.1145/3025453.3025573Google ScholarDigital Library
- Rick Dale, Riccardo Fusaroli, Nicholas D Duran, and Daniel C Richardson. 2013. The self-organization of human interaction. In Psychology of learning and motivation. Elsevier, 43--95.Google Scholar
- Jean Decety and Margarita Svetlova. 2012. Putting together phylogenetic and ontogenetic perspectives on empathy. Dev. Cogn. Neurosci. 2, 1 (2012), 1--24. DOI:https://doi.org/https://doi.org/10.1016/j.dcn.2011.05.003Google ScholarCross Ref
- Dejana Diziol, Erin Walker, Nikol Rummel, and Kenneth R Koedinger. 2010. Using Intelligent Tutor Technology to Implement Adaptive Support for Student Collaboration. Educ. Psychol. Rev. 22, 1 (March 2010), 89--102. DOI:https://doi.org/10.1007/s10648-009--9116--9Google Scholar
- Heather A Faucett, Matthew L Lee, and Scott Carter. 2017. I Should Listen More: Real-time Sensing and Feedback of Non-Verbal Communication in Video Telehealth. Proc. ACM Hum.-Comput. Interact. 1, CSCW (December 2017), 44:1--44:19. DOI:https://doi.org/10.1145/3134679Google ScholarDigital Library
- Michael Flor, Su-Youn Yoon, Jiangang Hao, Lei Liu, and Alina von Davier. 2016. Automated classification of collaborative problem solving interactions in simulated science tasks. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, 31--41.Google ScholarCross Ref
- Daniel Gatica-Perez. 2009. Automatic nonverbal analysis of social interaction in small groups: A review. Image Vis. Comput. 27, 12 (2009), 1775--1787.Google ScholarDigital Library
- Arthur C Graesser, Stephen M Fiore, Samuel Greiff, Jessica Andrews-Todd, Peter W Foltz, and Friedrich W Hesse. 2018. Advancing the Science of Collaborative Problem Solving. Psychol. Sci. Public Interes. 19, 2 (2018), 59--92. DOI:https://doi.org/10.1177/1529100618808244Google ScholarCross Ref
- Patrick Griffin, Esther Care, and Barry McGaw. 2012. The Changing Role of Education and Schools. In Assessment and Teaching of 21st Century Skills, Patrick Griffin, Barry McGaw and Esther Care (eds.). Springer Netherlands, Dordrecht, 1--15. DOI:https://doi.org/10.1007/978--94-007--2324--5_1Google Scholar
- Deborah H. Gruenfeld, Elizabeth A. Mannix, Katherine Y. Williams, and Margaret A. Neale. 1996. Group composition and decision making: How member familiarity and information distribution affect process and performance. Organ. Behav. Hum. Decis. Process. 67, 1 (1996), 1--15. DOI:https://doi.org/10.1006/obhd.1996.0061Google ScholarCross Ref
- Carl Gutwin, Scott Bateman, Gaurav Arora, and Ashley Coveney. 2017. Looking Away and Catching Up: Dealing with Brief Attentional Disconnection in Synchronous Groupware. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17), 2221--2235. DOI:https://doi.org/10.1145/2998181.2998226Google ScholarDigital Library
- J A Hanley and B J McNeil. 1982. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143, 1 (1982), 29--36. DOI:https://doi.org/10.1148/radiology.143.1.7063747Google ScholarCross Ref
- Jiangang Hao, Lei Chen, Michael Flor, Lei Liu, and Alina A von Davier. 2017. CPS-Rater: Automated Sequential Annotation for Conversations in Collaborative Problem-Solving Activities. ETS Res. Rep. Ser. 2017, 1 (2017), 1--9. DOI:https://doi.org/10.1002/ets2.12184Google ScholarCross Ref
- Friedrich Hesse, Esther Care, Juergen Buder, Kai Sassenberg, and Patrick Griffin. 2015. A Framework for Teachable Collaborative Problem Solving Skills. In Assessment and Teaching of 21st Century Skills: Methods and Approach, Patrick Griffin and Esther Care (eds.). Springer Netherlands, Dordrecht, 37--56. DOI:https://doi.org/10.1007/978--94-017--9395--7_2Google Scholar
- Hayley Hung and Daniel Gatica-Perez. 2010. Estimating cohesion in small groups using audio-visual nonverbal behavior. IEEE Trans. Multimed. 12, 6 (2010), 563--575.Google ScholarDigital Library
- Melvyn J Hunt. 1990. Figures of merit for assessing connected-word recognisers. Speech Commun. 9, 4 (1990), 329--336. DOI:https://doi.org/https://doi.org/10.1016/0167--6393(90)90008-WGoogle ScholarCross Ref
- Kerttu Huttunen, Heikki Keränen, Eero Väyrynen, Rauno Pääkkönen, and Tuomo Leino. 2011. Effect of cognitive load on speech prosody in aviation: Evidence from military simulator flights. Appl. Ergon. 42, 2 (2011), 348--357.Google ScholarCross Ref
- Hirofumi Inaguma, Koji Inoue, Shizuka Nakamura, Katsuya Takanashi, and Tatsuya Kawahara. 2016. Prediction of ice-breaking between participants using prosodic features in the first meeting dialogue. In Proceedings of the 2nd Workshop on Advancements in Social Signal Processing for Multimodal Interaction, 11--15.Google ScholarDigital Library
- Ryo Ishii, Kazuhiro Otsuka, Shiro Kumano, Ryuichiro Higashinaka, and Junji Tomita. 2018. Analyzing Gaze Behavior and Dialogue Act During Turn-taking for Estimating Empathy Skill Level. In Proceedings of the 2018 on International Conference on Multimodal Interaction (ICMI '18), 31--39. DOI:https://doi.org/10.1145/3242969.3242978Google ScholarDigital Library
- Patrick Jermann and Marc-Antoine Nüssli. 2012. Effects of sharing text selections on gaze cross-recurrence and interaction quality in a pair programming task. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work, 1125--1134.Google ScholarDigital Library
- Kristiina Jokinen, Hirohisa Furukawa, Masafumi Nishida, and Seiichi Yamamoto. 2013. Gaze and turn-taking behavior in casual conversational interactions. ACM Trans. Interact. Intell. Syst. 3, 2 (August 2013), 12:1--12:30. DOI:https://doi.org/10.1145/2499474.2499481Google ScholarDigital Library
- Janice R Kelly and Sigal G Barsade. 2001. Mood and emotions in small groups and work teams. Organ. Behav. Hum. Decis. Process. 86, 1 (2001), 99--130.Google ScholarCross Ref
- Guillaume Lemaitre, Fernando Nogueira, and Christos K. Aridas. 2017. Imbalanced-learn: A python toolbox to tackle the curse of imbalanced datasets in machine learning. J. Mach. Learn. Res. 18, 1 (2017), 559--563.Google ScholarDigital Library
- Rivka Levitan, Agustin Gravano, Laura Willson, Stefan Benus, Julia Hirschberg, and Ani Nenkova. 2012. Acoustic-prosodic entrainment and social behavior. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT '12), 11--19.Google ScholarDigital Library
- Dekang Lin. 1998. Extracting collocations from text corpora. In First workshop on computational terminology, 57--63.Google Scholar
- Rachel Tsz-Wai Lo, Ben He, and Iadh Ounis. 2005. Automatically building a stopword list for an information retrieval system. In Journal on Digital Information Management: Special Issue on the 5th Dutch-Belgian Information Retrieval Workshop (DIR), 17--24.Google Scholar
- Moira McGregor and John C Tang. 2017. More to Meetings: Challenges in Using Speech-Based Technology to Support Meetings. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17), 2208--2220. DOI:https://doi.org/10.1145/2998181.2998335Google ScholarDigital Library
- Christian Müller, Barbara Großmann-Hutter, Anthony Jameson, Ralf Rummer, and Frank Wittig. 2001. Recognizing time pressure and cognitive load on the basis of speech: An experimental study. In International Conference on User Modeling, 24--33.Google ScholarCross Ref
- Philipp Müller, Michael Xuelin Huang, and Andreas Bulling. 2018. Detecting Low Rapport During Natural Interactions in Small Groups from Non-Verbal Behaviour. In 23rd International Conference on Intelligent User Interfaces, 153--164.Google ScholarDigital Library
- Gabriel Murray and Catharine Oertel. 2018. Predicting Group Performance in Task-Based Interaction. In Proceedings of the 20th ACM International Conference on Multimodal Interaction (ICMI '18), 14--20. DOI:https://doi.org/10.1145/3242969.3243027Google ScholarDigital Library
- Fumio Nihei, Yukiko I Nakano, Yuki Hayashi, Hung-Hsuan Hung, and Shogo Okada. 2014. Predicting Influential Statements in Group Discussions Using Speech and Head Motion Information. In Proceedings of the 16th International Conference on Multimodal Interaction (ICMI '14), 136--143. DOI:https://doi.org/10.1145/2663204.2663248Google ScholarDigital Library
- José C. Pinheiro and Douglas M. Bates. 2000. Mixed-effects models in S and S-PLUS. Springer Verlag, New York, NY.Google Scholar
- Martin F. Porter. 2001. Snowball: A Language for Stemming Algorithms.Google Scholar
- David Nadler Prata, Ryan S J d Baker, Evandro d B Costa, Carolyn P Rosé, Yue Cui, and Adriana M J B De Carvalho. 2009. Detecting and Understanding the Impact of Cognitive and Interpersonal Conflict in Computer Supported Collaborative Learning Environments. Int. Work. Gr. Educ. Data Min. (2009).Google Scholar
- Jeremy Roschelle and Stephanie D Teasley. 1995. The Construction of Shared Knowledge in Collaborative Problem Solving. In Computer Supported Collaborative Learning, 69--97.Google Scholar
- Carolyn Rosé, Yi-Chia Wang, Yue Cui, Jaime Arguello, Karsten Stegmann, Armin Weinberger, and Frank Fischer. 2008. Analyzing collaborative learning processes automatically: Exploiting the advances of computational linguistics in computer-supported collaborative learning. Int. J. Comput. Collab. Learn. 3, 3 (September 2008), 237--271. DOI:https://doi.org/10.1007/s11412-007--9034-0Google ScholarCross Ref
- Yigal Rosen. 2015. Computer-based assessment of collaborative problem solving: Exploring the feasibility of human-to-agent approach. Int. J. Artif. Intell. Educ. 25, 3 (2015), 380--406. DOI:https://doi.org/10.1007/s40593-015-0042--3Google ScholarCross Ref
- Samiha Samrose, Ru Zhao, Jeffery White, Vivian Li, Luis Nova, Yichen Lu, Mohammad Rafayet Ali, and Mohammed Ehsan Hoque. 2018. CoCo: Collaboration Coach for Understanding Team Dynamics During Video Conferencing. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 4 (January 2018), 160:1--160:24. DOI:https://doi.org/10.1145/3161186Google ScholarDigital Library
- C Schlösser, A Harrer, and A Kienle. 2018. Supporting Dyadic Chat Communication with Eye Tracking Based Reading Awareness. In 2018 IEEE 18th International Conference on Advanced Learning Technologies (ICALT), 149--151. DOI:https://doi.org/10.1109/ICALT.2018.00042Google Scholar
- Christian Schlösser. 2018. Towards concise gaze sharing. In Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, 78.Google ScholarDigital Library
- Bertrand Schneider and Roy Pea. 2013. Real-time mutual gaze perception enhances collaborative learning and collaboration quality. Int. J. Comput. Collab. Learn. 8, 4 (2013), 375--397.Google ScholarCross Ref
- Julian Schulze and Stefan Krumm. 2017. The ?virtual team player": A review and initial model of knowledge, skills, abilities, and other characteristics for virtual collaboration. Organ. Psychol. Rev. 7, 1 (February 2017), 66--95. DOI:https://doi.org/10.1177/2041386616675522Google Scholar
- H Andrew Schwartz, Johannes C Eichstaedt, Margaret L Kern, Lukasz Dziurzynski, Stephanie M Ramones, Megha Agrawal, Achal Shah, Michal Kosinski, David Stillwell, Martin E P Seligman, and others. 2013. Personality, gender, and age in the language of social media: The open-vocabulary approach. PLoS One 8, 9 (2013), e73791.Google ScholarCross Ref
- Valerie J. Shute. 2008. Focus on Formative Feedback. Rev. Educ. Res. 78, 1 (March 2008), 153--189. DOI:https://doi.org/10.3102/0034654307313795Google ScholarCross Ref
- Tanmay Sinha and Justine Cassell. 2015. We click, we align, we learn: Impact of influence and convergence processes on student learning and rapport building. In Proceedings of the 1st Workshop on Modeling INTERPERsonal SynchrONy And infLuence, 13--20.Google ScholarDigital Library
- Angela E. B. Stewart and Sidney K D'Mello. 2018. Connecting the Dots Towards Collaborative AIED: Linking Group Makeup to Process to Learning. In International Conference on Artificial Intelligence in Education, 545--556.Google Scholar
- Kristin Stoeffler, Yigal Rosen, Maria Bolsinova, and Alina von Davier. 2018. Gamified Assessment of Collaborative Skills with Chatbots. . 343--347. DOI:https://doi.org/10.1007/978--3--319--93846--2_64Google Scholar
- Chen Sun, Valerie Shute, Angela E.B. Stewart, Jade Yonehiro, Nicholas Duran, and Sidney K. D'Mello. Toward a generalized competency model of collaborative problem solving. In Press Computers and Education.Google Scholar
- Yla R Tausczik and James W Pennebaker. 2010. The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods. J. Lang. Soc. Psychol. 29, 1 (2010), 24--54. DOI:https://doi.org/10.1177/0261927X09351676Google ScholarCross Ref
- F. B. M. de Waal. 2012. The Antiquity of Empathy. Science (80-. ). 336, 6083 (May 2012), 874--876. DOI:https://doi.org/10.1126/science.1220999Google Scholar
- Mary Webb and David Gibson. 2015. Technology enhanced assessment in complex collaborative settings. Educ. Inf. Technol. 20, 4 (December 2015), 675--695. DOI:https://doi.org/10.1007/s10639-015--9413--5Google ScholarDigital Library
- Miaomiao Wen, Diyi Yang, and Carolyn Penstein Rosé. 2015. Virtual Teams in Massive Open Online Courses. In Artificial Intelligence in Education, 820--824.Google Scholar
- Ian A.G. G Wilkinson and Irene Y.Y. Y Fung. 2002. Small-group composition and peer effects. Int. J. Educ. Res. 37, 5 (2002), 425--447. DOI:https://doi.org/10.1016/S0883-0355(03)00014--4Google ScholarCross Ref
- Nancy Yao, Jeff Brewer, Sarah D'Angelo, Mike Horn, and Darren Gergle. 2018. Visualizing Gaze Information from Multiple Students to Support Remote Instruction. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, LBW051.Google ScholarDigital Library
- Bo Yin and Fang Chen. 2007. Towards automatic cognitive load measurement from speech analysis. In International Conference on Human-Computer Interaction, 1011--1020.Google ScholarCross Ref
- Yanxia Zhang, Jeffrey Olenick, Chu-Hsiang Chang, Steve W J Kozlowski, and Hayley Hung. 2018. The I in Team: Mining Personal Social Interaction Routine with Topic Models from Long-Term Team Data. In 23rd International Conference on Intelligent User Interfaces, 421--426.Google ScholarDigital Library
- Code Studio. Retrieved April 1, 2018 from https://studio.code.org/s/mc/stage/1/puzzle/1Google Scholar
- IBM. Retrieved May 2, 2018 from https://www.ibm.com/watson/services/speech-to-text/Google Scholar
- Scikit Learn. Retrieved May 3, 2018 from https://github.com/scikit-learn/scikit-learnGoogle Scholar
- 2015. PISA 2015 Collaborative Problem Solving Framework.Google Scholar
Index Terms
- I Say, You Say, We Say: Using Spoken Language to Model Socio-Cognitive Processes during Computer-Supported Collaborative Problem Solving
Recommendations
When to say "Enough is Enough!": A Study on the Evolution of Collaboratively Created Process Models
Organizations conduct series of face-to-face meetings aiming to improve work practices. In these meetings, participants from different backgrounds collaboratively design artifacts, such as knowledge or process maps. Such meetings are orchestrated and ...
I say have you say tem: profiling verbs in children data in English and Portuguese
EACL '12: Proceedings of the Workshop on Computational Models of Language Acquisition and LossIn this paper we present a profile of verb usage across ages in child-produced sentences in English and Portuguese. We examine in particular lexical and syntactic characteristics of verbs and find common trends in these languages as children's ages ...
You are what you say: using meeting participants' speech to detect their roles and expertise
ACTS '09: Proceedings of the HLT-NAACL 2006 Workshop on Analyzing Conversations in Text and SpeechOur goal is to automatically detect the functional roles that meeting participants play, as well as the expertise they bring to meetings. To perform this task, we build decision tree classifiers that use a combination of simple speech features (speech ...
Comments