ABSTRACT
Effective collaboration and team communication are critical across many sectors. However, the complex dynamics of collaboration in physical learning spaces, with overlapping dialogue segments and varying participant interactions, pose assessment challenges for educators and self-reflection difficulties for students. Epistemic network analysis (ENA) is a relatively novel technique that has been used in learning analytics (LA) to unpack salient aspects of group communication. Yet, most LA works based on ENA have primarily sought to advance research knowledge rather than directly aid teachers and students by closing the LA loop. We address this gap by conducting a study in which we i) engaged teachers in designing human-centred versions of epistemic networks; ii) formulated an NLP methodology to code physically distributed dialogue segments of students based on multimodal (audio and positioning) data, enabling automatic generation of epistemic networks; and iii) deployed the automatically generated epistemic networks in 28 authentic learning sessions and investigated how they can support teaching. The results indicate the viability of completing the analytics loop through the design of streamlined epistemic network representations that enable teachers to support students’ reflections.
- A. Alonso and D. Dunleavy. 2012. Building teamwork skills in healthcare: the case for communication and coordination competencies. In Improving patient safety through teamwork and team training, E. Salas and K. Frush (Eds.). Oxford University Press, 41–58.Google Scholar
- M. André, R. F. Mello, A. Nascimento, R. D. Lins, and D. Gašević. 2021. Toward Automatic Classification of Online Discussion Messages for Social Presence. IEEE Transactions on Learning Technologies 14, 6 (2021), 802–816. https://doi.org/10.1109/TLT.2022.3150663Google ScholarCross Ref
- G. Arastoopour, N. C. Chesler, D. W. Shaffer, and Z. Swiecki. 2015. Epistemic Network Analysis as a Tool for Engineering Design Assessment. In 2015 ASEE Annual Conference & Exposition. ASEE Conferences, Seattle, Washington. https://peer.asee.org/24016.Google Scholar
- G. Arastoopour Irgens, T. Famaye, C. Lancaster, and H. Vega Quesada. 2023. Participatory Quantitative Ethnography: Exploring New Possibilities. In Proceedings of the International Conference of Quantitative Ethnography (ICQE). in press.Google Scholar
- P. W. Brady and L. M. Goldenhar. 2014. A qualitative study examining the influences on situation awareness and the identification, mitigation and escalation of recognised patient risk. BMJ Quality & Safety 23, 2 (2014), 153–161. https://doi.org/10.1136/bmjqs-2012-001747Google ScholarCross Ref
- V. Braun and V. Clarke. 2012. Thematic analysis. In APA handbook of research methods in Psychology, Vol 2: Research designs: Quantitative, qualitative, neuropsychological, and biological., H. Cooper, P. M. Camic, D. L. Long, A. T. Panter, D. Rindskopf, and K. J. She (Eds.). APA handbooks in psychology., Vol. 2. American Psychological Association, Washington, DC, US, 57–71. https://doi.org/10.1037/13620-004Google ScholarCross Ref
- S. Buckingham Shum, R. Ferguson, and R. Martinez-Maldonado. 2019. Human-centred learning analytics. Journal of Learning Analytics 6, 2 (2019), 1–9.Google ScholarCross Ref
- M. Burtsev, A. Seliverstov, R. Airapetyan, M. Arkhipov, D. Baymurzina, N. Bushkov, O. Gureenkova, T. Khakhulin, Y. Kuratov, D. Kuznetsov, 2018. Deeppavlov: Open-source library for dialogue systems. In Proceedings of ACL 2018, System Demonstrations. 122–127.Google ScholarCross Ref
- C. Cortez, M. Nussbaum, G. Woywood, and R. Aravena. 2009. Learning to collaborate by collaborating: a face-to-face collaborative activity for measuring and learning basics about teamwork 1. Journal of Computer Assisted Learning 25, 2 (2009), 126–142.Google ScholarCross Ref
- A. Csanadi, B. Eagan, I. Kollar, D. W. Shaffer, and F. Fischer. 2018. When coding-and-counting is not enough: using epistemic network analysis (ENA) to analyze verbal data in CSCL research. International Journal of Computer-Supported Collaborative Learning 13, 4 (2018), 419–438.Google ScholarCross Ref
- J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).Google Scholar
- G. M. Fernandez-Nieto, R. Martinez-Maldonado, K. Kitto, and S. Buckingham Shum. 2021. Modelling Spatial Behaviours in Clinical Team Simulations Using Epistemic Network Analysis: Methodology and Teacher Evaluation. In LAK21: 11th International Learning Analytics and Knowledge Conference (Irvine, CA, USA) (LAK21). New York, NY, USA, 386–396. https://doi.org/10.1145/3448139.3448176Google ScholarDigital Library
- M. Härgestam, M. Lindkvist, C. Brulin, M. Jacobsson, and M. Hultin. 2013. Communication in interdisciplinary teams: exploring closed-loop communication during in situ trauma team training. BMJ Open 3, 10 (2013). https://doi.org/10.1136/bmjopen-2013-003525Google ScholarCross Ref
- T. Herder, Z. Swiecki, S. S. Fougt, A. L. Tamborg, B. B. Allsopp, D. W. Shaffer, and M. Misfeldt. 2018. Supporting Teachers’ Intervention in Students’ Virtual Collaboration Using a Network Based Model. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (Sydney, New South Wales, Australia) (LAK ’18). Association for Computing Machinery, New York, NY, USA, 21–25. https://doi.org/10.1145/3170358.3170394Google ScholarDigital Library
- E. Jensen, M. Dale, P. J. Donnelly, C. Stone, S. Kelly, A. Godley, and S. K. D’Mello. 2020. Toward Automated Feedback on Teacher Discourse to Enhance Teacher Learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376418Google ScholarDigital Library
- C. M. Jorm, S. White, and T. Kaneen. 2009. Clinical handover: critical communications. Medical Journal of Australia 190, 11 (2009), 108–109.Google ScholarCross Ref
- S. Kelly, A. M. Olney, P. Donnelly, M. Nystrand, and S. K. D’Mello. 2018. Automatically Measuring Question Authenticity in Real-World Classrooms. Educational Researcher 47, 7 (2018), 451–464. https://doi.org/10.3102/0013189X18785613Google ScholarCross Ref
- J. R. Landis and G. G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics 33, 1 (1977), 159–174.Google ScholarCross Ref
- S. Lee, S.-H. Kim, and B. C. Kwon. 2016. Vlat: Development of a visualization literacy assessment test. IEEE transactions on visualization and computer graphics 23, 1 (2016), 551–560.Google Scholar
- Y. Li, M. Rakovic, B. X. Poh, D. Gasevic, and G. Chen. 2022. Automatic Classification of Learning Objectives Based on Bloom’s Taxonomy. In Proceedings of the 15th International Conference on Educational Data Mining. International Educational Data Mining Society, 530–537.Google Scholar
- J. Lin, W. Tan, L. Du, W. Buntine, D. Lang, D. Gašević, and G. Chen. 2023. Enhancing Educational Dialogue Act Classification with Discourse Context and Sample Informativeness. IEEE Transactions on Learning Technologies (2023), 1–13. https://doi.org/10.1109/TLT.2023.3302573Google ScholarDigital Library
- T. Manser. 2009. Teamwork and patient safety in dynamic domains of healthcare: a review of the literature. Acta Anaesthesiologica Scandinavica 53, 2 (2009), 143–151. https://doi.org/10.1111/j.1399-6576.2008.01717.xGoogle ScholarCross Ref
- Marquart, C. L., Swiecki, Z., Collier, W., Eagan, B., Woodward, R., Shaffer, and D. W.2017. rENA: Epistemic Network Analysis.Google Scholar
- R. Martinez-Maldonado, V. Echeverria, G. Fernandez-Nieto, L. Yan, L. Zhao, R. Alfredo, X. Li, S. Dix, H. Jaggard, R. Wotherspoon, A. Osborne, D. Gašević, and S. B. Shum. 2023. Lessons Learnt from a Multimodal Learning Analytics Deployment In-the-wild. arxiv:2303.09099 [cs.HC]Google Scholar
- N. McDonald, S. Schoenebeck, and A. Forte. 2019. Reliability and Inter-rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–23. https://doi.org/10.1145/3359174Google ScholarDigital Library
- M. L. Mchugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica 22, 3 (2012), 276–282.Google Scholar
- K. Miller, W. Riley, and S. Davis. 2009. Identifying key nursing and team behaviours to achieve high reliability. Journal of Nursing Management 17, 2 (2009), 247–255. https://doi.org/10.1111/j.1365-2834.2009.00978.xGoogle ScholarCross Ref
- S. Praharaj, M. Scheffel, M. Specht, and H. Drachsler. 2023. Measuring Collaboration Quality Through Audio Data and Learning Analytics. Springer International Publishing, Cham, 91–110. https://doi.org/10.1007/978-3-031-30992-2_6Google ScholarCross Ref
- S. L. Pugh, A. Rao, A. E. Stewart, and S. K. D’Mello. 2022. Do Speech-Based Collaboration Analytics Generalize Across Task Contexts?. In LAK22: 12th International Learning Analytics and Knowledge Conference (Online, USA) (LAK22). Association for Computing Machinery, New York, NY, USA, 208–218. https://doi.org/10.1145/3506860.3506894Google ScholarDigital Library
- S. L. Pugh, S. K. Subburaj, A. R. Rao, A. E. Stewart, J. Andrews-Todd, and S. K. D’Mello. 2021. Say What? Automatic Modeling of Collaborative Problem Solving Skills from Student Speech in the Wild.International Educational Data Mining Society (2021), 55–67.Google Scholar
- A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. 2022. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356 (2022).Google Scholar
- W. Riley, H. Hansen, A. P. Gürses, S. Davis, K. Miller, and R. Priester. 2008. The nature, characteristics and patterns of perinatal critical events teams. Advances in Patient Safety: New Directions and Alternative Approaches (Vol. 3: Performance and Tools) (2008).Google Scholar
- E. Salas, M. A. Rosen, J. D. Held, and J. J. Weissmuller. 2009. Performance Measurement in Simulation-Based Training: A Review and Best Practices. Simulation & Gaming 40, 3 (2009), 328–376. https://doi.org/10.1177/1046878108326734Google ScholarDigital Library
- C. Sellberg, O. Lindmark, and H. Rystedt. 2018. Learning to navigate: the centrality of instructions and assessments for developing students’ professional competencies in simulator-based training. WMU Journal of Maritime Affairs 17, 2 (2018), 249–265.Google ScholarCross Ref
- D. W. Shaffer, W. Collier, and A. R. Ruis. 2016. A tutorial on epistemic network analysis: Analyzing the structure of connections in cognitive, social, and interaction data. Journal of Learning Analytics 3, 3 (2016), 9–45. https://doi.org/10.18608/jla.2016.33.3Google ScholarCross Ref
- G. G. Smith, C. Sorensen, A. Gump, A. J. Heindel, M. Caris, and C. D. Martinez. 2011. Overcoming student resistance to group work: Online versus face-to-face. The Internet and Higher Education 14, 2 (2011), 121–128.Google ScholarCross Ref
- R. Southwell, S. Pugh, E. M. Perkoff, C. Clevenger, J. B. Bush, R. Lieber, W. Ward, P. Foltz, and S. D’Mello. 2022. Challenges and Feasibility of Automatic Speech Recognition for Modeling Student Collaborative Discourse in Classrooms. thinking 27 (2022), 29. https://doi.org/10.5281/zenodo.6853109Google ScholarCross Ref
- A. S. Tejani, Y. S. Ng, Y. Xi, J. R. Fielding, T. G. Browning, and J. C. Rayan. 2022. Performance of Multiple Pretrained BERT Models to Automate and Accelerate Data Annotation for Large Datasets. Radiology: Artificial Intelligence 4, 4 (2022), e220007.Google Scholar
- P. Van den Bossche, W. Gijselaers, M. Segers, G. Woltjer, and P. Kirschner. 2011. Team learning: building shared mental models. Instructional Science 39, 3 (2011), 283–301. https://doi.org/10.1007/s11251-010-9128-3Google ScholarCross Ref
- D. Williamson Shaffer and A. Ruis. 2017. Epistemic Network Analysis: A Worked Example of Theory-Based Learning Analytics. In The Handbook of Learning Analytics (1 ed.), C. Lang, G. Siemens, A. F. Wise, and D. Gaševic (Eds.). Society for Learning Analytics Research (SoLAR), Alberta, Canada, 175–187. http://solaresearch.org/hla-17/hla17-chapter1Google Scholar
- L. Yan, R. Martinez-Maldonado, L. Zhao, S. Dix, H. Jaggard, R. Wotherspoon, X. Li, and D. Gašević. 2022. The role of indoor positioning analytics in assessment of simulation-based learning. British Journal of Educational Technology (2022). https://doi.org/10.1111/bjet.13262Google ScholarCross Ref
- L. Yan, L. Zhao, D. Gasevic, and R. Martinez-Maldonado. 2022. Scalability, Sustainability, and Ethicality of Multimodal Learning Analytics. In LAK22: 12th International Learning Analytics and Knowledge Conference (Online, USA) (LAK22). New York, NY, USA, 13–23.Google Scholar
- L. Zhao, Z. Swiecki, D. Gasevic, L. Yan, S. Dix, H. Jaggard, R. Wotherspoon, A. Osborne, X. Li, R. Alfredo, and R. Martinez-Maldonado. 2023. METS: Multimodal Learning Analytics of Embodied Teamwork Learning. In LAK23: 13th International Learning Analytics and Knowledge Conference (Arlington, TX, USA) (LAK2023). Association for Computing Machinery, New York, NY, USA, 186–196. https://doi.org/10.1145/3576050.3576076Google ScholarDigital Library
Index Terms
- Epistemic Network Analysis for End-users: Closing the Loop in the Context of Multimodal Analytics for Collaborative Team Learning
Recommendations
METS: Multimodal Learning Analytics of Embodied Teamwork Learning
LAK2023: LAK23: 13th International Learning Analytics and Knowledge ConferenceEmbodied team learning is a form of group learning that occurs in co-located settings where students need to interact with others while actively using resources in the physical learning space to achieve a common goal. In such situations, communication ...
Accessible multimodal-interaction platform for computer-supported collaborative learning system
IMCOM '17: Proceedings of the 11th International Conference on Ubiquitous Information Management and CommunicationIn order to facilitate learning in the context of social interactions, a number of studies have proposed computer-supported collaborative learning (CSCL) systems by using information and communication technologies. The basic design of a CSCL system is ...
Multimodal data to design visual learning analytics for understanding regulation of learning
AbstractThe increased interest in multimodal data collection in the learning sciences demands for new and powerful methodological and analytical techniques and technologies. It is especially challenging for learning scientists to handle, ...
Highlights- SLAM-KIT is designed to facilitate data visualisation for regulation of learning.
Comments