skip to main content
10.1145/3636555.3636855acmotherconferencesArticle/Chapter ViewAbstractPublication PageslakConference Proceedingsconference-collections
research-article
Open Access

Epistemic Network Analysis for End-users: Closing the Loop in the Context of Multimodal Analytics for Collaborative Team Learning

Published:18 March 2024Publication History

ABSTRACT

Effective collaboration and team communication are critical across many sectors. However, the complex dynamics of collaboration in physical learning spaces, with overlapping dialogue segments and varying participant interactions, pose assessment challenges for educators and self-reflection difficulties for students. Epistemic network analysis (ENA) is a relatively novel technique that has been used in learning analytics (LA) to unpack salient aspects of group communication. Yet, most LA works based on ENA have primarily sought to advance research knowledge rather than directly aid teachers and students by closing the LA loop. We address this gap by conducting a study in which we i) engaged teachers in designing human-centred versions of epistemic networks; ii) formulated an NLP methodology to code physically distributed dialogue segments of students based on multimodal (audio and positioning) data, enabling automatic generation of epistemic networks; and iii) deployed the automatically generated epistemic networks in 28 authentic learning sessions and investigated how they can support teaching. The results indicate the viability of completing the analytics loop through the design of streamlined epistemic network representations that enable teachers to support students’ reflections.

References

  1. A. Alonso and D. Dunleavy. 2012. Building teamwork skills in healthcare: the case for communication and coordination competencies. In Improving patient safety through teamwork and team training, E. Salas and K. Frush (Eds.). Oxford University Press, 41–58.Google ScholarGoogle Scholar
  2. M. André, R. F. Mello, A. Nascimento, R. D. Lins, and D. Gašević. 2021. Toward Automatic Classification of Online Discussion Messages for Social Presence. IEEE Transactions on Learning Technologies 14, 6 (2021), 802–816. https://doi.org/10.1109/TLT.2022.3150663Google ScholarGoogle ScholarCross RefCross Ref
  3. G. Arastoopour, N. C. Chesler, D. W. Shaffer, and Z. Swiecki. 2015. Epistemic Network Analysis as a Tool for Engineering Design Assessment. In 2015 ASEE Annual Conference & Exposition. ASEE Conferences, Seattle, Washington. https://peer.asee.org/24016.Google ScholarGoogle Scholar
  4. G. Arastoopour Irgens, T. Famaye, C. Lancaster, and H. Vega Quesada. 2023. Participatory Quantitative Ethnography: Exploring New Possibilities. In Proceedings of the International Conference of Quantitative Ethnography (ICQE). in press.Google ScholarGoogle Scholar
  5. P. W. Brady and L. M. Goldenhar. 2014. A qualitative study examining the influences on situation awareness and the identification, mitigation and escalation of recognised patient risk. BMJ Quality & Safety 23, 2 (2014), 153–161. https://doi.org/10.1136/bmjqs-2012-001747Google ScholarGoogle ScholarCross RefCross Ref
  6. V. Braun and V. Clarke. 2012. Thematic analysis. In APA handbook of research methods in Psychology, Vol 2: Research designs: Quantitative, qualitative, neuropsychological, and biological., H. Cooper, P. M. Camic, D. L. Long, A. T. Panter, D. Rindskopf, and K. J. She (Eds.). APA handbooks in psychology., Vol. 2. American Psychological Association, Washington, DC, US, 57–71. https://doi.org/10.1037/13620-004Google ScholarGoogle ScholarCross RefCross Ref
  7. S. Buckingham Shum, R. Ferguson, and R. Martinez-Maldonado. 2019. Human-centred learning analytics. Journal of Learning Analytics 6, 2 (2019), 1–9.Google ScholarGoogle ScholarCross RefCross Ref
  8. M. Burtsev, A. Seliverstov, R. Airapetyan, M. Arkhipov, D. Baymurzina, N. Bushkov, O. Gureenkova, T. Khakhulin, Y. Kuratov, D. Kuznetsov, 2018. Deeppavlov: Open-source library for dialogue systems. In Proceedings of ACL 2018, System Demonstrations. 122–127.Google ScholarGoogle ScholarCross RefCross Ref
  9. C. Cortez, M. Nussbaum, G. Woywood, and R. Aravena. 2009. Learning to collaborate by collaborating: a face-to-face collaborative activity for measuring and learning basics about teamwork 1. Journal of Computer Assisted Learning 25, 2 (2009), 126–142.Google ScholarGoogle ScholarCross RefCross Ref
  10. A. Csanadi, B. Eagan, I. Kollar, D. W. Shaffer, and F. Fischer. 2018. When coding-and-counting is not enough: using epistemic network analysis (ENA) to analyze verbal data in CSCL research. International Journal of Computer-Supported Collaborative Learning 13, 4 (2018), 419–438.Google ScholarGoogle ScholarCross RefCross Ref
  11. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).Google ScholarGoogle Scholar
  12. G. M. Fernandez-Nieto, R. Martinez-Maldonado, K. Kitto, and S. Buckingham Shum. 2021. Modelling Spatial Behaviours in Clinical Team Simulations Using Epistemic Network Analysis: Methodology and Teacher Evaluation. In LAK21: 11th International Learning Analytics and Knowledge Conference (Irvine, CA, USA) (LAK21). New York, NY, USA, 386–396. https://doi.org/10.1145/3448139.3448176Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. M. Härgestam, M. Lindkvist, C. Brulin, M. Jacobsson, and M. Hultin. 2013. Communication in interdisciplinary teams: exploring closed-loop communication during in situ trauma team training. BMJ Open 3, 10 (2013). https://doi.org/10.1136/bmjopen-2013-003525Google ScholarGoogle ScholarCross RefCross Ref
  14. T. Herder, Z. Swiecki, S. S. Fougt, A. L. Tamborg, B. B. Allsopp, D. W. Shaffer, and M. Misfeldt. 2018. Supporting Teachers’ Intervention in Students’ Virtual Collaboration Using a Network Based Model. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (Sydney, New South Wales, Australia) (LAK ’18). Association for Computing Machinery, New York, NY, USA, 21–25. https://doi.org/10.1145/3170358.3170394Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. E. Jensen, M. Dale, P. J. Donnelly, C. Stone, S. Kelly, A. Godley, and S. K. D’Mello. 2020. Toward Automated Feedback on Teacher Discourse to Enhance Teacher Learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376418Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. C. M. Jorm, S. White, and T. Kaneen. 2009. Clinical handover: critical communications. Medical Journal of Australia 190, 11 (2009), 108–109.Google ScholarGoogle ScholarCross RefCross Ref
  17. S. Kelly, A. M. Olney, P. Donnelly, M. Nystrand, and S. K. D’Mello. 2018. Automatically Measuring Question Authenticity in Real-World Classrooms. Educational Researcher 47, 7 (2018), 451–464. https://doi.org/10.3102/0013189X18785613Google ScholarGoogle ScholarCross RefCross Ref
  18. J. R. Landis and G. G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics 33, 1 (1977), 159–174.Google ScholarGoogle ScholarCross RefCross Ref
  19. S. Lee, S.-H. Kim, and B. C. Kwon. 2016. Vlat: Development of a visualization literacy assessment test. IEEE transactions on visualization and computer graphics 23, 1 (2016), 551–560.Google ScholarGoogle Scholar
  20. Y. Li, M. Rakovic, B. X. Poh, D. Gasevic, and G. Chen. 2022. Automatic Classification of Learning Objectives Based on Bloom’s Taxonomy. In Proceedings of the 15th International Conference on Educational Data Mining. International Educational Data Mining Society, 530–537.Google ScholarGoogle Scholar
  21. J. Lin, W. Tan, L. Du, W. Buntine, D. Lang, D. Gašević, and G. Chen. 2023. Enhancing Educational Dialogue Act Classification with Discourse Context and Sample Informativeness. IEEE Transactions on Learning Technologies (2023), 1–13. https://doi.org/10.1109/TLT.2023.3302573Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. T. Manser. 2009. Teamwork and patient safety in dynamic domains of healthcare: a review of the literature. Acta Anaesthesiologica Scandinavica 53, 2 (2009), 143–151. https://doi.org/10.1111/j.1399-6576.2008.01717.xGoogle ScholarGoogle ScholarCross RefCross Ref
  23. Marquart, C. L., Swiecki, Z., Collier, W., Eagan, B., Woodward, R., Shaffer, and D. W.2017. rENA: Epistemic Network Analysis.Google ScholarGoogle Scholar
  24. R. Martinez-Maldonado, V. Echeverria, G. Fernandez-Nieto, L. Yan, L. Zhao, R. Alfredo, X. Li, S. Dix, H. Jaggard, R. Wotherspoon, A. Osborne, D. Gašević, and S. B. Shum. 2023. Lessons Learnt from a Multimodal Learning Analytics Deployment In-the-wild. arxiv:2303.09099 [cs.HC]Google ScholarGoogle Scholar
  25. N. McDonald, S. Schoenebeck, and A. Forte. 2019. Reliability and Inter-rater Reliability in Qualitative Research: Norms and Guidelines for CSCW and HCI Practice. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–23. https://doi.org/10.1145/3359174Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. M. L. Mchugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica 22, 3 (2012), 276–282.Google ScholarGoogle Scholar
  27. K. Miller, W. Riley, and S. Davis. 2009. Identifying key nursing and team behaviours to achieve high reliability. Journal of Nursing Management 17, 2 (2009), 247–255. https://doi.org/10.1111/j.1365-2834.2009.00978.xGoogle ScholarGoogle ScholarCross RefCross Ref
  28. S. Praharaj, M. Scheffel, M. Specht, and H. Drachsler. 2023. Measuring Collaboration Quality Through Audio Data and Learning Analytics. Springer International Publishing, Cham, 91–110. https://doi.org/10.1007/978-3-031-30992-2_6Google ScholarGoogle ScholarCross RefCross Ref
  29. S. L. Pugh, A. Rao, A. E. Stewart, and S. K. D’Mello. 2022. Do Speech-Based Collaboration Analytics Generalize Across Task Contexts?. In LAK22: 12th International Learning Analytics and Knowledge Conference (Online, USA) (LAK22). Association for Computing Machinery, New York, NY, USA, 208–218. https://doi.org/10.1145/3506860.3506894Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. S. L. Pugh, S. K. Subburaj, A. R. Rao, A. E. Stewart, J. Andrews-Todd, and S. K. D’Mello. 2021. Say What? Automatic Modeling of Collaborative Problem Solving Skills from Student Speech in the Wild.International Educational Data Mining Society (2021), 55–67.Google ScholarGoogle Scholar
  31. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. 2022. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356 (2022).Google ScholarGoogle Scholar
  32. W. Riley, H. Hansen, A. P. Gürses, S. Davis, K. Miller, and R. Priester. 2008. The nature, characteristics and patterns of perinatal critical events teams. Advances in Patient Safety: New Directions and Alternative Approaches (Vol. 3: Performance and Tools) (2008).Google ScholarGoogle Scholar
  33. E. Salas, M. A. Rosen, J. D. Held, and J. J. Weissmuller. 2009. Performance Measurement in Simulation-Based Training: A Review and Best Practices. Simulation & Gaming 40, 3 (2009), 328–376. https://doi.org/10.1177/1046878108326734Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. C. Sellberg, O. Lindmark, and H. Rystedt. 2018. Learning to navigate: the centrality of instructions and assessments for developing students’ professional competencies in simulator-based training. WMU Journal of Maritime Affairs 17, 2 (2018), 249–265.Google ScholarGoogle ScholarCross RefCross Ref
  35. D. W. Shaffer, W. Collier, and A. R. Ruis. 2016. A tutorial on epistemic network analysis: Analyzing the structure of connections in cognitive, social, and interaction data. Journal of Learning Analytics 3, 3 (2016), 9–45. https://doi.org/10.18608/jla.2016.33.3Google ScholarGoogle ScholarCross RefCross Ref
  36. G. G. Smith, C. Sorensen, A. Gump, A. J. Heindel, M. Caris, and C. D. Martinez. 2011. Overcoming student resistance to group work: Online versus face-to-face. The Internet and Higher Education 14, 2 (2011), 121–128.Google ScholarGoogle ScholarCross RefCross Ref
  37. R. Southwell, S. Pugh, E. M. Perkoff, C. Clevenger, J. B. Bush, R. Lieber, W. Ward, P. Foltz, and S. D’Mello. 2022. Challenges and Feasibility of Automatic Speech Recognition for Modeling Student Collaborative Discourse in Classrooms. thinking 27 (2022), 29. https://doi.org/10.5281/zenodo.6853109Google ScholarGoogle ScholarCross RefCross Ref
  38. A. S. Tejani, Y. S. Ng, Y. Xi, J. R. Fielding, T. G. Browning, and J. C. Rayan. 2022. Performance of Multiple Pretrained BERT Models to Automate and Accelerate Data Annotation for Large Datasets. Radiology: Artificial Intelligence 4, 4 (2022), e220007.Google ScholarGoogle Scholar
  39. P. Van den Bossche, W. Gijselaers, M. Segers, G. Woltjer, and P. Kirschner. 2011. Team learning: building shared mental models. Instructional Science 39, 3 (2011), 283–301. https://doi.org/10.1007/s11251-010-9128-3Google ScholarGoogle ScholarCross RefCross Ref
  40. D. Williamson Shaffer and A. Ruis. 2017. Epistemic Network Analysis: A Worked Example of Theory-Based Learning Analytics. In The Handbook of Learning Analytics (1 ed.), C. Lang, G. Siemens, A. F. Wise, and D. Gaševic (Eds.). Society for Learning Analytics Research (SoLAR), Alberta, Canada, 175–187. http://solaresearch.org/hla-17/hla17-chapter1Google ScholarGoogle Scholar
  41. L. Yan, R. Martinez-Maldonado, L. Zhao, S. Dix, H. Jaggard, R. Wotherspoon, X. Li, and D. Gašević. 2022. The role of indoor positioning analytics in assessment of simulation-based learning. British Journal of Educational Technology (2022). https://doi.org/10.1111/bjet.13262Google ScholarGoogle ScholarCross RefCross Ref
  42. L. Yan, L. Zhao, D. Gasevic, and R. Martinez-Maldonado. 2022. Scalability, Sustainability, and Ethicality of Multimodal Learning Analytics. In LAK22: 12th International Learning Analytics and Knowledge Conference (Online, USA) (LAK22). New York, NY, USA, 13–23.Google ScholarGoogle Scholar
  43. L. Zhao, Z. Swiecki, D. Gasevic, L. Yan, S. Dix, H. Jaggard, R. Wotherspoon, A. Osborne, X. Li, R. Alfredo, and R. Martinez-Maldonado. 2023. METS: Multimodal Learning Analytics of Embodied Teamwork Learning. In LAK23: 13th International Learning Analytics and Knowledge Conference (Arlington, TX, USA) (LAK2023). Association for Computing Machinery, New York, NY, USA, 186–196. https://doi.org/10.1145/3576050.3576076Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Epistemic Network Analysis for End-users: Closing the Loop in the Context of Multimodal Analytics for Collaborative Team Learning

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in
      • Published in

        cover image ACM Other conferences
        LAK '24: Proceedings of the 14th Learning Analytics and Knowledge Conference
        March 2024
        962 pages
        ISBN:9798400716188
        DOI:10.1145/3636555

        Copyright © 2024 ACM

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 18 March 2024

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Research
        • Refereed limited

        Acceptance Rates

        Overall Acceptance Rate236of782submissions,30%
      • Article Metrics

        • Downloads (Last 12 months)62
        • Downloads (Last 6 weeks)62

        Other Metrics

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format .

      View HTML Format