Skip to main content
SpringerLink
Log in

Gesture-mediated collaboration with augmented reality headsets in a problem-based astronomy task

  • Published:
International Journal of Computer-Supported Collaborative Learning Aims and scope Submit manuscript

Abstract

Extended reality technologies such as headset-based augmented reality (AR) unlock unique opportunities to integrate gestures into the collaborative problem-solving process. The following qualitative study documents the collection and analysis of group interaction data in an astronomy sky simulation across AR and tablet technologies in a classroom setting. A total of 15 groups were coded for episodes of on-task problem-solving, conceptual engagement, and use of gesture. Analysis of coded interactions assisted in identifying vignettes facilitating exploration, orientation, perspective sharing, and communication of mental models. In addition, the use of gesture by some groups enabled the creation of shared situated conceptual spaces, bridging the AR and tablet experiences and facilitating collaborative exchange of spatial information. The patterns of gesture and collaborative knowledge interactions documented here have implications for the design of future collaborative learning environments leveraging extended reality technologies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data Availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  • Alibali, M. W. (2005). Gesture in spatial cognition: Expressing, communicating, and thinking about spatial information. Spatial Cognition and Computation, 5(4), 307–331.

    Article  Google Scholar 

  • Andrews, J. J., John Chamberlain, C. O. R. D., Koon, A., & von Davier, A. A. (2017). Designing simulations for evaluating collaborative problem solving in electronics. Making a Difference: Prioritizing Equity and Access in (CSCL) 2017, Volume 1. Philadelphia, PA: International Society of the Learning Sciences.

  • Arici, F., Yildirim, P., Caliklar, Ş., & Yilmaz, R. M. (2019). Research trends in the use of augmented reality in science education: Content and bibliometric mapping analysis. Computers & Education, 142, 103647. https://doi.org/10.1016/j.compedu.2019.103647

  • Barron, B. (2003). When smart groups fail. Journal of the Learning Sciences, 12(3), 307–359. https://doi.org/10.1207/S15327809JLS1203_1

    Article  Google Scholar 

  • Barron, B., & Darling-Hammond, L. (2008). How can we teach for meaningful learning. Powerful Learning: What We Know about Teaching for Understanding, 1, 11–16.

    Google Scholar 

  • Barsalou, L. W. (2010). Grounded cognition: Past, present, and future. Topics in Cognitive Science, 2(4), 716–724. https://doi.org/10.1111/j.1756-8765.2010.01115.x

    Article  Google Scholar 

  • Borge, M., Ong, Y. S., & Rosé, C. P. (2018). Learning to monitor and regulate collective thinking processes. International Journal of Computer-Supported Collaborative Learning, 1–32. https://doi.org/10.1007/s11412-018-9270-5

  • Bork, F., Lehner, A., Eck, U., Navab, N., Waschke, J., & Kugelmann, D. (2021). The effectiveness of collaborative augmented reality in gross anatomy teaching: A quantitative and qualitative pilot study. Anatomical Sciences Education, 14(5), 590–604. https://doi.org/10.1002/ase.2016

    Article  Google Scholar 

  • Chen, P., Liu, X., Cheng, W., & Huang, R. (2017). A review of using augmented reality in education from 2011 to 2016. In Innovations in Smart Learning (pp. 13–18). Springer. https://doi.org/10.1007/978-981-10-2419-1_2

  • Chen, W., Looi, C.-K., & Tan, S. (2010). What do students do in a F2F CSCL classroom? The optimization of multiple communications modes. Computers & Education, 55(3), 1159–1170. https://doi.org/10.1016/j.compedu.2010.05.013

    Article  Google Scholar 

  • Chen, Y., Andrews, C. D., Hmelo-Silver, C. E., & D’Angelo, C. (2019). Coding schemes as lenses on collaborative learning. Information and Learning Sciences, 121(1/2), 1–18. https://doi.org/10/ghjtht

  • Chin, C., & Chia, L. G. (2006). Problem-based learning: Using ill-structured problems in biology project work. Science Education, 90(1), 44–67. https://doi.org/10.1002/sce.20097

    Article  Google Scholar 

  • Cohen, A. D. (1994). Assessing language ability in the classroom. Heinle and Heinle.

  • Crowder, E. M. (1996). Gestures at work in sense-making science talk. Journal of the Learning Sciences, 5(3), 173–208. https://doi.org/10.1207/s15327809jls0503_2

    Article  Google Scholar 

  • Danish, J. A., Enyedy, N., Saleh, A., & Humburg, M. (2020). Learning in embodied activity framework: A sociocultural framework for embodied cognition. International Journal of Computer-Supported Collaborative Learning, 15(1), 49–87. https://doi.org/10/gmhjh9

  • DeSutter, D., & Stieff, M. (2017). Teaching students to think spatially through embodied actions: Design principles for learning environments in science, technology, engineering, and mathematics. Cognitive Research, 2(1). https://doi.org/10.1186/s41235-016-0039-y

  • Diegmann, P., Schmidt-Kraepelin, M., Eynden, S., & Basten, D. (2015). Benefits of augmented reality in educational environments-a systematic literature review. Wirtschaftsinformatik Proceedings (2015), pp. 1542–1556

  • Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences, 111(23), 8410–8415. https://doi.org/10/gctkrm

  • Gallagher, S. (2006). How the body shapes the mind. Clarendon Press.

    Google Scholar 

  • Glenberg, A. M. (2010). Embodiment as a unifying perspective for psychology. Wiley Interdisciplinary Reviews: Cognitive Science, 1(4), 586–596. https://doi.org/10.1002/wcs.55

    Article  Google Scholar 

  • Hakkarainen, K., Paavola, S., Kangas, K., & Seitamaa-Hakkarainen, P. (2013). Sociocultural perspectives on collaborative learning: Toward collaborative knowledge creation. In The International Handbook of Collaborative Learning (pp. 57–73). Routledge.

  • Hesse, F., Care, E., Buder, J., Sassenberg, K., & Griffin, P. (2015). A framework for teachable collaborative problem solving skills. In Assessment and Teaching of 21st Century Skills (pp. 37–56). Springer Netherlands. https://doi.org/10.1007/978-94-017-9395-7_2

  • Horwitz, P., von Davier, A., Chamberlain, J., Koon, A., Andrews, J., & McIntyre, C. (2017). Teaching teamwork: Electronics instruction in a collaborative environment. Community College Journal of Research and Practice, 41(6), 341–343.

    Article  Google Scholar 

  • Hung, W., Jonassen, D. H., Liu, R., et al. (2008). Problem-based learning. Handbook of Research on Educational Communications and Technology, 3(1), 485–506.

    Google Scholar 

  • Ibáñez, M.-B., & Delgado-Kloos, C. (2018). Augmented reality for STEM learning: A systematic review. Computers & Education, 123, 109–123. https://doi.org/10/gdvxnd

  • Johri, A., Williams, C., & Pembridge, J. (2013). Creative collaboration: A case study of the role of computers in supporting representational and relational interaction in student engineering design teams. International Journal of Engineering Education, 29(1), 33–44.

    Google Scholar 

  • Kang, S., Tversky, B., & Black, J. B. (2015). Coordinating gesture, word, and diagram: Explanations for experts and novices. Spatial Cognition & Computation, 15(1), 1–26. https://doi.org/10.1080/13875868.2014.958837

    Article  Google Scholar 

  • Kerr, N. L., & Tindale, R. S. (2004). Group performance and decision making. Annual Review of Psychology, 55, 623–655.

    Article  Google Scholar 

  • Kozlowski, S. W., & Ilgen, D. R. (2006). Enhancing the effectiveness of work groups and teams. Psychological Science in the Public Interest, 7(3), 77–124.

    Article  Google Scholar 

  • Kreijns, K., Kirschner, P. A., & Jochems, W. (2003). Identifying the pitfalls for social interaction in computer-supported collaborative learning environments: A review of the research. Computers in Human Behavior, 19(3), 335–353.

    Article  Google Scholar 

  • Lawrence, L., & Mercier, E. (2019). A review of the evolving definition of orchestration: Implications for research and design. Computer-Supported Collaborative Learning Conference, CSCL, 2, 829–830. https://doi.org/10.22318/CSCL2019.829

  • Lindgren, R., & Johnson-Glenberg, M. (2013). Emboldened by embodiment: Six precepts for research on embodied learning and mixed reality. Educational Researcher, 42(8), 445–452.

  • Lindgren, R., Wallon, R. C., Brown, D. E., Mathayas, N., & Kimball, N. (2016). “Show me” what you mean: Learning and design implications of eliciting gesture in student explanations In Looi, C. K., Polman, J. L., Cress, U., and Reimann, P. (Eds.). Transforming Learning, Empowering Learners: The International Conference of the Learning Sciences (ICLS) 2016, Volume 2. Singapore: International Society of the Learning Sciences.

  • Mathayas, N., Brown, D. E., & Lindgren, R. (2021). “I got to see, and I got to be a part of it”: How cued gesturing facilitates middle-school students’ explanatory modeling of thermal conduction. Journal of Research in Science Teaching, 58(10), 1557–1589. https://doi.org/10.1002/tea.21718

  • McNeill, D. (1998). Speech and gesture integration. New Directions for Child and Adolescent Development, 1998(79), 11–27.

    Article  Google Scholar 

  • Menekse, M., & Chi, M. T. H. (2019). The role of collaborative interactions versus individual construction on students’ learning of engineering concepts. European Journal of Engineering Education, 44(5), 702–725. https://doi.org/10.1080/03043797.2018.1538324

    Article  Google Scholar 

  • Mercier, E., Vourloumi, G., & Higgins, S. (2017). Student interactions and the development of ideas in multi-touch and paperbased collaborative mathematical problem solving. British Journal of Educational Technology, 48(1), 162–175. https://doi.org/10.1111/bjet.12351

  • Mercier, E., & Higgins, S. (2015). The four Ts of the collaborative classroom. CSCL 2015.

  • Mercier, E., & Higgins, S. (2014). Creating joint representations of collaborative problem solving with multi-touch technology. Journal of Computer Assisted Learning, 30(6), 497–510. https://doi.org/10.1111/jcal.12052

  • Miyake, N., & Shirouzu, H. (2019). Understanding and scaffolding constructive collaboration. In Proceedings of the twenty-fourth annual conference of the cognitive science society (pp. 48–48). Routledge.

  • Nathan, M. J. (2012). Rethinking formalisms in formal education. Educational Psychologist, 47(2), 125–148. https://doi.org/10.1080/00461520.2012.667063

    Article  Google Scholar 

  • NGSS Lead States. (2013). Next generation science standards: For states, by states. The National Academies Press.

    Google Scholar 

  • Ngo, T., Unsworth, L., & Herrington, M. (2022). Teacher orchestration of language and gesture in explaining science concepts in images. Research in Science Education, 52(3), 1013–1030. https://doi.org/10.1007/s11165-021-10011-z

    Article  Google Scholar 

  • Nokes-Malach, T. J., Zepeda, C. D., Richey, J. E., & Gadgil, S. (2019). Collaborative learning: the benefits and costs. In The Cambridge Handbook of Cognition and Education (pp. 500–527). Cambridge University Press. https://doi.org/10.1017/9781108235631.021

  • Nokes-Malach, T. J., Richey, J. E., & Gadgil, S. (2015). When is it better to learn together? Insights from research on collaborative learning. Educational Psychology Review, 27(4), 645–656.Chicago

  • O’Donnell, A. M. (2006). The role of peers and group learning. In P. A. Alexander & P. H. Winne (Eds.), Handbook of educational psychology (pp. 781–802). Lawrence Erlbaum Associates Publishers.

  • Pazos, P., Ringleb, S. I., Kidd, J., & Jones, R. (2019). Scaffolding project-based learning in an engineering and education partnership using open-access technology. International Journal of Engineering Education, 35(5), 1306–1315.

    Google Scholar 

  • Plummer, J. D., Bower, C. A., & Liben, L. S. (2016). The role of perspective taking in how children connect reference frames when explaining astronomical phenomena. International Journal of Science Education, 38(3), 345–365. https://doi.org/10.1080/09500693.2016.1140921

    Article  Google Scholar 

  • Roschelle, J. (1992). Learning by collaborating: Convergent conceptual change. Journal of the Learning Sciences, 2(3), 235–276. https://doi.org/10.1207/s15327809jls0203_1

    Article  Google Scholar 

  • Roth, W.-M., & Lawless, D. (2002). Scientific investigations, metaphorical gestures, and the emergence of abstract scientific concepts. Learning and Instruction, 12(3), 285–304. https://doi.org/10.1016/S0959-4752(01)00023-8

    Article  Google Scholar 

  • Schiffeler, N., & Stehling, V., & Hees, F., & Isenhardt, I. (2019). Effects of collaborative augmented reality on communication and interaction in learning contexts – Results of a Qualitative Pre-Study Paper presented at 2019 ASEE Annual Conference & Exposition, Tampa, Florida. https://doi.org/10.18260/1-2--32694

  • Shapiro, L., & Stolz, S. A. (2018). Embodied cognition and its significance for education. Theory and Research in Education, 1477878518822149. https://doi.org/10.1177/1477878518822149

  • Shapiro, L. (2019). Embodied cognition. Routledge. https://doi.org/10.4324/9781315180380

    Article  Google Scholar 

  • Shehab, S., & Mercier, E. (2020). Exploring the relationship between the types of interactions and progress on a task during collaborative problem solving. In I. S. Horn & M. Gresalfi (Eds.), The Interdisciplinarity of the Learning Sciences, 14th International Conference of the Learning Sciences (Vol. 3, pp. 1285–1292). International Society of the Learning Sciences (ISLS). https://repository.isls.org//handle/1/6326

  • Shin, N., Jonassen, D. H., & McGee, S. (2003). Predictors of well-structured and ill-structured problem solving in an astronomy simulation. Journal of Research in Science Teaching, 40(1), 6–33. https://doi.org/10.1002/tea.10058

    Article  Google Scholar 

  • Steier, R., Kersting, M., & Silseth, K. (2019). Imagining with improvised representations in CSCL environments. International Journal of Computer-Supported Collaborative Learning, 14(1), 109–136. https://doi.org/10.1007/s11412-019-09295-1

    Article  Google Scholar 

  • Stieff, M., Lira, M. E., & Scopelitis, S. A. (2016). Gesture supports spatial thinking in STEM. Cognition and Instruction, 34(2), 80–99. https://doi.org/10/ggcpwg

  • Tenenberg, J. (2019). Factors affecting free riding on teams: Implications for engineering education. International Journal of Engineering Education, 35(6A), 1703–1724.

    Google Scholar 

  • Villanueva, A., Zhu, Z., Liu, Z., Peppler, K., Redick, T., & Ramani, K. (2020). Meta-AR-App: An authoring platform for collaborative augmented reality in STEM classrooms. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–14. https://doi.org/10.1145/3313831.3376146

  • Volet, S., Summers, M., & Thurman, J. (2009). High-level co-regulation in collaborative learning: How does it emerge and how is it sustained? Learning and Instruction, 19(2), 128–143. https://doi.org/10.1016/j.learninstruc.2008.03.001

    Article  Google Scholar 

  • Vygotsky, L. S. (1978). Mind and society: The development of higher mental processes. Harvard University Press.

    Google Scholar 

  • Webb, N. M. (2013). Collaboration in the classroom. In International guide to student achievement (pp. 215–217). Routledge.

  • Whitelock, D., & Scanlon, E. (1998). The roles of gaze, gesture and gender in CSCL. Journal of Computer Assisted Learning, 14(2), 158–165. https://doi.org/10.1046/j.1365-2729.1998.1420158.x

    Article  Google Scholar 

  • Wilson, M. (2002). Six views of embodied cognition. Psychonomic Bulletin & Review, 9(4), 625–636. https://doi.org/10.3758/BF03196322

    Article  Google Scholar 

Download references

Acknowledgements

This material is based upon work supported by the National Science Foundation under Grant No.#1822796. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. We would like to thank the entire CEASAR staff for making this study possible, particularly Nathan Kimball and his team at Concord Consortium.

Author information

Authors and Affiliations

  1. University of Illinois Urbana-Champaign, Champaign, IL, USA

    James Planey, Robin Jephthah Rajarathinam, Emma Mercier & Robb Lindgren

Authors
  1. James Planey
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robin Jephthah Rajarathinam
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Emma Mercier
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Robb Lindgren
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to James Planey.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Planey, J., Rajarathinam, R.J., Mercier, E. et al. Gesture-mediated collaboration with augmented reality headsets in a problem-based astronomy task. Intern. J. Comput.-Support. Collab. Learn 18, 259–289 (2023). https://doi.org/10.1007/s11412-023-09398-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11412-023-09398-w

Keywords

Search

Navigation