Skip to main content
SpringerLink
Log in

Robust Team Communication Analytics with Transformer-Based Dialogue Modeling

  • Conference paper
  • First Online:
Artificial Intelligence in Education (AIED 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13916))

Included in the following conference series:

  • 3078 Accesses

Abstract

Adaptive training environments that can provide reliable insight into team communication offer great potential for team training and assessment. However, traditional techniques that enable meaningful analysis of team communication such as human transcription and speech classification are especially resource-intensive without machine assistance. Additionally, developing computational models that can perform robust team communication analytics based on small datasets poses significant challenges. We present a transformer-based team communication analysis framework that classifies each team member utterance according to dialogue act and the type of information flow exhibited. The framework utilizes domain-specific transfer learning of transformer-based language models pre-trained with large-scale external data and a prompt engineering method that represents both speaker utterances and speaker roles. Results from our evaluation of team communication data collected from live team training exercises suggest the transformer-based framework fine-tuned with team communication data significantly outperforms state-of-the-art models on both dialogue act recognition and information flow classification and additionally demonstrates improved domain-transfer capabilities.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Kaliisa, R., Rienties, B., Mørch, A.I., Kluge, A.: Social learning analytics in computer-supported collaborative learning environments: a systematic review of empirical studies. Comput. Educ. Open 3, 100073 (2022)

    Article  Google Scholar 

  2. Meslec, N., Duel, J., Soeters, J.: The role of teamwork on team performance in extreme military environments: an empirical study. Team Perform. Manage.: an Int. J. 26(5/6), 325–339 (2020)

    Article  Google Scholar 

  3. Marlow, S.L., Lacerenza, C.N., Paoletti, J., Burke, C.S., Salas, E.: Does team communication represent a one-size-fits-all approach?: A meta-analysis of team communication and performance. Organ. Behav. Hum. Decis. Process. 144, 145–170 (2018)

    Article  Google Scholar 

  4. Grimm, D.A.P., Gorman, J.C., Robinson, E., Winner, J.: Measuring adaptive team coordination in an enroute care training scenario. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 66, no. 1, pp. 50–54. SAGE Publications, Los Angeles (2022)

    Google Scholar 

  5. Waring, S., Alison, L., Shortland, N., Humann, M.: The role of information sharing on decision delay during multiteam disaster response. Cogn. Technol. Work 22(2), 263–279 (2019)

    Article  Google Scholar 

  6. Endsley, M.R.: Situation awareness. In: Salvendy, G., Karwowski, W. (eds.) Handbook of Human Factors and Ergonomics, 5th edn., pp. 434–455. John Wiley & Sons Inc, Hoboken (2021)

    Chapter  Google Scholar 

  7. Gorman, J.C., Grimm, D.A., Stevens, R.H., Galloway, T., Willemsen-Dunlap, A.M., Halpin, D.J.: Measuring real-time team cognition during team training. Hum. Factors 62(5), 825–860 (2020)

    Article  Google Scholar 

  8. Stucky, C.H., De Jong, M.J., Kabo, F.W.: Military surgical team communication: implications for safety. Mil. Med. 185(3–4), e448–e456 (2020)

    Article  Google Scholar 

  9. Min, W., Spain, R., Saville, J.D., Mott, B., Brawner, K., Johnston, J., et al.: Multidimensional team communication modeling for adaptive team training: a hybrid deep learning and graphical modeling framework. In: Roll, I., McNamara, D., Sosnovsky, S., Luckin, R., Dimitrova, V. (eds.) AIED 2021, LNCS, vol. 12748, pp. 293–305. Springer International Publishing, Cham (2021)

    Google Scholar 

  10. Foltz, P.W., Martin, M.J.: Automated communication analysis of teams. In: Salas, E., Goodwin, G.F., Burke, C.S. (eds.) Team Effectiveness in Complex Organizations, 1st edn., pp. 445–466. Routledge, New York (2008)

    Google Scholar 

  11. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., et al.: Attention is all you need. In: von Luxburg, U. et al., (eds.) Advances in Neural Information Processing Systems 30 (NIPS 2017), pp. 5999–6009. Curran Associates Inc, Red Hook (2017)

    Google Scholar 

  12. Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Burstein, J., Doran, C., Solorio, T. (eds.) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186. Association for Computational Linguistics, Stroudsburg (2019)

    Google Scholar 

  13. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., et al.: Language models are few-shot learners. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems 33 (NeurIPS 2020), pp. 1877–1901. Curran Associates Inc., Red Hook (2020)

    Google Scholar 

  14. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21(1), 5485–5551 (2020)

    MathSciNet  MATH  Google Scholar 

  15. Litman, D.: Natural language processing for enhancing teaching and learning. In: AAAI’16: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pp. 4170–4176. AAAI Press, Washington (2016)

    Google Scholar 

  16. Sawatzki, J., Schlippe, T., Benner-Wickner, M.: Deep learning techniques for automatic short answer grading: predicting scores for english and german answers. In: Cheng, E.C.K., Koul, R.B., Wang, T., Yu, X. (eds.) Artificial Intelligence in Education: Emerging Technologies, Models and Applications. LNDECT, vol. 104, pp. 65–75. Springer, Singapore (2022)

    Google Scholar 

  17. Ramesh, D., Sanampudi, S.K.: An automated essay scoring systems: a systematic literature review. Artif. Intell. Rev. 55(3), 2495–2527 (2021)

    Article  Google Scholar 

  18. Emerson, A., Min, W., Azevedo, R., Lester, J.: Early prediction of student knowledge in game-based learning with distributed representations of assessment questions. Br. J. Edu. Technol. 54(1), 40–57 (2022)

    Article  Google Scholar 

  19. Gupta, A., Carpenter, D., Min, W., Rowe, J., Azevedo, R., Lester, J.: Enhancing multimodal goal recognition in open-world games with natural language player reflections. In: Ware, S.G., Eger, M. (eds.) Proceedings of the Eighteenth Annual AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, pp. 37–44. AAAI Press, Palo Alto (2022)

    Google Scholar 

  20. Park, K., Sohn, H., Min, W., Mott, B., Glazewski, K., Hmelo-Silver, C., et al.: Disruptive talk detection in multi-party dialogue within collaborative learning environments with a regularized user-aware network. In: Lemon, O., et al. (eds.) Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 490–499. Association for Computational Linguistics, Stroudsburg (2022)

    Google Scholar 

  21. Lun, J., Zhu, J., Tang, Y., Yang, M.: Multiple data augmentation strategies for improving performance on automatic short answer scoring. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, ed. 09, pp. 13389–13396. AAAI Press, Palo Alto (2020)

    Google Scholar 

  22. Bragg, J., Cohan, A., Lo, K., Beltagy, I.: FLEX: unifying evaluation for few-shot NLP. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P.S., Wortman Vaughan, J. (eds.) Advances in Neural Information Processing Systems 34 (NeurIPS 2021), pp. 15787–15800. Curran Associates, Inc., Red Hook (2021)

    Google Scholar 

  23. Carpenter, D., Emerson, A., Mott, B., Saleh, A., Glazewski, K., Hmelo-Silver, C., et al.: Detecting off-task behavior from student dialogue in game-based collaborative learning. In: Bittencourt, I.I., Cukurova, M., Muldner, K., Luckin, R., Millán, E. (eds.) AIED 2020, LNCS, vol. 12163, pp. 56–66. Springer, Heidelberg (2020)

    Google Scholar 

  24. Sottilare, R.A., Burke, C.S., Salas, E., Sinatra, A.M., Johnston, J.H., Gilbert, S.B.: Designing adaptive instruction for teams: a meta-analysis. Int. J. Artif. Intell. Educ. 28(2), 225–264 (2018)

    Article  Google Scholar 

  25. Foltz, P.W.: Automating the assessment of team collaboration through communication analysis. In: Sottilare, R.A., Graesser, A.C., Hu, X., Sinatra, A.M. (eds.) Design Recommendations for Intelligent Tutoring Systems, vol. 6, pp. 179–185. US Army Research Laboratory, Orlando (2018)

    Google Scholar 

  26. Dowell, N.M.M., Nixon, T.M., Graesser, A.C.: Group communication analysis: a computational linguistics approach for detecting sociocognitive roles in multiparty interactions. Behav. Res. Methods 51(3), 1007–1041 (2019)

    Article  Google Scholar 

  27. Ayala, D.F.M., Balasingam, B., McComb, S., Pattipati, K.R.: Markov modeling and analysis of team communication. IEEE Trans. Syst. Man, and Cybern. Syst. 50(4), 1230–1241 (2020)

    Article  Google Scholar 

  28. Yu, M., Litman, D., Paletz, S.: Investigating the relationship between multi-party linguistic entrainment, team characteristics, and the perception of team social outcomes. In: Barták, R., Brawner, K. (eds.) Proceedings of the 32nd International Florida Artificial Intelligence Research Society Conference (FLAIRS-32), pp. 227–232. AAAI Press, Palo Alto (2019)

    Google Scholar 

  29. Goodwin, G.F., Blacksmith, N., Coats, M.R.: The science of teams in the military: contributions from over 60 years of research. Am. Psychol. 73(4), 322–333 (2018)

    Article  Google Scholar 

  30. Cannon-Bowers, J.A., Salas, E., Converse, S.A.: Cognitive psychology and team training: training shared mental models of complex systems. Hum. Factors Soc. Bull. 33(12), 1–4 (1990)

    Google Scholar 

  31. Gontar, P., Fischer, U., Bengler, K.: Methods to evaluate pilots’ cockpit communication: cross-recurrence analyses vs. speech act–based analyses. J. Cogn. Eng. Decis. Making 11(4), 337–352 (2017)

    Article  Google Scholar 

  32. Nonose, K., Kanno, T., Furuta, K.: An evaluation method of team communication based on a task flow analysis. Cogn. Technol. Work 17(4), 607–618 (2015)

    Article  Google Scholar 

  33. Bowers, C.A., Jentsch, F., Salas, E., Braun, C.C.: Analyzing communication sequences for team training needs assessment. Hum. Factors 40(4), 672–679 (1998)

    Article  Google Scholar 

  34. Mesmer-Magnus, J.R., DeChurch, L.A.: Information sharing and team performance: a meta-analysis. J. Appl. Psychol. 94(2), 535–546 (2009)

    Article  Google Scholar 

  35. Spain, R.D., Min, W., Saville, J.D., Emerson, A., Pande, J., Brawner, K., et al.: Leveraging advances in natural language processing to support team communication analytics in GIFT. In: Sinatra, A. M. (ed.) Proceedings of the 10th Annual GIFT Users Symposium, pp. 147–156. US Army Combat Capabilities Development Command–Soldier Center, Orlando (2022)

    Google Scholar 

  36. Johnston, J.H., Phillips, H.L., Milham, L.M., Riddle, D.L., Townsend, L.N., DeCostanza, A.H., et al.: A team training field research study: extending a theory of team development. Front. Psychol. 10, 1480 (2019)

    Article  Google Scholar 

  37. Traum, D.: USC Institute for Creative Technologies, personal communication, July 25 (2022)

    Google Scholar 

  38. Saville, J.D., Spain, R.D., Johnston, J.H., Lester, J.: Exploration of team communication behaviors from a live training event. In: Nazir, S., Ahram, T.Z., Karwowski, W. (eds.) AHFE 2021. LNNS, vol. 269, pp. 101–108. Springer, Cham (2021)

    Chapter  Google Scholar 

  39. Saville, J.D., Spain, R.D., Johnston, J.H., Lester, J.C.: An analysis of squad communication behaviors during a field-training exercise to support tactical decision making. In: Wright, J., Barber, D. (eds.) Human Factors and Simulation, vol. 30, pp. 109–116. AHFE Open Access (2022)

    Google Scholar 

  40. config.json t5-small at main, https://huggingface.co/t5-small/blob/main/config.json. Last accessed 16 Jan 2023

Download references

Acknowledgements

The research described herein has been sponsored by the U.S. Army DEVCOM, Soldier Center under cooperative agreement W912CG-19-2-0001. The statements and opinions expressed in this article do not necessarily reflect the position or the policy of the United States Government, and no official endorsement should be inferred.

Author information

Authors and Affiliations

  1. North Carolina State University, Raleigh, NC, 27695, USA

    Jay Pande, Wookhee Min, Jason D. Saville & James Lester

  2. U.S. Army Combat Capabilities Development Command, Soldier Center, Orlando, FL, 32826, USA

    Randall D. Spain

Authors
  1. Jay Pande
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wookhee Min
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Randall D. Spain
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jason D. Saville
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. James Lester
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jay Pande .

Editor information

Editors and Affiliations

  1. University of Southern California, Los Angeles, CA, USA

    Ning Wang

  2. University of British Columbia, Vancouver, BC, Canada

    Genaro Rebolledo-Mendez

  3. North Carolina State University, Raleigh, NC, USA

    Noboru Matsuda

  4. Despacho 3.01, UNED-Grupo de Investigación aDeNu, Madrid, Spain

    Olga C. Santos

  5. University of Leeds, Leeds, UK

    Vania Dimitrova

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Pande, J., Min, W., Spain, R.D., Saville, J.D., Lester, J. (2023). Robust Team Communication Analytics with Transformer-Based Dialogue Modeling. In: Wang, N., Rebolledo-Mendez, G., Matsuda, N., Santos, O.C., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2023. Lecture Notes in Computer Science(), vol 13916. Springer, Cham. https://doi.org/10.1007/978-3-031-36272-9_52

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-36272-9_52

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-36271-2

  • Online ISBN: 978-3-031-36272-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation