Skip to main content
SpringerLink
Log in
Book cover

Innovative Assessment of Collaboration pp 173–184Cite as

Multimodal Behavioral Analytics in Intelligent Learning and Assessment Systems

  • Chapter
  • First Online:

Part of the book series: Methodology of Educational Measurement and Assessment ((MEMA))

Abstract

As the boundary blurs between the real and the virtual in today’s learning environments, there is a growing need for new assessment tools that capture behavioral aspects key to evaluating skills such as problem solving, communication, and collaboration. A key challenge is to capture and understand student behavior at fidelity sufficient to estimate cognitive and affective states as they manifest through multiple media, including speech, body pose, gestures and gaze. However, analyzing each of these modalities in isolation may result in incongruities. In addition, the affective states of a person show significant variations in time. To address these technical challenges, this paper presents a framework for developing hierarchical computational models that provide a systematic approach for extracting meaningful evidence from noisy, unstructured data. This approach utilizes multimodal data, including audio, video, and activity log files and models the temporal dynamics of student behavior patterns. To demonstrate the efficacy of our methodology, we present two pilot studies from the domains of collaborative learning and in vivo assessments of nonverbal behavior where this approach has been successfully implemented.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   119.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   159.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   159.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  • Amer, M. R., Siddiquie, B., Khan, S., Divakaran, A., & Sawhney, H. (2014). Multimodal fusion using dynamic hybrid models. In Institute of Electrical and Electronics Engineers (Ed.), 2014 IEEE Winter Conference on Applications of Computer Vision (WACV) (pp. 556–563). Los Alamitos, CA: IEEE.

    Google Scholar 

  • Baker, R. S., D’Mello, S. K., Rodrigo, M. M. T., & Graesser, A. C. (2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive-affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68(4), 223–241.

    Article  Google Scholar 

  • Bejar, I. I. (1984). Educational diagnostic assessment. Journal of Educational Measurement, 21(2), 175–189.

    Article  Google Scholar 

  • Calvo, R. A., & D’Mello, S. (2010). Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing, 1(1), 18–37.

    Article  Google Scholar 

  • Choudhury, T., & Pentland, A. (2003). Sensing and modeling human networks using the sociometer. In Institute of Electrical and Electronics Engineers (Ed.), Proceedings of the 7th IEEE International Symposium on Wearable Computers (pp. 216–222). Los Alamitos, CA: IEEE.

    Google Scholar 

  • D’Mello, S., & Graesser, A. (2012). Dynamics of affective states during complex learning. Learning and Instruction, 22(2), 145–157.

    Article  Google Scholar 

  • Ekman, P., & Friesen, W. V. (1977). Facial action coding system.

    Google Scholar 

  • Farrington, C. A., Roderick, M., Allensworth, E., Nagaoka, J., Keyes, T. S., Johnson, D. W., et al. (2012). Teaching adolescents to become learners: The role of noncognitive factors in shaping school performance: A critical literature review. Chicago, IL: Consortium on Chicago School Research.

    Google Scholar 

  • Fisch, S. M. (2005). Making educational computer games “educational.” In Association for Computing Machinery (Ed.), Proceedings of the 4th International Conference for Interaction Design and Children (pp. 56–61). New York, NY: Association for Computing Machinery.

    Google Scholar 

  • Kipp, M. (2001) Anvil—A generic annotation tool for multimodal dialogue. In Proceedings of the 7th European Conference on Speech Communication and Technology (Eurospeech) (pp. 1367–1370).

    Google Scholar 

  • Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill based, and affective learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, 311–328.

    Article  Google Scholar 

  • Kyllonen, P. C., Lipnevich, A. A., Burrus, J., & Roberts, R. D. (2014). Personality, motivation, and college readiness: A prospectus for assessment and development (Research Report No. RR-14-06). Princeton, NJ: Educational Testing Service.

    Google Scholar 

  • Lakin, J. L., Jefferis, V. E., Cheng, C. M., & Chartrand, T. L. (2003). The chameleon effect as social glue: Evidence for the evolutionary significance of nonconscious mimicry. Journal of Nonverbal Behavior, 27(3), 145–162.

    Article  Google Scholar 

  • Lang, D. T. (2013). XML: Tools for parsing and generating XML within R and S-Plus (R package version 3.98–1.1). Retrieved from http://CRAN.R-project.org/package=XML

  • Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J. R., et al. (2011). The computer expression recognition toolbox (CERT). In Institute of Electrical and Electronics Engineers (Ed.), Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition (pp. 298–305). Los Alamitos, CA: IEEE.

    Google Scholar 

  • Liu, L., Hao, J., von Davier, A., Kyllonen, P., & Zapata-Rivera, D. (2016). A tough nut to crack: Measuring collaborative problem solving. In Y. Rosen, S. Ferrara, & M. Mosharraf (Eds.), Handbook of research on computational tools for real-world skill development. Hershey, PA: IGI-Global.

    Google Scholar 

  • Luna Bazaldua, D. A., Hao, J., Khan, S., Liu, L., von Davier, A. A., & Wang, Z. (2015). On convergence of cognitive and non-cognitive behavior in collaborative activity. In O. C. Santos, J. G. Boticario, C. Romero, M. Pechenizkiy, A. Merceron, P. Mitros, … M. Desmarais (Eds.), Proceedings of the 8th International Conference on Educational Data Mining Conference (pp. 496–499). Madrid, Spain: International Educational Data Mining Society.

    Google Scholar 

  • Mahmoud, M., & Robinson, P. (2011). Interpreting hand-over-face gestures. In S. D’Mello, A. Graesser, B. Schuller, & J-C Martin (Eds.), Proceedings of the International Conference on Affective Computing and Intelligent Interaction (pp. 248–255). New York, NY: Springer.

    Google Scholar 

  • Morency, L. P., de Kok, I., & Gratch, J. (2010). A probabilistic multimodal approach for predicting listener backchannels. Autonomous Agents and Multi-Agent Systems, 20(1), 70–84.

    Article  Google Scholar 

  • National Research Council. (2011). Assessing 21st century skills. Washington, DC: National Academies Press.

    Google Scholar 

  • Olguin, D. O., & Pentland, A. (2010). Assessing group performance from collective behavior. In Association for Computing Machinery (Ed.), CSCW 2010 Workshop on Collective Intelligence in Organizations. New York, NY: Association for Computing Machinery.

    Google Scholar 

  • Pentland, A. (2008). Honest signals: How they shape our world. Cambridge, MA: MIT Press.

    Google Scholar 

  • Romero, C., & Ventura, S. (2007). Educational data mining: A survey from 1995 to 2005. Expert Systems with Applications, 35, 135–146.

    Article  Google Scholar 

  • Siddiquie, B., Khan, S., Divakaran, A., & Sawhney, H. (2013, July). Affect Analysis in natural human interaction using joint hidden conditional random fields. In Institute of Electrical and Electronics Engineers (Ed.), 2013 IEEE International Conference on Multimedia and Expo (ICME 2013) (pp. 1–6). Los Alamitos, CA: IEEE.

    Google Scholar 

  • Smarter Balanced Assessment Consortium. (n.d.). Thermometer crickets: Grade 11 performance task. Retrieved from http://www.smarterbalanced.org/wordpress/wp-content/uploads/2012/09/performance-tasks/crickets.pdf

  • Tosi, H. L., Mero, N. P., & Rizzo, J. R. (2000). Managing organizational behavior (4th ed.). Cambridge, MA: Blackwell Publishers.

    Google Scholar 

  • Whitehill, J., Serpell, Z., Lin, Y. C., Foster, A., & Movellan, J. R. (2014). The faces of engagement: Automatic recognition of student engagement from facial expressions. IEEE Transactions on Affective Computing, 5(1), 86–98.

    Article  Google Scholar 

  • Woolf, B., Burleson, W., Arroyo, I., Dragon, T., Cooper, D., & Picard, R. (2009). Affect-aware tutors: recognizing and responding to student affect. International Journal of Learning Technology, 4(3–4), 129–164.

    Article  Google Scholar 

  • Woolley, A. W., Chabris, C. F., Pentland, A., Hashmi, N., & Malone, T. W. (2010). Evidence for a collective intelligence factor in the performance of human groups. Science, 330(6004), 686–688.

    Article  Google Scholar 

  • Zapata-Rivera, D., Jackson, T., Liu, L., Bertling, M., Vezzu, M., & Katz, I. R. (2014). Assessing science inquiry skills using trialogues. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 8474, pp. 625–626). Berlin, Germany: Springer.

    Google Scholar 

Download references

Acknowledgements

This research has benefited from help in data collection efforts, technical insight and valuable feedback the author received from colleagues including Diego Luna Bazaldua, Alina von Davier, Jiangang Hao, Robert Mislevy and Ketly Jean Pierre.

Author information

Authors and Affiliations

  1. Educational Testing Service, Princeton, NJ, USA

    Saad M. Khan

Authors
  1. Saad M. Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Saad M. Khan .

Editor information

Editors and Affiliations

  1. ACT , Iowa City, Iowa, USA

    Alina A. von Davier

  2. Research and Development Division, Educational Testing Service, Princeton, New Jersey, USA

    Mengxiao Zhu

  3. Research and Development Division, Educational Testing Service, Princeton, New Jersey, USA

    Patrick C. Kyllonen

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Khan, S.M. (2017). Multimodal Behavioral Analytics in Intelligent Learning and Assessment Systems. In: von Davier, A., Zhu, M., Kyllonen, P. (eds) Innovative Assessment of Collaboration. Methodology of Educational Measurement and Assessment. Springer, Cham. https://doi.org/10.1007/978-3-319-33261-1_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-33261-1_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-33259-8

  • Online ISBN: 978-3-319-33261-1

  • eBook Packages: EducationEducation (R0)

Publish with us

Policies and ethics

Search

Navigation