Skip to main content
SpringerLink
Log in

Assessing Social Communication and Collaboration in Autism Spectrum Disorder Using Intelligent Collaborative Virtual Environments

  • Original Paper
  • Published:
Journal of Autism and Developmental Disorders Aims and scope Submit manuscript

Abstract

Existing literature regarding social communication outcomes of interventions in autism spectrum disorder (ASD) depends upon human raters, with limited generalizability to real world settings. Technological innovation, particularly virtual reality (VR) and collaborative virtual environments (CVE), could offer a replicable, low cost measurement platform when endowed with intelligent agent technology and peer-based interactions. We developed and piloted a novel collaborative virtual environment and intelligent agent (CRETA) for the assessment of social communication and collaboration within system and peer interactions. The system classified user statements with moderate to high accuracies. We found moderate to high agreement in displayed communication and collaboration skills between human–human and human–agent interactions. CRETA offers a promising avenue for future development of autonomous measurement systems for ASD research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Amirhosseini, M. H., & Kazemian, H. (2019). Automating the process of identifying the preferred representational system in neuro linguistic programming using natural language processing. Cognitive Processing. https://doi.org/10.1007/s10339-019-00912-3.

    Article  PubMed  Google Scholar 

  • Anagnostou, E., Jones, N., Huerta, M., Halladay, A. K., Wang, P., Scahill, L., … Dawson, G. (2015). Measuring social communication behaviors as a treatment endpoint in individuals with autism spectrum disorder. Autism, 19(5), 622–636. https://doi.org/10.1177/1362361314542955.

    Article  Google Scholar 

  • Battocchi, A., Pianesi, F., Tomasini, D., Zancanaro, M., Esposito, G., Venuti, P., … Weiss, P. L. (2009). Collaborative Puzzle Game: A tabletop interactive game for fostering collaboration in children with Autism Spectrum Disorders (ASD). Paper presented at the Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces.

  • Bauminger-Zviely, N., Eden, S., Zancanaro, M., Weiss, P. L., & Gal, E. (2013). Increasing social engagement in children with high-functioning autism spectrum disorder using collaborative technologies in the school environment. Autism,17(3), 317–339. https://doi.org/10.1177/1362361312472989.

    Article  PubMed  Google Scholar 

  • Bekele, E., Crittendon, J., Zheng, Z., Swanson, A., Weitlauf, A., Warren, Z., et al. (2014). Assessing the utility of a virtual environment for enhancing facial affect recognition in adolescents with autism. Journal of Autism and Developmental Disorders,44(7), 1641–1650. https://doi.org/10.1007/s10803-014-2035-8.

    Article  PubMed  PubMed Central  Google Scholar 

  • Bekele, E., Zheng, Z., Swanson, A., Crittendon, J., Warren, Z., & Sarkar, N. (2013). Understanding how adolescents with autism respond to facial expressions in virtual reality environments. IEEE Transactions on Visualization and Computer Graphics,19(4), 711–720. https://doi.org/10.1109/TVCG.2013.42.

    Article  PubMed  Google Scholar 

  • Benford, S., Greenhalgh, C., Rodden, T., & Pycock, J. (2001). Collaborative virtual environments. Communications of the ACM,44(7), 79–85.

    Article  Google Scholar 

  • Bennett, A. E., Miller, J. S., Stollon, N., Prasad, R., & Blum, N. J. (2018). Autism spectrum disorder and transition-aged youth. Current Psychiatry Reports,20, 103.

    Article  Google Scholar 

  • Cauell, J., Bickmore, T., Campbell, L., & Vilhjálmsson, H. (2000). Designing embodied conversational agents. In J. Cassell, J. Sullivan, S. Prevost, & E. Churchi (Eds.), Embodied conversational agents (pp. 29–63). Cambridge: The MIT Press.

    Google Scholar 

  • Cheng, Y., Chiang, H.-C., Ye, J., & Cheng, L.-H. (2010). Enhancing empathy instruction using a collaborative virtual learning environment for children with autistic spectrum conditions. Computers & Education,55(4), 1449–1458.

    Article  Google Scholar 

  • Clarke, E. M., Emerson, E. A., & Sistla, A. P. (1986). Automatic verification of finite-state concurrent systems using temporal logic specifications. ACM Transactions on Programming Languages and Systems,8(2), 244–263. https://doi.org/10.1145/5397.5399.

    Article  Google Scholar 

  • Connor, A., Sung, C., Strain, Z., Zeng, S., & Fabrizi, S. (2019). Building skills, confidence, and wellness: Psychosocial effects of soft skills training for young adults with autism. Journal of Autism and Developmental Disorders. https://doi.org/10.1007/s10803-019-03962-w.

    Article  PubMed  Google Scholar 

  • Constantino, J. N., & Gruber, C. P. (2002). The social responsiveness scale. Los Angeles: Western Psychological Services.

    Google Scholar 

  • Curtis, D. D., & Lawson, M. J. (2001). Exploring collaborative online learning. Journal of Asynchronous Learning Networks,5(1), 21–34.

    Google Scholar 

  • Desmarais, M. C., & Baker, R. S. (2012). A review of recent advances in learner and skill modeling in intelligent learning environments. User Modeling and User-Adapted Interaction,22(1–2), 9–38.

    Article  Google Scholar 

  • Dowd, A. M., McGinley, J. L., Taffe, J. R., & Rinehart, N. J. (2012). Do planning and visual integration difficulties underpin motor dysfunction in autism? A kinematic study of young children with autism. Journal of Autism and Developmental Disorders,42(8), 1539–1548.

    Article  Google Scholar 

  • Fulceri, F., Grossi, E., Contaldo, A., Narzisi, A., Apicella, F., Parrini, I.,… & Muratori, F. (2018). Motor skills as moderators of core symptoms in Autism Spectrum Disorders: Preliminary data from an exploratory analysis with Artificial Neural Networks. Frontiers in Psychology, 9, 2683.

  • Gogoulou, A., Gouli, E., & Grigoriadou, M. (2008). Adapting and personalizing the communication in a synchronous communication tool. Journal of Computer Assisted Learning,24(3), 203–216.

    Article  Google Scholar 

  • Goodwin, M. S. (2008). Enhancing and accelerating the pace of autism research and treatment the promise of developing innovative technology. Focus on Autism and Other Developmental Disabilities,23(2), 125–128. https://doi.org/10.1177/1088357608316678.

    Article  Google Scholar 

  • Hourcade, J. P., Williams, S. R., Miller, E. A., Huebner, K. E., & Liang, L. J. (2013). Evaluation of tablet apps to encourage social interaction in children with autism spectrum disorders. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 3197–3206).

  • Josman, N., Ben-Chaim, H. M., Friedrich, S., & Weiss, P. L. (2008). Effectiveness of virtual reality for teaching street-crossing skills to children and adolescents with autism. International Journal on Disability and Human Development,7(1), 49–56.

    Article  Google Scholar 

  • Kandalaft, M. R., Didehbani, N., Krawczyk, D. C., Allen, T. T., & Chapman, S. B. (2013). Virtual reality social cognition training for young adults with high-functioning autism. Journal of Autism and Developmental Disorders,43(1), 34–44. https://doi.org/10.1007/s10803-012-1544-6.

    Article  PubMed  Google Scholar 

  • Kaur, M., Srinivasan, S. M., & Bhat, A. N. (2018). Comparing motor performance, praxis, coordination, and interpersonal synchrony between children with and without Autism Spectrum Disorder (ASD). Research in Developmental Disabilities,72, 79–95.

    Article  Google Scholar 

  • Klakow, D., & Peters, J. (2002). Testing the correlation of word error rate and perplexity. Speech Communication,38(1–2), 19–28.

    Article  Google Scholar 

  • Kopp, S., Gesellensetter, L., Krämer, N. C., & Wachsmuth, I. (2005). A conversational agent as museum guide–design and evaluation of a real-world application. Paper presented at the International Workshop on Intelligent Virtual Agents.

  • Krishnaiah, P. R. (1980). Handbook of statistics (Vol. 31). New Delhi: Motilal Banarsidass Publishe.

    Google Scholar 

  • Kumar, R., Rosé, C. P., Wang, Y.-C., Joshi, M., & Robinson, A. (2007). Tutorial dialogue as adaptive collaborative learning support. Frontiers in Artificial Intelligence and Applications,158, 383.

    Google Scholar 

  • Lahiri, U., Bekele, E., Dohrmann, E., Warren, Z., & Sarkar, N. (2013). Design of a virtual reality based adaptive response technology for children with autism. IEEE Transactions on Neural Systems and Rehabilitation Engineering,21(1), 55–64. https://doi.org/10.1109/TNSRE.2012.2218618.

    Article  PubMed  Google Scholar 

  • Laugeson, E. A., Ellingsen, R., Sanderson, J., Tucci, L., & Bates, S. (2014). The ABC’s of teaching social skills to adolescents with autism spectrum disorder in the classroom: The UCLA PEERS® program. Journal of Autism and Developmental Disorders,44(9), 2244–2256.

    Article  Google Scholar 

  • Laugeson, E. A., Frankel, F., Gantman, A., Dillon, A. R., & Mogil, C. (2012). Evidence-based social skills training for adolescents with autism spectrum disorders: The UCLA PEERS program. Journal of Autism and Developmental Disorders,42(6), 1025–1036.

    Article  Google Scholar 

  • Leman, P. J. (2015). How do groups work? Age differences in performance and the social outcomes of peer collaboration. Cognitive science,39(4), 804–820.

    Article  Google Scholar 

  • Linton, F., Goodman, B., Gaimari, R., Zarrella, J., & Ross, H. (2003). Student modeling for an intelligent agent in a collaborative learning environment. Paper presented at the International Conference on User Modeling.

  • Lord, C., Risi, S., Lambrecht, L., Cook, E. H., Leventhal, B. L., DiLavore, P. C., … Rutter, M. (2000). The Autism Diagnostic Observation Schedule-Generic: A standard measure of social and communication deficits associated with the spectrum of autism. Journal of Autism and Developmental Disorders, 30(3), 205–223. https://doi.org/10.1023/a:1005592401947.

  • McManus, M. M., & Aiken, R. M. (1995). Monitoring computer-based collaborative problem solving. Journal of Interactive Learning Research,6(4), 307.

    Google Scholar 

  • McPheeters, M. L., Warren, Z., Sathe, N., Bruzek, J. L., Krishnaswami, S., Jerome, R. N., et al. (2011). A systematic review of medical treatments for children with autism spectrum disorders. Pediatrics,127(5), e1312–e1321. https://doi.org/10.1542/peds.2011-0427.

    Article  PubMed  Google Scholar 

  • Millen, L., Hawkins, T., Cobb, S., Zancanaro, M., Glover, T., Weiss, P. L., & Gal, E. (2011). Collaborative technologies for children with autism. Paper presented at the Proceedings of the 10th International Conference on Interaction Design and Children.

  • Mitchell, P., Parsons, S., & Leonard, A. (2007). Using virtual environments for teaching social understanding to 6 adolescents with autistic spectrum disorders. Journal of Autism and Developmental Disorders,37(3), 589–600. https://doi.org/10.1007/s10803-006-0189-8.

    Article  PubMed  Google Scholar 

  • Nabeth, T., Razmerita, L., Angehrn, A., & Roda, C. (2005). InCA: A cognitive multi-agents architecture for designing intelligent & adaptive learning systems. Computer Science and Information Systems,2(2), 99–114.

    Article  Google Scholar 

  • Ohl, A., Grice Sheff, M., Small, S., Nguyen, J., Paskor, K., & Zanjirian, A. (2017). Predictors of employment status among adults with Autism Spectrum Disorder. Work,56(2), 345–355.

    Article  Google Scholar 

  • Owen-DeSchryver, J. S., Carr, E. G., Cale, S. I., & Blakeley-Smith, A. (2008). Promoting social interactions between students with autism spectrum disorders and their peers in inclusive school settings. Focus on Autism and Other Developmental Disabilities,23(1), 15–28.

    Article  Google Scholar 

  • Parsons, L., Cordier, R., Munro, N., & Joosten, A. (2018). The feasibility and appropriateness of a peer-to-peer, play-based intervention for improving pragmatic language in children with autism spectrum disorder. International Journal of Speech-Language Pathology,2, 1–13.

    Google Scholar 

  • Parsons, L., Cordier, R., Munro, N., Joosten, A., & Speyer, R. (2017). A systematic review of pragmatic language interventions for children with autism spectrum disorder. PLoS ONE,12(4), e0172242.

    Article  Google Scholar 

  • Parsons, S., & Mitchell, P. (2002). The potential of virtual reality in social skills training for people with autistic spectrum disorders. Journal of Intellectual Disability Research,46(Pt 5), 430–443.

    Article  Google Scholar 

  • Rutter, M., Bailey, A., & Lord, C. (2003). The social communication questionnaire: Manual. Los Angeles: Western Psychological Services.

    Google Scholar 

  • Samad, M. D., Diawara, N., Bobzien, J. L., Harrington, J. W., Witherow, M. A., & Iftekharuddin, K. M. (2017). A feasibility study of autism behavioral markers in spontaneous facial, visual, and hand movement response data. IEEE Transactions on Neural Systems and Rehabilitation Engineering,26(2), 353–361.

    Article  Google Scholar 

  • Scahill, L., Aman, M. G., Lecavalier, L., Halladay, A. K., Bishop, S. L., Bodfish, J. W., … Dawson, G. (2015). Measuring repetitive behaviors as a treatment endpoint in youth with autism spectrum disorder. Autism, 19(1), 38–52. https://doi.org/10.1177/1362361313510069.

    Article  Google Scholar 

  • Scheuer, O., Loll, F., Pinkwart, N., & McLaren, B. M. (2010). Computer-supported argumentation: A review of the state of the art. International Journal of Computer-Supported Collaborative Learning,5(1), 43–102.

    Article  Google Scholar 

  • Schmidt, M., Laffey, J. M., Schmidt, C. T., Wang, X., & Stichter, J. (2012). Developing methods for understanding social behavior in a 3D virtual learning environment. Computers in Human Behavior,28(2), 405–413.

    Article  Google Scholar 

  • Schmidt, M., Laffey, J., & Stichter, J. (2011). Virtual social competence instruction for individuals with autism spectrum disorders: Beyond the single-user experience. Paper presented at the Proceedings of CSCL.

  • Srinivasan, S., & Petkovic, D. (2000). Phonetic confusion matrix based spoken document retrieval. Paper presented at the Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval.

  • Stolcke, A., Coccaro, N., Bates, R., Taylor, P., Van Ess-Dykema, C., Ries, K., … Meteer, M. (2000). Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational Linguistics, 26(3), 339-373.

    Article  Google Scholar 

  • Van Rosmalen, P., Brouns, F., Tattersall, C., Vogten, H., Bruggen, J., Sloep, P., et al. (2005). Towards an open framework for adaptive, agent-supported e-learning. International Journal of Continuing Engineering Education and Life Long Learning,15(3–6), 261–275.

    Article  Google Scholar 

  • Veenstra-VanderWeele, J., & Warren, Z. (2015). Intervention in the context of development: Pathways toward new treatments. Neuropsychopharmacology,40(1), 225–237. https://doi.org/10.1038/npp.2014.232.

    Article  PubMed  Google Scholar 

  • Vernon, T. W., Miller, A. R., Ko, J. A., Barrett, A. C., & McGarry, E. S. (2018). A randomized controlled trial of the Social Tools And Rules for Teens (START) program: An immersive socialization intervention for adolescents with autism spectrum disorder. Journal of Autism and Developmental Disorders,48(3), 892–904.

    Article  Google Scholar 

  • Vieira, A. C., Teixeira, L., Timóteo, A., Tedesco, P., & Barros, F. (2004). Analyzing on-line collaborative dialogues: The oxentchê–chat. Paper presented at the International Conference on Intelligent Tutoring Systems.

  • Walker, E., Rummel, N., & Koedinger, K. R. (2014). Adaptive intelligent support to improve peer tutoring in algebra. International Journal of Artificial Intelligence in Education,24(1), 33–61.

    Article  Google Scholar 

  • Wallace, S., Parsons, S., & Bailey, A. (2015). Self-reported sense of presence and responses to social stimuli by adolescents with ASD in a collaborative virtual reality environment. Journal of Intellectual and Developmental Disability,42(2), 131–141.

    Article  Google Scholar 

  • Weitlauf, A. S., McPheeters, M. L., Peters, B., Sathe, N., Travis, R., Aiello, R., … Warren, Z. (2014). Therapies for Children With Autism Spectrum Disorder: Behavioral Interventions Update. Rockville: Agency for Healthcare Research and Quality.

  • Wilson, G. F., & Russell, C. A. (2007). Performance enhancement in an uninhabited air vehicle task using psychophysiologically determined adaptive aiding. Human Factors, 49(6), 1005–1018.

    Article  Google Scholar 

  • Zancanaro, M., Pianesi, F., Stock, O., Venuti, P., Cappelletti, A., Iandolo, G., … Rossi, F. (2007). Children in the museum: an environment for collaborative storytelling. In O. Stock, M. Zancanaro (Eds.), PEACH-intelligent interfaces for museum visits (pp. 165–184). Berlin: Springer.

Download references

Funding

This project was funded by the National Institutes of Health (1R21MH111548-01).

Author information

Authors and Affiliations

  1. Department of Electrical Engineering and Computer Science, Vanderbilt University, Nashville, TN, USA

    Lian Zhang, Ashwaq Zaini Amat & Nilanjan Sarkar

  2. Department of Pediatrics, Vanderbilt University Medical Center, Nashville, USA

    Amy S. Weitlauf & Zachary E. Warren

  3. Vanderbilt Kennedy Center, Treatment and Research Institute of Autism Spectrum Disorders, Vanderbilt University Medical Center, 230 Appleton Pl., PMB 74, Nashville, TN, 37212, USA

    Amy S. Weitlauf, Amy Swanson & Zachary E. Warren

  4. Department of Special Education, Vanderbilt University, Nashville, TN, USA

    Zachary E. Warren

  5. Department of Psychiatry, Vanderbilt University Medical Center, Nashville, TN, USA

    Zachary E. Warren

  6. Department of Mechanical Engineering, Vanderbilt University, Nashville, TN, USA

    Nilanjan Sarkar

Authors
  1. Lian Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Amy S. Weitlauf
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ashwaq Zaini Amat
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Amy Swanson
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zachary E. Warren
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Nilanjan Sarkar
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

LZ, AW, ZW, and NS conceived of the study and crafted the experimental design. LZ, NS, and ZW helped design the CVE procedure. LZ, ZW, and AS provided oversight of study implementation across Vanderbilt sites. LZ, AA, and AS assisted with data collection and analysis for manuscript preparation. LZ, AW, AA, ZW, and NS significantly participated in drafting the article, revising it critically, and providing final approval of the manuscript. All authors are in agreement with accountability for all aspects of the work.

Corresponding author

Correspondence to Amy S. Weitlauf.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Ethical Approval

All procedures performed in studies involving human subjects were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, L., Weitlauf, A.S., Amat, A.Z. et al. Assessing Social Communication and Collaboration in Autism Spectrum Disorder Using Intelligent Collaborative Virtual Environments. J Autism Dev Disord 50, 199–211 (2020). https://doi.org/10.1007/s10803-019-04246-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10803-019-04246-z

Keywords

Search

Navigation