Abstract
People with autism spectrum disorder (ASD) are known to show difficulties in the interpretation of human conversational facial expressions. With the recent advent of artificial intelligence, and more specifically, deep learning techniques, new possibilities arise in this context to support people with autism in the recognition of such expressions. This work aims at developing a deep neural network model capable of recognizing conversational facial expressions which are prone to misinterpretation in ASD. To that end, a publicly available dataset of conversational facial expressions is used to train various CNN-LSTM architectures. Training results are promising; however, the model shows limited generalization. Therefore, better conversational facial expressions datasets are required before attempting to build a full-fledged ASD-oriented support system.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
References
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017). https://doi.org/10.1145/3065386
Brain Power — Autism Education - Empowering Every Brain. https://brain-power.com/. Accessed 01 Mar 2021
Learn to Play Emotion Charades. https://youtu.be/lGoxUd2nTDc. Accessed 01 Apr 2021
Glass - Glass. https://www.google.com/glass/start/. Accessed 01 Apr 2021
Autism Glass Project. http://autismglass.stanford.edu/. Accessed 01 Apr 2021
Daniels, J., et al.: Feasibility testing of a wearable behavioral aid for social learning in children with autism. Appl. Clin. Inform. 9(1), 129–140 (2018). https://doi.org/10.1055/s-0038-1626727
Daniels, J., et al.: Exploratory study examining the at-home feasibility of a wearable tool for social-affective learning in children with autism. NPJ Digit. Med. 1(1), 32 (2018). https://doi.org/10.1038/s41746-018-0035-3
Voss, C., et al.: Effect of wearable digital intervention for improving socialization in children with autism spectrum disorder: a randomized clinical trial. JAMA Pediatr. 173(5), 446–454 (2019). https://doi.org/10.1001/jamapediatrics.2019.0285
Voss, C., et al.: Superpower glass: delivering unobtrusive eal-time social cues in wearable systems. In: UbiComp 2016 Adjunct - Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 1218–1226, September 2016. https://doi.org/10.1145/2968219.2968310
Washington, P., et al.: SuperpowerGlass: a wearable aid for the at-home therapy of children with autism. Proc. ACM Interactive Mobile Wearable Ubiquitous Technol. 1(3), 1–22 (2017). https://doi.org/10.1145/3130977
Gross, R.: Face databases. In: Li, S.Z., Jain, A.K. (eds.) Handbook of Face Recognition, pp. 301–327. Springer, New York (2005). https://doi.org/10.1007/0-387-27257-7_14
Ekman, P., et al.: What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford University Press, Oxford (1997)
Mollahosseini, A., Hasani, B., Mahoor, M.H.: AffectNet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput. 10(1), 18–31 (2019). https://doi.org/10.1109/TAFFC.2017.2740923
Goodfellow, I.J., et al.: Challenges in representation learning: a report on three machine learning contests. In: Lee, M., Hirose, A., Hou, Z.-G., Kil, R.M. (eds.) ICONIP 2013. LNCS, vol. 8228, pp. 117–124. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-42051-1_16
Dhall, A., Goecke, R., Joshi, J., Wagner, M., Gedeon, T.: Emotion recognition in the wild challenge 2013. In: Proceedings of the 15th ACM on International Conference on Multimodal Interaction, pp. 509–516 (2013). https://doi.org/10.1145/2522848.2531739
Kaulard, K., Cunningham, D.W., Bülthoff, H.H., Wallraven, C.: The MPI facial expression database - a validated database of emotional and conversational facial expressions. PLoS ONE 7(3) (2012). https://doi.org/10.1371/journal.pone.0032321
Elfenbein, H.A., Beaupré, M., Lévesque, M., Hess, U.: Toward a Dialect Theory: Cultural Differences in the Expression and Recognition of Posed Facial Expressions, psycnet.apa.org (2007). https://doi.org/10.1037/1528-3542.7.1.131
Li, Y.: Deep Learning of Human Emotion Recognition in Videos. https://uu.diva-portal.org/smash/get/diva2:1174434/FULLTEXT01.pdf. Accessed 01 Mar 2021
Cunningham, D.W., Nusseck, M., Wallraven, C., Bülthoff, H.H.: The role of image size in the recognition of conversational facial expressions. Comput. Animation Virtual Worlds 15(3–4), 305–310 (2004). https://doi.org/10.1002/cav.33
Rajan, S., Chenniappan, P., Devaraj, S., Madian, N.: Novel deep learning model for facial expression recognition based on maximum boosted CNN and LSTM. IET Image Process. 14(7), 1373–1381 (2020). https://doi.org/10.1049/iet-ipr.2019.1188
Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, CVPRW 2010, pp. 94–101 (2010). https://doi.org/10.1109/CVPRW.2010.5543262
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: MobileNetV2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 4510–4520, December 2018. https://doi.org/10.1109/CVPR.2018.00474
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Salgado, P., Banos, O., Villalonga, C. (2021). Facial Expression Interpretation in ASD Using Deep Learning. In: Rojas, I., Joya, G., Català , A. (eds) Advances in Computational Intelligence. IWANN 2021. Lecture Notes in Computer Science(), vol 12861. Springer, Cham. https://doi.org/10.1007/978-3-030-85030-2_27
Download citation
DOI: https://doi.org/10.1007/978-3-030-85030-2_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-85029-6
Online ISBN: 978-3-030-85030-2
eBook Packages: Computer ScienceComputer Science (R0)