Jiaxiong Hu
ABSTRACT
The perceived emotional intelligence of a conversational agent (CA) can significantly impact people’s interaction with the CA. Prior research applies text-based sentiment analysis and emotional response generation to improve CAs’ emotional intelligence. However, acoustic features in speech containing rich contexts are underexploited. In this work, we designed and implemented an emotionally aware CA, called HUE (Heard yoUr Emotion) that stylized responses with emotion regulation strategies and empathetic interjections. We conducted a user study with 75 participants to evaluate their perceived emotional intelligence (PEI) of HUE by having them observe conversations between people and HUE in different emotional scenarios. Our results show that participants’ PEI was significantly higher with the acoustic features than without.
- Samuel Albanie, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. 2018. Emotion Recognition in Speech Using Cross-Modal Transfer in the Wild. In Proceedings of the 26th ACM International Conference on Multimedia (Seoul, Republic of Korea) (MM ’18). Association for Computing Machinery, New York, NY, USA, 292–301. https://doi.org/10.1145/3240508.3240578Google Scholar
Digital Library
- T. W. Bickmore, R. Fernando, L. Ring, and D. Schulman. 2010. Empathic Touch by Relational Agents. IEEE Transactions on Affective Computing 1, 1 (2010), 60–71. https://doi.org/10.1109/T-AFFC.2010.4Google ScholarDigital Library
- Ana Paula Chaves and Marco Aurélio Gerosa. 2019. How should my chatbot interact? A survey on human-chatbot interaction design. CoRR abs/1904.02743(2019). arxiv:1904.02743Google Scholar
- Huimin Chen, Maosong Sun, Cunchao Tu, Yankai Lin, and Zhiyuan Liu. 2016. Neural sentiment classification with user and product attention. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 1650–1659.Google ScholarCross Ref
- Shizhe Chen, Qin Jin, Jinming Zhao, and Shuai Wang. 2017. Multimodal multi-task learning for dimensional and continuous emotion recognition. In Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge. ACM, 19–26.Google ScholarDigital Library
- Michelle Cohn, Chun-Yen Chen, and Zhou Yu. 2019. A Large-Scale User Study of an Alexa Prize Chatbot: Effect of TTS Dynamism on Perceived Quality of Social Dialog. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue. 293–306.Google ScholarCross Ref
- Michelle G Craske, Linda Street, and David H Barlow. 1989. Instructions to focus upon or distract from internal cues during exposure treatment of agoraphobic avoidance. Behaviour research and therapy 27, 6 (1989), 663–672.Google Scholar
- Kohji Dohsaka, Ryota Asai, Ryuichiro Higashinaka, Yasuhiro Minami, and Eisaku Maeda. 2009. Effects of Conversational Agents on Human Communication in Thought-Evoking Multi-Party Dialogues. In Proceedings of the SIGDIAL 2009 Conference: The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue(London, United Kingdom) (SIGDIAL ’09). Association for Computational Linguistics, USA, 217–224.Google ScholarDigital Library
- Martina Drescher. 1997. French interjections and their use in discourse. The Language of Emotions(1997), 233–246.Google Scholar
- Kathleen Kara Fitzpatrick, Alison Darcy, and Molly Vierhile. 2017. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial. JMIR mental health 4, 2 (2017), e19.Google Scholar
- Barbara L. Fredrickson. 2013. Positive Emotions Broaden and Build. Advances in Experimental Social Psychology, Vol. 47. Academic Press, 1 – 53. https://doi.org/10.1016/B978-0-12-407236-7.00001-2Google ScholarCross Ref
- Béatrice S Hasler, Gilad Hirschberger, Tal Shani-Sherman, and Doron A Friedman. 2014. Virtual peacemakers: Mimicry increases empathy in simulated contact with virtual outgroup members. Cyberpsychology, Behavior, and Social Networking 17, 12(2014), 766–771.Google ScholarCross Ref
- Tianran Hu, Anbang Xu, Zhe Liu, Quanzeng You, Yufan Guo, Vibha Sinha, Jiebo Luo, and Rama Akkiraju. 2018. Touch Your Heart: A Tone-Aware Chatbot for Customer Care on Social Media. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3173574.3173989Google ScholarDigital Library
- Jing Huang, Qi Li, Yuanyuan Xue, Taoran Cheng, Shuangqing Xu, Jia Jia, and Ling Feng. 2015. Teenchat: a chatterbot system for sensing and releasing adolescents’ stress. In International Conference on Health Information Science. Springer, 133–145.Google ScholarCross Ref
- Mirjana Ivanović, Miloš Radovanović, Zoran Budimac, Dejan Mitrović, Vladimir Kurbalija, Weihui Dai, and Weidong Zhao. 2014. Emotional Intelligence and Agents: Survey and Possible Applications. In International Conference on Web Intelligence.Google ScholarDigital Library
- Tom Johnstone and Klaus R Scherer. 1999. The effects of emotions on voice quality. In Proceedings of the XIVth international congress of phonetic sciences. Citeseer, 2029–2032.Google Scholar
- Philipp Kanske, Janine Heissler, Sandra Schönfelder, André Bongers, and Michele Wessa. 2011. How to regulate emotion? Neural networks for reappraisal and distraction. Cerebral Cortex 21, 6 (2011), 1379–1388.Google ScholarCross Ref
- Soomin Kim, Joonhwan Lee, and Gahgene Gweon. 2019. Comparing data from chatbot and web surveys: Effects of platform and conversational style on survey response quality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.Google ScholarDigital Library
- Yi-Chieh Lee, Naomi Yamashita, Yun Huang, and Wai Fu. 2020. “I Hear You, I Feel You”: Encouraging Deep Self-Disclosure through a Chatbot. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376175Google ScholarDigital Library
- Q Vera Liao, Muhammed Mas-ud Hussain, Praveen Chandar, Matthew Davis, Yasaman Khazaeni, Marco Patricio Crasso, Dakuo Wang, Michael Muller, N Sadat Shami, and Werner Geyer. 2018. All Work and No Play?. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–13.Google ScholarDigital Library
- Xiaojuan Ma, Emily Yang, and Pascale Fung. 2019. Exploring Perceived Emotional Intelligence of Personality-Driven Virtual Agents in Handling User Challenges. In The World Wide Web Conference. ACM, 1222–1233.Google Scholar
- Seyedmahdad Mirsamadi, Emad Barsoum, and Cha Zhang. 2017. Automatic speech emotion recognition using recurrent neural networks with local attention. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2227–2231.Google ScholarDigital Library
- Christos N Moridis and Anastasios A Economides. 2012. Affective learning: Empathetic agents with emotional facial and tone of voice expressions. IEEE Transactions on Affective Computing 3, 3 (2012), 260–272.Google ScholarDigital Library
- Jonathan Mumm and Bilge Mutlu. 2011. Designing motivational agents: The role of praise, social comparison, and embodiment in computer feedback. Computers in Human Behavior 27, 5 (2011), 1643–1650.Google ScholarCross Ref
- Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. 2017. VoxCeleb: A Large-Scale Speaker Identification Dataset. In Proc. Interspeech 2017. 2616–2620. https://doi.org/10.21437/Interspeech.2017-950Google ScholarCross Ref
- John B Nezlek, Kristof Vansteelandt, Iven Van Mechelen, and Peter Kuppens. 2008. Appraisal-emotion relationships in daily life.Emotion 8, 1 (2008), 145.Google Scholar
- A.I. Niculescu, S.S. Ge, Elisabeth M.A.G. van Dijk, Antinus Nijholt, Haizhou Li, and Swan Lan See. 2013. Making social robots more attractive: the effects of voice pitch, humor and empathy. International journal of social robotics 5, 2 (21 4 2013), 171–191. https://doi.org/10.1007/s12369-012-0171-x eemcs-eprint-22397.Google ScholarCross Ref
- Kevin N Ochsner and James J Gross. 2005. The cognitive control of emotion. Trends in cognitive sciences 9, 5 (2005), 242–249.Google Scholar
- Kevin N Ochsner and James J Gross. 2008. Cognitive emotion regulation: Insights from social cognitive and affective neuroscience. Current directions in psychological science 17, 2 (2008), 153–158.Google ScholarCross Ref
- Qiao Qian, Minlie Huang, Jinhao Lei, and Xiaoyan Zhu. 2017. Linguistically Regularized LSTM for Sentiment Classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, Canada, 1679–1689. https://doi.org/10.18653/v1/P17-1154Google ScholarCross Ref
- Peter Salovey and John D Mayer. 1990. Emotional intelligence. Imagination, cognition and personality 9, 3 (1990), 185–211.Google Scholar
- Peter Ed Salovey and David J Sluyter. 1997. Emotional development and emotional intelligence: Educational implications.Basic Books.Google Scholar
- Heung-Yeung Shum, Xiao-dong He, and Di Li. 2018. From Eliza to XiaoIce: challenges and opportunities with social chatbots. Frontiers of Information Technology & Electronic Engineering 19, 1(2018), 10–26.Google ScholarCross Ref
- Zhenqiao Song, Xiaoqing Zheng, Lu Liu, Mu Xu, and Xuan-Jing Huang. 2019. Generating Responses with a Specific Emotion in Dialog. In Proceedings of the 57th Conference of the Association for Computational Linguistics. 3685–3695.Google ScholarCross Ref
- Carlos Toxtli, Andrés Monroy-Hernández, and Justin Cranshaw. 2018. Understanding chatbot-mediated task management. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–6.Google ScholarDigital Library
- Jeng-Yi Tzeng and Cheng-Te Chen. 2012. Computer praise, attributional orientations, and games: A reexamination of the CASA theory relative to children. Computers in Human Behavior 28, 6 (2012), 2420–2430.Google ScholarDigital Library
- Justin D Weisz, Mohit Jain, Narendra Nath Joshi, James Johnson, and Ingrid Lange. 2019. BigBlueBot: teaching strategies for successful human-agent interactions. In Proceedings of the 24th International Conference on Intelligent User Interfaces. 448–459.Google ScholarDigital Library
- Wierzbicka and Anna. 1992. The semantics of interjection. Journal of Pragmatics 18, 2-3 (1992), 159–192.Google Scholar
- Anna Wierzbicka. 1999. Emotions across Languages and Cultures: Diversity and Universals. Cambridge University Press. https://doi.org/10.1017/CBO9780511521256Google ScholarCross Ref
- Carl E Williams and Kenneth N Stevens. 1972. Emotions and speech: Some acoustical correlates. The Journal of the Acoustical Society of America 52, 4B (1972), 1238–1250.Google ScholarCross Ref
- Emily C Willroth, Jayde AM Flett, and Iris B Mauss. 2020. Depressive symptoms and deficits in stress-reactive negative, positive, and within-emotion-category differentiation: A daily diary study. Journal of personality 88, 2 (2020), 174–184.Google ScholarCross Ref
- Ziang Xiao, Michelle X. Zhou, Q. Vera Liao, Gloria Mark, Changyan Chi, Wenxi Chen, and Huahai Yang. 2020. Tell Me About Yourself: Using an AI-Powered Chatbot to Conduct Conversational Surveys with Open-Ended Questions. ACM Trans. Comput.-Hum. Interact. 27, 3, Article 15 (June 2020), 37 pages. https://doi.org/10.1145/3381804Google ScholarDigital Library
- Xi Yang, Marco Aurisicchio, and Weston Baxter. 2019. Understanding Affective Experiences with Conversational Agents. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300772Google ScholarDigital Library
- Yang Yang, Xiaojuan Ma, and Pascale Fung. 2017. Perceived emotional intelligence in virtual agents. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 2255–2262.Google ScholarDigital Library
- Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. 2018. Emotional chatting machine: Emotional conversation generation with internal and external memory. In Thirty-Second AAAI Conference on Artificial Intelligence.Google ScholarCross Ref
- Suping Zhou, Jia Jia, Qi Wang, Yufei Dong, Yufeng Yin, and Kehua Lei. 2018. Inferring emotion from conversational voice data: A semi-supervised multi-path generative neural network approach. In Thirty-Second AAAI Conference on Artificial Intelligence.Google ScholarCross Ref
Index Terms
- Enhancing the Perceived Emotional Intelligence of Conversational Agents through Acoustic Cues
Index terms have been assigned to the content through auto-classification.
Recommendations
Investigating Acoustic Cues of Emotional Valence in Mandarin Speech Prosody - A Corpus Approach
Chinese Lexical SemanticsAbstractThe impact of emotion on prosody in the context of speech communication has yielded inconclusive results when it comes to the prosodic patterns associated with high-arousal emotions of different emotional valences, such as “Happy” and “Anger”. To ...
Enhancing Conversational Agents with Empathic Abilities
IVA '21: Proceedings of the 21st ACM International Conference on Intelligent Virtual AgentsConversational agents are getting increasingly popular and find applications in health and customer services. Conversations in these fields are often emotionally charged. It is, therefore, necessary to handle the conversation with some degree of empathy ...
Bots with Feelings: Should AI Agents Express Positive Emotion in Customer Service?
The rise of emotional intelligence technology and the recent debate about the possibility of a “sentient” artificial intelligence (AI) urge the need to study the role of emotion during people’s interactions with AIs. In customer service, human employees ...
Customer service employees are generally advised to express positive emotion during their interactions with customers. The rise and maturity of artificial intelligence (AI)–powered conversational agents, also known as chatbots, beg the question: should AI ...
Comments