Skip to main content
SpringerLink
Log in
Book cover

The Conversational Interface pp 329–357Cite as

Affective Conversational Interfaces

  • Chapter
  • First Online:

Abstract

In order to build artificial conversational interfaces that display behaviors that are credible and expressive, we should endow them with the capability to recognize, adapt to, and render emotion. In this chapter, we explain how the recognition of emotional aspects is managed within conversational interfaces, including modeling and representation, emotion recognition from physiological signals, acoustics, text, facial expressions, and gestures and how emotion synthesis is managed through expressive speech and multimodal embodied agents. We also cover the main open tools and databases available for developers wishing to incorporate emotion into their conversational interfaces.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    https://www.w3.org/TR/emotionml/. Accessed February 27, 2016.

  2. 2.

    http://www.w3.org/TR/emotion-voc/. Accessed March 1, 2016.

  3. 3.

    https://www.informatik.uni-augsburg.de/de/lehrstuehle/hcm/projects/tools/aubt/. Accessed February 27, 2016.

  4. 4.

    http://compare.openaudio.eu/. Accessed February 27, 2016.

  5. 5.

    http://www.fon.hum.uva.nl/praat/. Accessed February 27, 2016.

  6. 6.

    https://uk.groups.yahoo.com/neo/groups/praat-users. Accessed February 27, 2016.

  7. 7.

    http://stackoverflow.com/questions/tagged/praat. Accessed February 27, 2016.

  8. 8.

    http://hcm-lab.de/projects/ssi/. Accessed February 27, 2016.

  9. 9.

    http://www.audeering.com/research/opensmile. Accessed February 27, 2016.

  10. 10.

    http://emotion-research.net/toolbox/toolbox_query_view?category=Database. Accessed February 27, 2016.

  11. 11.

    http://catalog.elra.info/. Accessed February 27, 2016.

  12. 12.

    http://www.nltk.org/. Accessed February 27, 2016.

  13. 13.

    https://opennlp.apache.org/. Accessed February 27, 2016.

  14. 14.

    http://sentiwordnet.isti.cnr.it/. Accessed February 27, 2016.

  15. 15.

    http://csea.phhp.ufl.edu/media.html#bottommedia. Accessed February 27, 2016.

  16. 16.

    https://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html. Accessed February 27, 2016.

  17. 17.

    http://www.wjh.harvard.edu/~inquirer/. Accessed February 27, 2016.

  18. 18.

    http://www3.nd.edu/~mcdonald/Word_Lists.html. Accessed February 27, 2016.

  19. 19.

    http://www.anvil-software.org/. Accessed February 27, 2016.

  20. 20.

    http://metashare.cst.dk/repository/browse/danish-first-encounters-nomco-corpus/6f4ee056444211e2b2e00050569b00003505d6478d484ae2b75b737aab697e99/. Accessed February 27, 2016.

  21. 21.

    http://www.pitt.edu/~emotion/ck-spread.htm. Accessed February 27, 2016.

  22. 22.

    http://mmifacedb.eu/. Accessed February 27, 2016.

  23. 23.

    http://emofilt.syntheticspeech.de/. Accessed February 27, 2016.

  24. 24.

    The example code shown is from the SEMAINE project: http://semaine.opendfki.de/wiki/FML. Accessed February 27, 2016.

  25. 25.

    http://secom.ru.is/fml/. Accessed February 27, 2016.

  26. 26.

    http://alma.dfki.de. Accessed February 27, 2016.

  27. 27.

    http://sourceforge.net/projects/fearnot/files/FAtiMA/FAtiMA/. Accessed February 27, 2016.

  28. 28.

    http://www.affective-sciences.org/content/exploring-your-ec. Accessed February 27, 2016.

  29. 29.

    http://sentiment.christopherpotts.net/. Accessed February 27, 2016.

References

  • Allwood J, Cerrato L, Jokinen K, Naravetta C, Paggio P (2008) The MUMIN coding scheme for the annotation of feedback, turn management and sequencing phenomena. Lang Resour Eval 41(3/4):273–287. doi:10.1007/s10579-007-9061-5

    Google Scholar 

  • Baccianella S, Esuli A, Sebastiani F (2010) SentiWordNet 3.0: an enhanced lexical resource for sentiment analysis and opinion mining. In: International conference on language resources and evaluation (LREC2010). European Language Resources Association (ELRA), Malta, 17–23 May 2010, pp 2200–2204

    Google Scholar 

  • Batliner A, Schuller B, Seppi D, Steidl S, Devilliers L, Vidrascu L, Vogt T, Aharonson V, Amir N (2011) The automatic recognition of emotions in speech. In: Cowie R, Pelachaud C, Petta P (eds) Emotion-oriented systems. Springer Berlin Heidelberg, Berlin, Heidelberg, pp 71–99. doi:10.1007/978-3-642-15184-2_6

    Google Scholar 

  • Ben Mimoun MS, Poncin I, Garnier M (2012) Case study—embodied virtual agents: an analysis on reasons for failure. J Retail Consum Serv 19(6):605–612. doi:10.1016/j.jretconser.2012.07.006

    Article  Google Scholar 

  • Boersma P, Weenink D (2016) Praat: doing phonetics by computer. http://www.fon.hum.uva.nl/praat/

  • Bos DO (2006) EEG-based emotion recognition; the influence of visual and auditory stimuli. http://hmi.ewi.utwente.nl/verslagen/capita-selecta/CS-Oude_Bos-Danny.pdf

  • Burkhardt F (2005) Emofilt: the simulation of emotional speech by prosody-transformation. In: Proceedings of the 9th European conference on speech communication and technology (Interspeech2005), Lisbon, Portugal, 4–8 Sept 2005, pp 509–512. http://www.isca-speech.org/archive/interspeech_2005/i05_0509.html

  • Cafaro A, Vilhjálmsson HH, Bickmore T, Heylen D, Pelachaud C(2014) Representing communicative functions in SAIBA with a unified function markup language. In: Bickmore T, Marsella S, Sidner C (eds) Intelligent virtual agents. Springer International Publishing, Switzerland, pp 81–94. doi:10.1007/978-3-319-09767-1_11

    Google Scholar 

  • Callejas Z, Griol D, López-Cózar R (2011) Predicting user mental states in spoken dialogue systems. EURASIP J Adv Signal Process 1:6. doi:10.1186/1687-6180-2011-6

    Article  Google Scholar 

  • Calvo RA, D’Mello S, Gratch J, Kappas A (eds) (2014) The Oxford handbook of affective computing, 1st edn. Oxford University Press, Oxford. doi:10.1093/oxfordhb/9780199942237.001.0001

    Google Scholar 

  • Castellano G, Leite I, Pereira A, Martinho C, Paiva A, McOwan PW (2010) Affect recognition for interactive companions: challenges and design in real world scenarios. J Multimodal User Interfaces 3(1–2):89–98. doi:10.1007/s12193-009-0033-5

    Article  Google Scholar 

  • Clavel C, Callejas Z (2016) Sentiment analysis: from opinion mining to human-agent interaction. IEEE Trans Affect Comput 7(1):74–93. doi:10.1109/TAFFC.2015.2444846

    Google Scholar 

  • Cowie R, Cornelius R (2003) Describing the emotional states that are expressed in speech. Speech Commun 40(1–2):5–32. doi:10.1016/S0167-6393(02)00071-7

    Article  MATH  Google Scholar 

  • Ekman P (1999) Basic emotions. In: Dalgleish T, Power MJ (eds) Handbook of cognition and emotion. Wiley, Chichester, pp 45–60. doi:10.1002/0470013494.ch3

    Google Scholar 

  • Ekman P (2003) Emotions revealed: recognizing faces and feelings to improve communication and emotional life, 1st edn. Times Books, New York

    Google Scholar 

  • Ekman P, Rosenberg EL (eds) (2005) What the face reveals: basic and applied studies of spontaneous expression using the facial action coding system (FACS), 2nd edn. Oxford University Press, Oxford. doi:10.1093/acprof:oso/9780195179644.001.0001

  • Eyben F, Weninger F, Gross F, Schuller B (2013) Recent developments in openSMILE, the munich open-source multimedia feature extractor. In: Proceedings of the 21st ACM international conference on multimedia (MM’13), Barcelona, Spain, 21–25 Oct 2013, pp 835–838. doi:10.1145/2502081.2502224

  • Feldman R (2013) Techniques and applications for sentiment analysis. Commun ACM 56(4):82. doi:10.1145/2436256.2436274

    Article  Google Scholar 

  • Govind D, Prasanna SRM (2012) Expressive speech synthesis: a review. IJST 16(2):237–260. doi:10.1007/s10772-012-9180-2

    Google Scholar 

  • Gratch J, Marsella S (eds) (2013) Social emotions in nature and artifact. Oxford University Press, Oxford 10.1093/acprof:oso/9780195387643.001.0001

    Google Scholar 

  • Haag A, Goronzy S, Schaich P, Williams J (2004) Emotion recognition using bio-sensors: first steps towards an automatic system. In: André E, Dybkjær L, Minker W, Heisterkamp P (eds) Affective dialogue systems. Springer Berlin Heidelberg, New York, pp 36–48. doi:10.1007/978-3-540-24842-2_4

    Google Scholar 

  • Jang E-H, Park B-J, Kim S-H, Chung M-A, Park M-S, Sohn J-H (2014) Emotion classification based on bio-signals emotion recognition using machine learning algorithms. In: Proceedings of 2014 international conference on information science, Electronics and Electrical Engineering (ISEEE), Sapporo, Japan, 26–28 April 2014, pp 104–109. doi:10.1109/InfoSEEE.2014.6946144

  • Jerritta S, Murugappan M, Nagarajan R, Wan K (2011) Physiological signals based human emotion recognition: a review. In: 2011 IEEE 7th international colloquium on signal processing and its applications (CSPA), Penang, Malaysia, 4–6 March 2011, pp 410–415. doi:10.1109/CSPA.2011.5759912

  • Kätsyri J, Förger K, Mäkäräinen M, Takala T (2015) A review of empirical evidence on different uncanny valley hypotheses: support for perceptual mismatch as one road to the valley of eeriness. Front Psychol 6:390. doi:10.3389/fpsyg.2015.00390

    Article  Google Scholar 

  • Kim J, André E (2008) Emotion recognition based on physiological changes in music listening. IEEE Trans Pattern Anal 30(12):2067–2083. doi:10.1109/TPAMI.2008.26

    Article  Google Scholar 

  • Kim KH, Bang SW, Kim SR (2004) Emotion recognition system using short-term monitoring of physiological signals. Med Biol Eng Comput 42(3):419–427. doi:10.1007/BF02344719

    Article  Google Scholar 

  • Kipp M (2012) ANVIL: a universal video research tool. In: Durand J, Gut U, Kristofferson G (eds) Handbook of corpus phonology. Oxford University Press, Oxford. doi:10.1093/oxfordhb/9780199571932.013.024

    Google Scholar 

  • Kopp S, Krenn B, Marsella S, Marshall AN, Pelachaud C, Pirker H, Thórisson KR, Vilhjálmsson H (2006) Towards a common framework for multimodal generation: the behavior markup language. In: Gratch J, Young M, Aylett R, Ballin D, Olivier P (eds) Intelligent virtual agents. Springer International Publishing, Switzerland, pp 205–217. doi:10.1007/11821830_17

    Google Scholar 

  • Liu B (2015) Sentiment analysis: mining opinions, sentiments, and emotions. Cambridge University Press, New York. doi:10.1017/CBO9781139084789

  • Marsella SC, Gratch J (2009) EMA: a process model of appraisal dynamics. Cogn Syst Res 10(1):70–90. doi:10.1016/j.cogsys.2008.03.005

    Article  Google Scholar 

  • Mathur MB, Reichling DB (2016) Navigating a social world with robot partners: a quantitative cartography of the Uncanny Valley. Cognition 146:22–32. doi:10.1016/j.cognition.2015.09.008

    Article  Google Scholar 

  • Nasoz F, Alvarez K, Lisetti CL, Finkelstein N (2003) Emotion recognition from physiological signals using wireless sensors for presence technologies. Cogn Technol Work 6(1):4–14. doi:10.1007/s10111-003-0143-x

    Article  Google Scholar 

  • Nass C, Lee KM (2000) Does computer-generated speech manifest personality? An experimental test of similarity-attraction. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI’00), The Hague, Netherlands, 1–6 April 2000, pp 329–336. doi:10.1145/332040.332452

  • Nass C, Yen C (2012) The man who lied to his laptop: what we can learn about ourselves from our machines. Penguin Group, New York

    Google Scholar 

  • Niewiadomski R, Hyniewska SJ, Pelachaud C (2013) Computational models of expressive behaviors for a virtual agent. In: Gratch J, Marsella S (eds) Social emotions in nature and artifact. Oxford University Press, Oxford, pp 143–161. doi:10.1093/acprof:oso/9780195387643.003.0010

    Google Scholar 

  • Paggio P, Navarretta C (2011) Head movements, facial expressions and feedback in danish first encounters interactions: a culture-specific analysis. In: Stephanidis C (ed) Universal access in human-computer interaction users diversity. Springer Berlin Heidelberg, New York, pp 583–590. doi:10.1007/978-3-642-21663-3_63

    Google Scholar 

  • Pantic M, Bartlett MS (2007) Machine analysis of facial expressions. In: Delac K, Grgic M (eds) Face recognition. I-Tech Education and Publishing, Vienna, Austria, pp 377–416. doi:10.5772/4847

    Google Scholar 

  • Pantic M, Valstar MF, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In: IEEE International conference on multimedia and expo (ICME), Amsterdam, The Netherlands, 6–8 July 2005, pp 317–321. doi:10.1109/ICME.2005.1521424

  • Pelachaud C (2009) Modelling multimodal expression of emotion in a virtual agent. Philos Trans R Soc B Biol Sci 364(1535):3539–3548. doi:10.1098/rstb.2009.0186

    Article  Google Scholar 

  • Petta P, Pelachaud C, Cowie R (eds) (2011) Emotion-oriented systems: the Humaine handbook. Springer, Berlin Heidelberg. doi:10.1007/978-3-642-15184-2

    Google Scholar 

  • Picard RW, Vyzas E, Healey J (2001) Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans Pattern Anal 23(10):1175–1191. doi:10.1109/34.954607

    Article  Google Scholar 

  • Polzin TS, Waibel A (2000) Emotion-sensitive human-computer interfaces. In: International speech communication association (ISCA) tutorial and research workshop on speech and emotion. Newcastle, Northern Ireland, UK, pp 201–206

    Google Scholar 

  • Ryan A, Cohn JF, Lucey S, Saragih J, Lucey P, De La Torre F, Rossi A (2009) Automated facial expression recognition system. In: 43rd annual international Carnahan conference on security technology, Zurich, Switzerland, 5–8 Oct 2009, pp 172–177. doi:10.1109/CCST.2009.5335546

  • Schröder M (2009) Expressive speech synthesis: past, present, and possible futures. In: Tao J, Tan T (eds) Affective information processing. Springer, London, pp 111–126. doi:10.1007/978-1-84800-306-4_7

  • Schuller B, Batliner A (2013) Computational paralinguistics: emotion, affect and personality in speech and language processing. Wiley, Chichester, UK. doi:10.1002/9781118706664

    Google Scholar 

  • Van Santen J, Mishra T, Klabbers E (2008) Prosodic processing. In: Benesty J, Sondhi MM, Huang Y (eds) Springer handbook of speech processing. Springer, Berlin Heidelberg, pp 471–488. doi:10.1007/978-3-540-49127-9_23

    Google Scholar 

  • Väyrynen E (2014) Emotion recognition from speech using prosodic features. Doctoral Dissertation, University of Oulu, Finland. http://urn.fi/urn:isbn:9789526204048

  • Ververidis D, Kotropoulos C (2006) Emotional speech recognition: resources, features and methods. Speech Commun 48(9):1162–1181. doi:10.1016/j.specom.2006.04.003

    Article  Google Scholar 

  • Wagner J, Lingenfelser F, Baur T, Damian I, Kistler F, André E (2013) The social signal interpretation (SSI) framework: multimodal signal processing and recognition in real-time. In: Proceedings of the 21st ACM international conference on Multimedia (MM’13), Barcelona, Spain, 21–25 Oct 2013, pp 831–834 doi:10.1145/2502081.2502223

  • Zhang C, Zhang Z (2010) A survey of recent advances in face detection. Microsoft TechReport MSR-TR-2010-66. http://research.microsoft.com/apps/pubs/default.aspx?id=132077

Download references

Author information

Authors and Affiliations

  1. School of Computing and Mathematics, Ulster University, Northern Ireland, UK

    Michael McTear

  2. ETSI Informática y Telecomunicación, University of Granada, Granada, Spain

    Zoraida Callejas

  3. Department of Computer Science, Universidad Carlos III de Madrid, Madrid, Spain

    David Griol

Authors
  1. Michael McTear
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zoraida Callejas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David Griol
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael McTear .

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this chapter

Cite this chapter

McTear, M., Callejas, Z., Griol, D. (2016). Affective Conversational Interfaces. In: The Conversational Interface. Springer, Cham. https://doi.org/10.1007/978-3-319-32967-3_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-32967-3_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-32965-9

  • Online ISBN: 978-3-319-32967-3

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation