Abstract
In this paper we investigate how natural language interfaces can be integrated with cars in a way such that their influence on driving performance is being minimized. In particular, we focus on how speech-based interaction can be supported through a visualization of the conversation. Our work is motivated by the fact that speech interfaces (like Alexa, Siri, Cortana, etc.) are increasingly finding their way into our everyday life. We expect such interfaces to become commonplace in vehicles in the future. Cars are a challenging environment, since speech interaction here is a secondary task that should not negatively affect the primary task, that is driving. At the outset of our work, we identify the design space for such interfaces. We then compare different visualization concepts in a driving simulator study with 64 participants. Our results yield that (1) text summaries support drivers in recalling information and enhances user experience but can also increase distraction, (2) the use of keywords minimizes cognitive load and influence on driving performance, and (3) the use of icons increases the attractiveness of the interface.
Similar content being viewed by others
Notes
https://www.apple.com/ios/carplay/, last access: 2017-08-12.
https://www.android.com/auto/, last access: 2017-08-12.
https://itunes.apple.com/de/app/remote-s-for-tesla/id991623777, last access: 2017-08-12.
https://azure.microsoft.com/de-de/services/cognitive-services/speech/,last access: 2017-05-14.
https://www.ibm.com/watson/developercloud/text-to-speech.html, last access: 2017-05-14.
http://www.ergoneers.com/eye-tracking/dikablis-glasses/, last access: 2017-05-13.
References
Asif A, Boll S (2010) Where to turn my car?: comparison of a tactile display and a conventional car navigation system under high load condition. In: Proceedings of the 2nd international conference on automotive user interfaces and interactive vehicular applications, Automotiveui ’10. ACM, New York, pp 64–71. https://doi.org/10.1145/1969773.1969786
Ballagas R, Borchers J, Rohs M, Sheridan JG (2006) The smart phone: a ubiquitous input device. IEEE Pervasive Comput 5(1):70–77. https://doi.org/10.1109/MPRV.2006.18
Beanland V, Fitzharris M, Young KL, Lenné MG (2013) Driver inattention and driver distraction in serious casualty crashes: Data from the australian national crash in-depth study. Accid Anal Prev 54:99–107. https://doi.org/10.1016/j.aap.2012.12.043 http://www.sciencedirect.com/science/article/pii/S000145751300047X
Bellegarda JR (2014) Spoken language understanding for natural interaction: the siri experience. In: Natural interaction with robots, knowbots and smartphones: putting spoken dialog systems into practice. Springer, New York, pp 3–14. https://doi.org/10.1007/978-1-4614-8280-2
Braun M, Pfleging B, Alt F (2018) A survey to understand emotional situations on the road and what they mean for affective automotive uis. Multimodal Technol. Interact 2(4). https://doi.org/10.3390/mti2040075 http://www.mdpi.com/2414-4088/2/4/75
Braun M, Pfleging B, Broy N, Alt F (2017) A design space for conversational in-vehicle information systems. In: Proceedings of the 19th international conference on human–computer interaction with mobile devices and services, MobileHCI ’17. ACM, New York
Braun M, Völkel ST, Wiegand G, Puls T, Steidl D, WeißD, Alt F (2018) The smile is the new like: controlling music with facial expressions to minimize driver distraction. In: Proceedings of the 17th international conference on mobile and ubiquitous multimedia, MUM ’18. ACM, New York. https://doi.org/10.1145/3282894.3289729
Bubb H (2003) Fahrerassistenz—primär ein beitrag zum komfort oder fuer die sicherheit? In: Der Fahrer im 21. Jahrhundert: Anforderungen, Anwendungen, Aspekte für Mensch-Maschine-Systeme; Tagung Braunschweig, 2. und 3. Juni 2003, VDI-Berichte 1768. VDI-Verlag, Düsseldorf, pp 25–44
Caird JK, Willness CR, Steel P, Scialfa C (2008) A meta-analysis of the effects of cell phones on driver performance. Accid Anal Prev 40(4):1282–1293
Commission of the European Communities (2007) Commission recommendation of 22 december 2006 on safe and efficient in-vehicle information and communication systems: Update of the european statement of principles on human machine interface (2007/78/ec). Off J Eur Union 50(L32): 200–241. http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=uriserv:OJ.L_.2007.032.01.0200.01.ENG
Cooper JM, Strayer DL (2015) Mental workload of voice interactions with 6 real-world driver interfaces. In: Proceedings of the eighth international driving symposium on human factors in driver assessment, training, and vehicle design, DA ’15, pp 2–8. http://drivingassessment.uiowa.edu/sites/default/files/DA2015/papers/002.pdf
Department of Transportation, National Highway Traffic Safety Administration (2013) Visual-manual nhtsa driver distraction guidelines for in-vehicle electronic device. Federal Regist 78(81), pp 24, 817–24, 890. https://federalregister.gov/a/2013-09883
Ehsani J, Li K, Simons-Morton BG (2015) Teenage drivers portable electronic device use while driving. In: Proceedings of the eighth international driving symposium on human factors in driver assessment, training, and vehicle design, DA ’15, pp 219–225. http://drivingassessment.uiowa.edu/sites/default/files/DA2015/papers/034.pdf
Engström J, Johansson E, Östlund J (2005) Effects of visual and cognitive load in real and simulated motorway driving. Transp Res Part F: Traffic Psychol Behav 8(2):97–120. https://doi.org/10.1016/j.trf.2005.04.012 http://www.sciencedirect.com/science/article/pii/S1369847805000185. The relationship between distraction and driving performance: towards a test regime for in-vehicle information systems In-vehicle information systems
Eyben F, Wöllmer M, Schuller B (2010) Opensmile: the munich versatile and fast open-source audio feature extractor. In: Proceedings of the 18th ACM international conference on Multimedia. ACM, pp 1459–1462
Foley JP, Young R, Angell L, Domeyer JE (2013) Towards operationalizing driver distraction. In: Proceedings of the seventh international driving symposium on human factors in driver assessment, training, and vehicle design, pp 57–63. http://drivingassessment.uiowa.edu/sites/default/files/DA2013/Papers/010_Foley_0.pdf
Geiger M, Zobl M, Bengler K, Lang M (2001) Intermodal differences in distraction effects while controlling automotive user interfaces. In: Proceedings of HCI international 2001 volume 1: usability evaluation and interface design: cognitive engineering, intelligent agents and virtual reality, HCII ’01. Lawrence Erlbaum, Hillsdale
Green P (2013) Standard definitions for driving measures and statistics: overview and status of recommended practice j2944. In: Proceedings of the 5th international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI ’13. ACM, New York, pp 184–191. https://doi.org/10.1145/2516540.2516542
Green PA (2008) Driver interface/hmi standards to minimize driver distraction/overload. In: Convergence 2008 conference proceedings. Society of Automotive Engineers, SAE International, Warrendale (2008)
Group ADFTW (2006) Statement of principles, criteria and verification procedures on driver interactions with advanced in- vehicle information and communication systems. Technical report, Alliance of Automobile Manufacturers
Hackenberg L, Bongartz S, Härtle C, Leiber P, Baumgarten T, Sison JA (2013) International evaluation of nlu benefits in the domain of in-vehicle speech dialog systems. In: Proceedings of the 5th international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI ’13. ACM, New York, pp 114–120. https://doi.org/10.1145/2516540.2516553
Haeuslschmid R, Pfleging B, Alt F (2016) A design space to support the development of windshield applications for the car. In: Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, pp 5076–5091
Harvey C, Stanton NA, Pickering CA, McDonald M, Zheng P (2011) In-vehicle information systems to meet the needs of drivers. Int J Hum Comp Interact 27(6):505–522. https://doi.org/10.1080/10447318.2011.555296
Hassenzahl M, Koller F, Burmester M (2008) Der user experience (ux) auf der spur: zum einsatz von www.attrakdiff.de. In: Brau H, Diefenbach S, Hassenzahl M, Koller F, Peissner M, Röse K (eds) Tagungsband UP08. Fraunhofer Verlag, Stuttgart, pp 78–82
Heenan A, Herdman CM, Brown MS, Robert N (2014) Effects of conversation on situation awareness and working memory in simulated driving. Hum Factors 56(6):1077–1092. https://doi.org/10.1177/0018720813519265 PMID: 25277018
Hernandez J, McDuff D, Benavides X, Amores J, Maes P, Picard R (2014) Autoemotive: bringing empathy to the driving experience to manage stress. In: Proceedings of the 2014 companion publication on designing interactive systems, DIS Companion ’14. ACM, New York, pp 53–56. https://doi.org/10.1145/2598784.2602780
ITFG on Driver Distraction (2013) Report on user interface requirements for automotive applications. Technical report, International Telecommunication Union
Jakus G, Dicke C, Sodnik J (2015) A user study of auditory, head-up and multi-modal displays in vehicles. Appl Ergon 46, Part A:184–192. https://doi.org/10.1016/j.apergo.2014.08.008 http://www.sciencedirect.com/science/article/pii/S0003687014001471
Kapoor A, Burleson W, Picard RW (2007) Automatic prediction of frustration. Int J Hum Comput Stud 65(8):724–736. https://doi.org/10.1016/j.ijhcs.2007.02.003 http://www.sciencedirect.com/science/article/pii/S1071581907000377
Kennington C, Kousidis S, Baumann T, Buschmeier H, Kopp S, Schlangen D (2014) Better driving and recall when in-car information presentation uses situationally-aware incremental speech output generation. In: Proceedings of the 6th international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI ’14. ACM, New York, pp 7:1–7:7. https://doi.org/10.1145/2667317.2667332
Kern D, Marshall P, Schmidt A (2010) Gazemarks: gaze-based visual placeholders to ease attention switching. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2093–2102. ACM
Kern D, Schmidt A (2009) Design space for driver-based automotive user interfaces. In: Proceedings of the 1st international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI ’09. ACM, New York, pp 3–10. https://doi.org/10.1145/1620509.1620511
Kim S, Dey AK (2009) Simulated augmented reality windshield display as a cognitive mapping aid for elder driver navigation. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI ’09. ACM, New York, pp 133–142. https://doi.org/10.1145/1518701.1518724
Kountouriotis GK, Wilkie RM, Gardner PH, Merat N (2015) Looking and thinking when driving: the impact of gaze and cognitive load on steering. Transp Res Part F: Traffic Psychol Behav 34:108–121. https://doi.org/10.1016/j.trf.2015.07.012 http://www.sciencedirect.com/science/article/pii/S1369847815001163
Laberge-Nadeau C, Maag U, Bellavance F, Lapierre SD, Desjardins D, Messier S, Saidi A (2003) Wireless telephones and the risk of road crashes. Accid Anal Prev 35(5):649–660
Lamble D, Kauranen T, Laakso M, Summala H (1999) Cognitive load and detection thresholds in car following situations: safety implications for using mobile (cellular) telephones while driving. Accid Anal Prev 31(6):617–623
Large DR, Burnett G, Anyasodo B, Skrypchuk L (2016) Assessing cognitive demand during natural language interactions with a digital driving assistant. In: Proceedings of the 8th international conference on automotive user interfaces and interactive vehicular applications, Automotive’UI 16. ACM, New York, pp 67–74. https://doi.org/10.1145/3003715.3005408
Lo VEW, Green PA (2013) Development and evaluation of automotive speech interfaces: Useful information from the human factors and the related literature. Int J Veh Technol 2013:924,170:1–924,170:13. https://doi.org/10.1155/2013/924170
Luger E, Sellen A (2016) “like having a really bad pa”: the gulf between user expectation and experience of conversational agents. In: Proceedings of the 2016 CHI conference on human factors in computing systems, CHI ’16. ACM, New York, pp 5286–5297. https://doi.org/10.1145/2858036.2858288
McEvoy SP, Stevenson MR, McCartt AT, Woodward M, Haworth C, Palamara P, Cercarelli R (2005) Role of mobile phones in motor vehicle crashes resulting in hospital attendance: a case-crossover study. Bmj 331(7514):428
Müller C, Weinberg G (2011) Multimodal input in the car, today and tomorrow. IEEE MultiMed 18(1):98–103. https://doi.org/10.1109/MMUL.2011.14
Nafari M, Weaver C (2013) Augmenting visualization with natural language translation of interaction: a usability study. Comput Graph Forum 32(3pt4):391–400. https://doi.org/10.1111/cgf.12126
Neßelrath R, Feld M (2013) Towards a cognitive load ready multimodal dialogue system for in-vehicle human-machine interaction. In: Adjunct proceedings of the 5th international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI Adjunct ’13, pp 49–52. http://www.auto-ui.org/13/docs/aui_adjunct_proceedings_final.pdf#page=49
Oh A, Fox H, Van Kleek M, Adler A, Gajos K, Morency LP, Darrell T (2002) Evaluating look-to-talk: a gaze-aware interface in a collaborative environment. In: CHI ’02 extended abstracts on human factors in computing systems, CHI EA ’02. ACM, New York, pp 650–651. https://doi.org/10.1145/506443.506528
Paas F, Tuovinen JE, Tabbers H, Van Gerven PWM (2003) Cognitive load measurement as a means to advance cognitive load theory. Educ Psychol 38(1):63–71. https://doi.org/10.1207/S15326985EP3801_8
Pauzié A (2008) A method to assess the driver mental workload: the driving activity load index (dali). Intell Transp Syst IET 2(4):315–322. https://doi.org/10.1049/iet-its:20080023 http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4706518&tag=1
Petzoldt T, Bellem H, Krems JF (2014) The critical tracking task: a potentially useful method to assess driver distraction? Hum Factors 56(4):789–808. https://doi.org/10.1177/0018720813501864
Pieraccini R, Dayanidhi K, Bloom J, Dahan JG, Phillips M, Goodman BR, Prasad KV (2004) Multimodal conversational systems for automobiles. Commun. ACM 47(1):47–49. https://doi.org/10.1145/962081.962104
Ranney TA, Garrott WR, Goodman MJ (2001) Nhtsa driver distraction research: Past, present, and future. In: Proceedings of the 17th international technical conference on the enhanced safety of vehicles. u.s. department of transportation, National Highway Traffic Safety Administration (2001). http://www-nrd.nhtsa.dot.gov/pdf/esv/esv17/proceed/00177.pdf
Redelmeier DA, Tibshirani RJ (1997) Association between cellular-telephone calls and motor vehicle collisions. N Engl J Med 336(7):453–458
Reimer B, Mehler B (2013) The effects of a production level “voice-command” interface on driver behavior: summary findings on reported workload, physiology, visual attention, and driving performance. Technical Report White Paper 2013-18A, MIT AgeLab—Massachusetts Institute of Technology (2013). http://agelab.mit.edu/files/MIT_AgeLab_White_Paper_2013-18A_(Voice_Interfaces).pdf
Rümelin S, Siegl P, Butz A (2013) Could you please... investigating cooperation in the car. In: Adjunct proceedings of the 5th international conference on automotive user interfaces and interactive vehicular applications, AutomotiveUI Adjunct ’13, pp 61–64. http://www.auto-ui.org/13/docs/aui_adjunct_proceedings_final.pdf#page=63
SAE International (2013) Operational definitions of driving performance measures and statistics. Technical Report SAE J2944, SAE International. http://standards.sae.org/wip/j2944/. Proposed Draft
Strayer DL, Cooper JM, Turrill J, Coleman JR, Hopman RJ (2015) The smartphone and the driver’s cognitive workload: a comparison of apple, google, and microsoft’s intelligent personal assistants. Technical report, AAA Foundation for Traffic Safety, Washington, DC. https://www.aaafoundation.org/sites/default/files/strayerIIIa_FINALREPORT.pdf
Tsimhoni O, Smith D, Green P (2004) Address entry while driving: speech recognition versus a touch-screen keyboard. Hum Factors 46(4):600–610. https://doi.org/10.1518/hfes.46.4.600.56813 PMID: 15709323
Vollrath M, Maciej J, Niederée U (2008) In-car distraction study—final report. Technical report, Kognitions-und Ingenieurpsychologie, Technische Universität Carolo-Wilhelmina zu Braunschweig, Braunschweig
Ward N (2006) Non-lexical conversational sounds in American english. Pragmat Cognit 14(1):129–182
Weinberg G, Harsham B, Medenica Z (2011) Investigating huds for the presentation of choice lists in car navigation systems. In: Proceedings of the sixth international driving symposium on human factors in driver assessment, training and vehicle design, DA ’11, pp 195–202
White MP, Eiser JR, Harris PR (2004) Risk perceptions of mobile phone use while driving. Risk Anal 24(2):323–334. https://doi.org/10.1111/j.0272-4332.2004.00434.x
Yan B, Weng F, Feng Z, Ratiu F, Raya M, Meng Y, Varges S, Purver M, Lien A, Scheideck T, Raghunathan B, Lin F, Mishra R, Lathrop B, Zhang Z, Bratt H, Peters S (2007) A conversational in-car dialog system. In: Proceedings of human language technologies: the annual conference of the North American chapter of the association for computational linguistics: demonstrations, NAACL-demonstrations ’07. Association for Computational Linguistics, Stroudsburg, pp 23–24. http://dl.acm.org/citation.cfm?id=1614164.1614176
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Braun, M., Broy, N., Pfleging, B. et al. Visualizing natural language interaction for conversational in-vehicle information systems to minimize driver distraction. J Multimodal User Interfaces 13, 71–88 (2019). https://doi.org/10.1007/s12193-019-00301-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12193-019-00301-2