Transactions on Transport Sciences 2023, 14(1):4-13 | DOI: 10.5507/tots.2022.019

Feasibility of AR-VR use in autonomous cars for user engagements and its effects on posture and vigilance during transit

Joseph Muguroa, b, *, Pringgo Widyo Laksonoc, Yuta Sasataked, Muhammad Ilhamdi Rusydie, Kojiro Matsushitad, Minoru Sasakia, *
a. Intelligent Production Technology Research & Development Center for Aerospace (IPTeCA), Tokai National Higher Education and Research System, Japan
b. School of Engineering, Dedan Kimanthi University of Technology, 657-10100, Nyeri, Kenya
c. Industrial Engineering, Universitas Sebelas Maret, Surakarta, Indonesia
d. Department of Mechanical Engineering, Faculty of Engineering, Gifu University, 1- 1 Yanagido, Gifu, 501- 1193, Japan
e. Department of Electrical Engineering, Engineering Faculty of Universitas Andalas, Padang 25163, Indonesia

Autonomous driving system (ADS) is anticipated to revolutionize travel by reclaiming lost time and improve safety on the roads. With automation, user-engagements that enhances road monitoring should be considered to maintain vigilance and safety. From the literature, virtual reality (VR) usage in cars offer productivity and increased privacy. This paper explores the efficacy of passenger use of VR headsets to enhance user-engagement during transit. User-engagement was quantified using physiological measures (pupillary response and electrodermal activity) during an in-car VR game/activity experiment. Further, the impacts of engaging with secondary tasks was evaluated using reaction time of pop-up objects. We designed a driving simulation with inbuilt entertaining activities, no-task, game-task, video-task, and mixed-task, played in a real car with a FOVE VR headset on the perimeter track of the Gifu University campus with 15 subjects (average 25.6 years, SD = 6.4). From reaction time, significant difference between tasks was found using one-way ANOVA (F(3,231) = 2.75, p = .0437). A post-hoc test revealed that game and mixed task reaction times were significantly different (p = .0126 and p = .016, respectively) suggesting that task design should consider hazard recognition in a real car. From physiological measures, an increased/sustained effect of user engagement was noted compared with baseline (no-task) suggesting effectiveness in maintaining vigilance. The results also reported a 10-fold improvement in sitting posture compared to baseline. The methodology employed is applicable as an indirect measure of engagement that would find use in productivity and vigilance study in an ADS.

Keywords: Autonomous Driving Systems; Driver, passenger engagement; Driving Related Tasks, 3D-VR, AR, In-Car VR

Received: July 22, 2022; Revised: October 14, 2022; Accepted: November 9, 2022; Prepublished online: November 30, 2022; Published: April 20, 2023  Show citation

ACS AIP APA ASA Harvard Chicago IEEE ISO690 MLA NLM Turabian Vancouver
Muguro, J., Laksono, P.W., Sasatake, Y., Rusydi, M.I., Matsushita, K., & Sasaki, M. (2023). Feasibility of AR-VR use in autonomous cars for user engagements and its effects on posture and vigilance during transit. Transactions on Transport Sciences, 14(1), 4-13. doi: 10.5507/tots.2022.019
Share...
Download citation
  • BibTeX (.bib)
  • Bookends (.ris)
  • EasyBib (.ris)
  • EndNote (.enw)
  • EndNote 8 (.xml)
  • ISI WoS (.isi)
  • Medlars (.medlars)
  • Mendeley (.ris)
  • MODS (.xml)
  • Papers (.ris)
  • RefWorks (.txt)
  • RefManager (.ris)
  • RIS (.ris)
  • MS Word (.xml)
  • Zotero (.ris)
    • Open full article

    References

    1. Anderson, J. M., Kalra, N., Stanley, K. D., Sorensen, P., Samaras, C., & Oluwatola, O. A. (2014). Brief History and Current State of Autonomous Vehicles. In Autonomous Vehicle Technology Book Subtitle: A Guide for Policymakers. https://doi.org/10.7249/j.ctt5hhwgz.11 Go to original source...
    2. Audi. (2019). holoride: Virtual Reality meets the real world | audi.com. https://www.audi.com/en/experience-audi/mobility-and-trends/digitalization/holoride-virtual-reality-meets-the-real-world.html
    3. BBCNews. (2018). Tesla in fatal California crash was on Autopilot - BBC News. BBC News. https://www.bbc.com/news/world-us-canada-43604440
    4. Bench, S. W., & Lench, H. C. (2013). On the function of boredom. In Behavioral Sciences (Vol. 3, Issue 3, pp. 459-472). MDPI Multidisciplinary Digital Publishing Institute. https://doi.org/10.3390/bs3030459 Go to original source...
    5. Bixler, R., & D'Mello, S. (2016). Automatic gaze-based user-independent detection of mind wandering during computerized reading. User Modeling and User-Adapted Interaction, 26(1), 33-68. https://doi.org/10.1007/s11257-015-9167-1 Go to original source...
    6. Braithwaite, J. J., Derrick, D., Watson, G., Jones, R., & Rowe, M. (2015). A Guide for Analysing Electrodermal Activity (EDA) & Skin Conductance Responses (SCRs) for Psychological Experiments.
    7. Broy, N., Goebl, S., Hauder, M., Kothmayr, T., Kugler, M., Reinhart, F., Salfer, M., Schlieper, K., & André, E. (2011). A cooperative in-car game for heterogeneous players. Proceedings of the 3rd International Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI 2011, 167-174. https://doi.org/10.1145/2381416.2381443 Go to original source...
    8. Èegovnik, T., Stojmenova, K., Jakus, G., & Sodnik, J. (2018). An analysis of the suitability of a low-cost eye tracker for assessing the cognitive load of drivers. Applied Ergonomics, 68, 1-11. https://doi.org/10.1016/j.apergo.2017.10.011 Go to original source...
    9. Chang, E., Kim, H. T., & Yoo, B. (2021). Predicting cybersickness based on user's gaze behaviors in HMD-based virtual reality. Journal of Computational Design and Engineering, 00(February), 1-12. https://doi.org/10.1093/jcde/qwab010 Go to original source...
    10. de Winkel, K. N., Pretto, P., Nooij, S. A. E., Cohen, I., & Bülthoff, H. H. (2021). Efficacy of augmented visual environments for reducing sickness in autonomous vehicles. Applied Ergonomics, 90, 103282. https://doi.org/10.1016/j.apergo.2020.103282 Go to original source...
    11. Deahl, D. (2018). Apple applies for patent that lets riders virtually fight zombies or hang glide in self-driving cars. The Verge. https://www.theverge.com/2018/4/2/17188782/apple-virtual-augmented-reality-patent-self-driving-cars
    12. Diels, C., & Bos, J. E. (2016). Self-driving carsickness. Applied Ergonomics, 53, 374-382. https://doi.org/10.1016/j.apergo.2015.09.009 Go to original source...
    13. Dingus, T., Klauer, S., Neale, V., Petersen, A., Lee, S., Sudweeks, J., Perez, M., Hankey, J., Ramsey, D., Gupta, S., Bucher, C., Doerzaph, Z., Jermeland, J., K. R. (2006). The 100-Car Naturalistic Driving Study Phase II-Results of the 100-Car Field Experiment. Go to original source...
    14. Doubek, F., Loosveld, E., Happee, R., & De Winter, J. (2020). Takeover Quality: Assessing the Effects of Time Budget and Traffic Density with the Help of a Trajectory-Planning Method. Journal of Advanced Transportation, 2020. https://doi.org/10.1155/2020/6173150 Go to original source...
    15. Ekchian, J., Graves, W., Anderson, Z., Giovanardi, M., Godwin, O., Kaplan, J., Ventura, J., Lackner, J. R., & Dizio, P. (2016). A High-Bandwidth Active Suspension for Motion Sickness Mitigation in Autonomous Vehicles. SAE Technical Papers, 2016-April(April). https://doi.org/10.4271/2016-01-1555 Go to original source...
    16. Elliott, D., Keen, W., & Miao, L. (2019). Recent advances in connected and automated vehicles. Journal of Traffic and Transportation Engineering (English Edition), 6(2), 109-131. https://doi.org/10.1016/J.JTTE.2018.09.005 Go to original source...
    17. Fransson, P. A., Patel, M., Jensen, H., Lundberg, M., Tjernström, F., Magnusson, M., & Ekvall Hansson, E. (2019). Postural instability in an immersive Virtual Reality adapts with repetition and includes directional and gender specific effects. Scientific Reports, 9(1), 1-10. https://doi.org/10.1038/s41598-019-39104-6 Go to original source...
    18. Gandolfi, M., Geroin, C., Dimitrova, E., Boldrini, P., Waldner, A., Bonadiman, S., Picelli, A., Regazzo, S., Stirbu, E., Primon, D., Bosello, C., Gravina, A. R., Peron, L., Trevisan, M., Garcia, A. C., Menel, A., Bloccari, L., Valè, N., Saltuari, L., … Smania, N. (2017). Virtual Reality Telerehabilitation for Postural Instability in Parkinson's Disease: A Multicenter, Single-Blind, Randomized, Controlled Trial. BioMed Research International, 2017. https://doi.org/10.1155/2017/7962826 Go to original source...
    19. Happee, R., Gold, C., Radlmayr, J., Hergeth, S., & Bengler, K. (2017). Take-over performance in evasive manoeuvres. Accident Analysis and Prevention, 106, 211-222. https://doi.org/10.1016/j.aap.2017.04.017 Go to original source...
    20. Henderson, R. R., Bradley, M. M., & Lang, P. J. (2018). Emotional imagery and pupil diameter. Psychophysiology, 55(6), 1-13. https://doi.org/10.1111/psyp.13050 Go to original source...
    21. Hock, P., Benedikter, S., Gugenheimer, J., & Rukzio, E. (2017). CarVR: Enabling in-car virtual reality entertainment. Conference on Human Factors in Computing Systems - Proceedings, 2017-May, 4034-4044. https://doi.org/10.1145/3025453.3025665 Go to original source...
    22. Iskander, J., Attia, M., Saleh, K., Nahavandi, D., Abobakr, A., Mohamed, S., Asadi, H., Khosravi, A., Lim, C. P., & Hossny, M. (2019). From car sickness to autonomous car sickness: A review. Transportation Research Part F: Traffic Psychology and Behaviour, 62, 716-726. https://doi.org/10.1016/J.TRF.2019.02.020 Go to original source...
    23. Jeong, H., & Liu, Y. (2019). Effects of non-driving-related-task modality and road geometry on eye movements, lane-keeping performance, and workload while driving. Transportation Research Part F: Traffic Psychology and Behaviour, 60, 157-171. https://doi.org/10.1016/j.trf.2018.10.015 Go to original source...
    24. Karjanto, J., Md. Yusof, N., Wang, C., Terken, J., Delbressine, F., & Rauterberg, M. (2018). The effect of peripheral visual feedforward system in enhancing situation awareness and mitigating motion sickness in fully automated driving. Transportation Research Part F: Traffic Psychology and Behaviour, 58, 678-692. https://doi.org/10.1016/j.trf.2018.06.046 Go to original source...
    25. Koch, A., Cascorbi, I., Westhofen, M., Dafotakis, M., Klapa, S., & Kuhtz-Buschbeck, J. P. (2018). The Neurophysiology and Treatment of Motion Sickness. Deutsches Arzteblatt International, 115(41), 687-696. https://doi.org/10.3238/arztebl.2018.0687 Go to original source...
    26. Körber, M., Cingel, A., Zimmermann, M., & Bengler, K. (2015). Vigilance Decrement and Passive Fatigue Caused by Monotony in Automated Driving. Procedia Manufacturing, 3, 2403-2409. https://doi.org/10.1016/j.promfg.2015.07.499 Go to original source...
    27. Kuiper, O. X., Bos, J. E., Diels, C., & Schmidt, E. A. (2020). Knowing what's coming: Anticipatory audio cues can mitigate motion sickness. Applied Ergonomics, 85, 103068. https://doi.org/10.1016/j.apergo.2020.103068 Go to original source...
    28. Le, A. S., Suzuki, T., & Aoki, H. (2020). Evaluating driver cognitive distraction by eye tracking: From simulator to driving. Transportation Research Interdisciplinary Perspectives, 4, 100087. https://doi.org/10.1016/j.trip.2019.100087 Go to original source...
    29. Leung, A. K., & Hon, K. L. (2019). Motion sickness: an overview. Drugs in Context, 8, 2014-2019. https://doi.org/10.7573/dic.2019-9-4 Go to original source...
    30. Li, J., George, C., Ngao, A., Holländer, K., Mayer, S., & Butz, A. (2021). Rear-seat productivity in virtual reality: Investigating vr interaction in the confined space of a car. Multimodal Technologies and Interaction, 5(4). https://doi.org/10.3390/mti5040015 Go to original source...
    31. Li, X., Schroeter, R., Rakotonirainy, A., Kuo, J., & Lenné, M. G. (2020). Effects of different non-driving-related-task display modes on drivers' eye-movement patterns during take-over in an automated vehicle. Transportation Research Part F: Traffic Psychology and Behaviour, 70, 135-148. https://doi.org/10.1016/j.trf.2020.03.001 Go to original source...
    32. Lin, R., Ma, L., & Zhang, W. (2018). An interview study exploring Tesla drivers' behavioural adaptation. Applied Ergonomics, 72, 37-47. https://doi.org/10.1016/j.apergo.2018.04.006 Go to original source...
    33. Lotz, A., Russwinkel, N., & Wohlfarth, E. (2019). Response times and gaze behavior of truck drivers in time critical conditional automated driving take-overs. Transportation Research Part F: Traffic Psychology and Behaviour, 64, 532-551. https://doi.org/10.1016/j.trf.2019.06.008 Go to original source...
    34. Lubben, A. (2018). Self-driving Uber killed a pedestrian as human safety driver watched. https://www.vice.com/en/article/kzxq3y/self-driving-uber-killed-a-pedestrian-as-human-safety-driver-watched
    35. Mai, C., & Khamis, M. (2018). Public HMDs: Modeling and understanding user behavior around public head-mounted displays. PerDis 2018 - Proceedings of the 7th ACM International Symposium on Pervasive Displays, June. https://doi.org/10.1145/3205873.3205879 Go to original source...
    36. Mark B. Rober, Sawyer I. Cohen, Daniel Kurz, Tobias Holl, Benjamin B. Lyon, Peter George Meier, Jeffrey M. Riepling, H. G. (2018). Immersive Virtual Dispaly (Patent No. US2018/0089901A1). United States.
    37. Mathis, F., & Khamis, M. (n.d.). Privacy, Security and Safety Concerns of using HMDs in Public and Semi-Public Spaces. In Proceedings of the 2019 CHI Conference Extended Abstracts on Human Factors in Computing Systems.
    38. McGill, M., Ng, A., & Brewster, S. (2017a). I am the passenger: How visual motion cues can influence sickness for in-car VR. Conference on Human Factors in Computing Systems - Proceedings, 2017-May(May 2017), 5655-5668. https://doi.org/10.1145/3025453.3026046 Go to original source...
    39. McGill, M., Ng, A., & Brewster, S. (2017b). I am the passenger: How visual motion cues can influence sickness for in-car VR. Conference on Human Factors in Computing Systems - Proceedings, 2017-May(May 2017), 5655-5668. https://doi.org/10.1145/3025453.3026046 Go to original source...
    40. McGill, M., Williamson, J., Ng, A., Pollick, F., & Brewster, S. (2019). Challenges in passenger use of mixed reality headsets in cars and other transportation. Virtual Reality, Prince 2014. https://doi.org/10.1007/s10055-019-00420-x Go to original source...
    41. Miller, D., Sun, A., Johns, M., Ive, H., Sirkin, D., Aich, S., & Ju, W. (2015). Distraction Becomes Engagement in Automated Driving. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 59(1), 1676-1680. https://doi.org/10.1177/1541931215591362 Go to original source...
    42. Muguro, J. K., Laksono, P. W., Sasatake, Y., Matsushita, K., & Sasaki, M. (2021). User Monitoring in Autonomous Driving System Using Gamified Task: A Case for VR/AR In-Car Gaming. Multimodal Technologies and Interaction, 5(8). https://doi.org/10.3390/mti5080040 Go to original source...
    43. Paredes, P. E., Balters, S., Qian, K., Murnane, E. L., Ordóñez, F., Ju, W., & Landay, J. A. (2018). Driving with the Fishes. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2(4), 1-21. https://doi.org/10.1145/3287062 Go to original source...
    44. Riegler, A., Riener, A., & Holzmann, C. (2020). A Research Agenda for Mixed Reality in Automated Vehicles. ACM International Conference Proceeding Series, 119-131. https://doi.org/10.1145/3428361.3428390 Go to original source...
    45. Salter, S., Diels, C., Herriotts, P., Kanarachos, S., & Thake, D. (2019). Motion sickness in automated vehicles with forward and rearward facing seating orientations. Applied Ergonomics, 78, 54-61. https://doi.org/10.1016/J.APERGO.2019.02.001 Go to original source...
    46. Schwall, M., Daniel, T., Victor, T., Favarò, F., & Hohnhold, H. (2020). Waymo Public Road Safety Performance Data.
    47. Shukla, J., Barreda-Angeles, M., Oliver, J., Nandi, G. C., & Puig, D. (2019). Feature Extraction and Selection for Emotion Recognition from Electrodermal Activity. IEEE Transactions on Affective Computing. https://doi.org/10.1109/TAFFC.2019.2901673 Go to original source...
    48. Standard, S. (2018). J3016B: Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles - SAE International. https://www.sae.org/standards/content/j3016_201806/
    49. Steinberger, F., Schroeter, R., & Watling, C. N. (2017). From road distraction to safe driving: Evaluating the effects of boredom and gamification on driving behaviour, physiological arousal, and subjective experience. Computers in Human Behavior, 75, 714-726. https://doi.org/10.1016/j.chb.2017.06.019 Go to original source...
    50. Steinhauer, S. R., Siegle, G. J., Condray, R., & Pless, M. (2004). Sympathetic and parasympathetic innervation of pupillary dilation during sustained processing. International Journal of Psychophysiology, 52(1), 77-86. https://doi.org/10.1016/j.ijpsycho.2003.12.005 Go to original source...
    51. Thiffault, P., & Bergeron, J. (2003). Monotony of road environment and driver fatigue: A simulator study. Accident Analysis and Prevention, 35(3), 381-391. https://doi.org/10.1016/S0001-4575(02)00014-3 Go to original source...
    52. Vibert, N., MacDougall, H. G., de Waele, C., Gilchrist, D. P. D., Burgess, A. M., Sidis, A., Migliaccio, A., Curthoys, I. S., & Vidal, P. P. (2001). Variability in the control of head movements in seated humans: A link with whiplash injuries? Journal of Physiology, 532(3), 851-868. https://doi.org/10.1111/j.1469-7793.2001.0851e.x Go to original source...
    53. Wada, T. (2017). Motion sickness in automated vehicles. Advanced Vehicle Control AVEC16 - Proceedings of the 13th International Symposium on Advanced Vehicle Control AVEC16, September, 169-176. https://doi.org/10.1201/9781315265285-28 Go to original source...
    54. Wada, T., Fujisawa, S., Imaizumi, K., Kamiji, N., & Doi, S. (2010). Effect of Driver's Head Tilt Strategy on Motion Sickness Incidence. IFAC Proceedings Volumes, 43(13), 192-197. https://doi.org/10.3182/20100831-4-FR-2021.00035 Go to original source...
    55. Wang, Y., Zhai, G., Chen, S., Min, X., Gao, Z., & Song, X. (2019). Assessment of eye fatigue caused by head-mounted displays using eye-tracking. BioMedical Engineering Online, 18(1), 1-19. https://doi.org/10.1186/S12938-019-0731-5/TABLES/8 Go to original source...
    56. Waymo. (2020). Waymo's Safety Methodologies and Safety Readiness Determinations. www.waymo.com/safety
    57. Wu, Y., Kihara, K., Takeda, Y., Sato, T., Akamatsu, M., Kitazaki, S., Nakagawa, K., Yamada, K., Oka, H., & Kameyama, S. (2021). Eye movements predict driver reaction time to takeover request in automated driving: A real-vehicle study. Transportation Research Part F: Traffic Psychology and Behaviour, 81, 355-363. https://doi.org/10.1016/J.TRF.2021.06.017 Go to original source...
    58. Zeeb, K., Buchner, A., & Schrauf, M. (2016). Is take-over time all that matters? the impact of visual-cognitive load on driver take-over quality after conditionally automated driving. Accident Analysis and Prevention, 92, 230-239. https://doi.org/10.1016/j.aap.2016.04.002 Go to original source...
    59. Zimasa, T., Jamson, S., & Henson, B. (2019). The influence of driver's mood on car following and glance behaviour: Using cognitive load as an intervention. Transportation Research Part F: Traffic Psychology and Behaviour, 66, 87-100. https://doi.org/10.1016/j.trf.2019.08.019 Go to original source...
    60. Zuckerman, O., Hoffman, G., & Gal-Oz, A. (2014). In-car game design for children: Promoting interactions inside and outside the car. International Journal of Child-Computer Interaction, 2(4), 109-119. https://doi.org/10.1016/j.ijcci.2015.03.001 Go to original source...

    This is an open access article distributed under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits use, distribution, and reproduction in any medium, provided the original publication is properly cited. No use, distribution or reproduction is permitted which does not comply with these terms.


    Return to the content