Abstract
The practicality of driving fatigue detection approaches largely depends on social acceptance. Physiological-based methods perform well, but they are rarely accepted by drivers, while subjective evaluations cannot support real-time detection. However, image-based approaches with relatively high take-up rates, ace detection accuracy problems, thus causing driver dissatisfaction. To satisfy the demand of logistic companies, in this paper, a customized driving fatigue detection method that integrates a subjective evaluation, physiological features and image features is proposed. As a result, a real-time driving fatigue detection system, RefreshingDrive, is designed. First, the relationships between questionnaire results, near-infrared spectroscopy indices, and oxyhemoglobin (HBO) and deoxyhemoglobin (HHB) values are investigated to express the individual differences among fatigue thresholds. Then, a long short-term memory network is constructed to find an individual-oriented HBO+HHB threshold, which is used to divide an image dataset into two subdatasets, namely, a fatigue dataset and a nonfatigue dataset. After that, a facial feature-based fatigue detection method, in which a multitasking cascaded eye convolution network is proposed to extract eye features while a 3D ConvNet is employed to recognize fatigue actions, is presented. Moreover, a fatigue evaluation rule, the Action-Blink Fusion Rule (A-B fusion), is proposed to recognize driving fatigue. Experimental results show that the proposed method achieves a fatigue recognition accuracy of 95.3%. Finally, an on-road test is performed to verify the effectiveness of the proposed approach.
Similar content being viewed by others
References
Ahmadi A, Bazregarzadeh H, Kazemi K (2021) Automated detection of driver fatigue from electroencephalography through wavelet-based connectivity. Biocybern Biomed Eng 41(1):316–332
Ahn S, Nguyen T, Jang H, Kim JG, Jun SC (2016) Exploring neuro-physiological correlates of drivers’ mental fatigue caused by sleep deprivation using simultaneous EEG, ECG, and FNIRS data. Front Hum Neurosci 10:219
Akerstedt T, Gillberg M (1990) Subjective and objective sleepiness in the active individual. Int J Neurosci 52(1–2):29–37
Akrout B, Mahdi W (2021) A novel approach for driver fatigue detection based on visual characteristics analysis. J Ambient Intell Human Comput 20:25
Alioua N, Amine A, Rziza M (2014) Driver’s fatigue detection based on yawning extraction. Int J Veh Technol 20:14
Ayachi R, Afif M, Said Y, Abdelali AB (2021) Drivers fatigue detection using efficientdet in advanced driver assistance systems. In: 2021 18th international multi-conference on systems, signals and devices (SSD), pp 738–742. IEEE
Azarnoosh M, Mohammadi MR, Nasrabadi AM, Firoozabadi SMP (2010) Evaluating variability of frequency features of EEG signals during mental fatigue. In: 2010 17th Iranian conference of biomedical engineering (ICBME), pp 1–4. IEEE
Bakker B, Zabłocki B, Baker A, Riethmeister V, Marx B, Iyer G, Anund A, Ahlström C (2022) A multi-stage, multi-feature machine learning approach to detect driver sleepiness in naturalistic road driving conditions. IEEE Trans Intell Transp Syst 23(5):4791–4800
Bhatt R, Naik N, Subramanian VK (2021) Ssim compliant modeling framework with denoising and deblurring applications. IEEE Trans Image Process 30:2611–2626
Chen M, Li F, Lei J, Zeng Z, Han Q, Chen Q (2017) Driving fatigue detecting method based on temperature insensitive ECG parameters. In: International conference on internet of vehicles. Springer, pp 105–118
Dang W, Gao Z, Lv D, Sun X, Cheng C (2020) Rhythm-dependent multilayer brain network for the detection of driving fatigue. IEEE J Biomed Health Inform 25(3):693–700
Daza IG, Bergasa LM, Bronte S, Yebes JJ, Almazán J, Arroyo R (2014) Fusion of optimized indicators from advanced driver assistance systems (ADAS) for driver drowsiness detection. Sensors (Basel, Switzerland) 14(1):1106–1131
Hara K, Kataoka H, Satoh Y (2018) Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6546–6555
Herscovitch J, Broughton R (1981) Sensitivity of the Stanford sleepiness scale to the effects of cumulative partial sleep deprivation and recovery oversleeping. Sleep 4(1):83–92
Hoddes E, Dement W, Zarcone V (1972) The development and use of the Stanford sleepiness scale (SSS). Psychophysiology 10:431–436
Huynh X-P, Park S-M, Kim Y-G (2016) Detection of driver drowsiness using 3d deep neural network and semi-supervised gradient boosting machine. In: Asian conference on computer vision. Springer, pp 134–145
Jia H, Xiao Z, Ji P (2021) Fatigue driving detection based on deep learning and multi-index fusion. IEEE Access 9:147054–147062
Kaida K, Takahashi M, Åkerstedt T, Nakata A, Otsuka Y, Haratani T, Fukasawa K (2006) Validation of the karolinska sleepiness scale against performance and EEG variables. Clin Neurophysiol 117(7):1574–1581
Klaiber M, Sauter D, Baumgartl H, Buettner R (2021) A systematic literature review on transfer learning for 3d-cnns. In: 2021 international joint conference on neural networks (IJCNN), pp 1–10. IEEE
Ku H, Dong W (2020) Face recognition based on mtcnn and convolutional neural network. Front Signal Process 4(1):37–42
Lam C, Epps J, Chen S (2021) Wearable fatigue detection based on blink-saccade synchronisation. In: 2021 IEEE international conference on systems, man, and cybernetics (SMC), pp 1186–1191. IEEE
Lei J, Han Q, Chen L, Lai Z, Zeng L, Liu X (2017) A novel side face contour extraction algorithm for driving fatigue statue recognition. IEEE Access 5:5723–5730
Lei J, Liu F, Han Q, Tang Y, Zeng L, Chen M, Ye L, Jin L (2018) Study on driving fatigue evaluation system based on short time period ECG signal. In: 2018 21st international conference on intelligent transportation systems (ITSC), pp 2466–2470. IEEE
Li Z, Zhang M, Zhang X, Dai S, Yu X, Wang Y (2009) Assessment of cerebral oxygenation during prolonged simulated driving using near infrared spectroscopy: its implications for fatigue development. Eur J Appl Physiol 107(3):281–287
Li R, Chen YV, Zhang L (2021) A method for fatigue detection based on driver’s steering wheel grip. Int J Ind Ergon 82:103083
Li X, Xia J, Cao L, Zhang G, Feng X (2021) Driver fatigue detection based on convolutional neural network and face alignment for edge computing device. Proc Inst Mech Eng Part D J Autom Eng 235(10–11):2699–2711
Lin CT, King JT, Chuang CH, Ding W, Wang YK (2019) Exploring the brain responses to driving fatigue through simultaneous EEG and FNIRS measurements. Int J Neural Syst 30:9
Liu Y, Zhang T, Li Z (2019) Dcnn-based real-time driver fatigue behavior detection in urban rail transit. IEEE Access 99:1
Liu Z, Peng Y, Hu W (2019) Driver fatigue detection based on deeply-learned facial expression representation. J Vis Commun Image Represent 71:102723
Lu Y, Wang Z (2007) Detecting driver yawning in successive images. In: 2007 1st international conference on bioinformatics and biomedical engineering, pp 581–583. IEEE
Mao H, Tang J, Zhao X, Tang M, Jiang Z (2022) A driver drowsiness detection scheme based on 3d convolutional neural networks. Int J Pattern Recognit Artif Intell 36(02):2252007
Min J, Xiong C, Zhang Y, Cai M (2021) Driver fatigue detection based on prefrontal EEG using multi-entropy measures and hybrid model. Biomed Signal Process Control 69:102857
NHTSA (2020) Preview of motor vehicle crashes in 2019. NHTSA’s National Center for Statistics and Analysis, 1
Nihashi T, Ishigaki T, Satake H, Ito S, Kaii O, Mori Y, Shimamoto K, Fukushima H, Suzuki K, Umakoshi H et al (2019) Monitoring of fatigue in radiologists during prolonged image interpretation using fnirs. Jpn J Radiol 37(6):437–448
Pan T, Wang H, Si H, Liu H, Xu M (2022) Research on the identification of pilots’ fatigue status based on functional near-infrared spectroscopy. Aerospace 9(3):173
Pei Z, Zhenghe S, Yiming Z (2002) Perclos-based recognition algorithms of motor driver fatigue. J China Agric Univ 7(2):104–109
Penson A, van Deuren S, Worm-Smeitink M, Bronkhorst E, van den Hoogen FHJ, van Engelen BGM, Peters M, Bleijenberg G, Vercoulen JH, Blijlevens N, van Dulmen-den Broeder E, Loonen J, Knoop H (2020) Short fatigue questionnaire: screening for severe fatigue. J Psychosom Res 137:110229
Połap D, Srivastava G (2021) Neural image reconstruction using a heuristic validation mechanism. Neural Comput Appl 33(17):10787–10797
Połap D, Woźniak M (2021) Meta-heuristic as manager in federated learning approaches for image processing purposes. Appl Soft Comput 113:107872
Połap D, Wawrzyniak N, Włodarczyk-Sielicka M (2022) Side-scan sonar analysis using roi analysis and deep neural networks. IEEE Trans Geosci Remote Sens 60:1–8
Putilov AA, Donskaya OG (2013) Construction and validation of the EEG analogues of the karolinska sleepiness scale based on the karolinska drowsiness test. Clin Neurophysiol 124(7):1346–1352
Tak S, Ye JC (2014) Statistical analysis of fnirs data: a comprehensive review. Neuroimage 85:72–91
Tran D, Bourdev L, Fergus R, Torresani L, Paluri M (2015) Learning spatiotemporal features with 3d convolutional networks. In: Proceedings of the IEEE international conference on computer vision, pp 4489–4497
Tuncer T, Dogan S, Ertam F, Subasi A (2021) A dynamic center and multi threshold point based stable feature extraction network for driver fatigue detection utilizing EEG signals. Cogn Neurodyn 15:7
Wang Z, Bovik A, Sheikh H, Simoncelli E (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612
Wang F, Hong W, Fu R (2018) Real-time ECG-based detection of fatigue driving using sample entropy. Entropy 20(3):196
Wang H, Dragomir A, Abbasi NI, Li J, Thakor NV, Bezerianos A (2018) A novel real-time driving fatigue detection system based on wireless dry EEG. Cogn Neurodyn 12(4):365–376
Wu EQ, Xiong P, Tang ZR, Li GJ, Song A, Zhu LM (2022) Detecting dynamic behavior of brain fatigue through 3-d-cnn-lstm. IEEE Trans Syst Man Cybern Syst 52(1):90–100
Xu S, Zhao X-h, Zhang X-J, Rong J (2011) A study of the identification method of driving fatigue based on physiological signals. In: ICCTP 2011: towards sustainable transportation systems, pp 2296–2307
Yan P, Sun Y, Li Z, Zou J, Hong D (2020) Driver fatigue detection system based on colored and infrared eye features fusion. Comput Mater Contin 63(3):1563–1574
Yang JH, Mao Z-H, Tijerina L, Pilutti T, Coughlin J, Feron E (2009) Detection of driver fatigue caused by sleep deprivation. IEEE Trans Syst Man Cybern Part A Syst Humans 39(4):694–705
Ye M, Zhang W, Cao P, Liu K (2021) Driver fatigue detection based on residual channel attention network and head pose estimation. Appl Sci 11(19):9195
Yin Z, Liu B, Hao D, Yang L, Feng Y (2022) Evaluation of vdt-induced visual fatigue by automatic detection of blink features. Sensors 22(3):916
Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett 23(10):1499–1503
Zhang F, Su J, Geng L, Xiao Z (2017) Driver fatigue detection based on eye state recognition. In: 2017 international conference on machine vision and information technology (CMVIT), pp 105–110. IEEE
Zhao Z, Zhou N, Zhang L, Yan H, Xu Y, Zhang Z (2020) Driver fatigue detection based on convolutional neural networks using EM-CNN. Comput Intell Neurosci 2020:7251280
Zhao G, He Y, Yang H, Tao Y (2022) Research on fatigue detection based on visual features. IET Image Proc 16(4):1044–1053
Zhu T, Zhang C, Wu T, Ouyang Z, Li H, Na X, Liang J, Li W (2022) Research on a real-time driver fatigue detection algorithm based on facial video sequences. Appl Sci 12(4):2224
Acknowledgements
Thank the anonymous reviewers for their insightful comments that resulted in significantly improved paper. This work was supported by the special key project of Chongqing Technology Innovation and Application Development under Grant No.csct 2021jscx-gksbX0057, in part by the National Nature Science Foundation of China, under Project 62172066 and U21A20448, in part by Central University Foundation of China, under project 2022CDJJJ-003.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Zeng, L., Zhou, K., Han, Q. et al. An fNIRS labeling image feature-based customized driving fatigue detection method. J Ambient Intell Human Comput 14, 12493–12509 (2023). https://doi.org/10.1007/s12652-022-04325-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-022-04325-7