Skip to main content
SpringerLink
Log in

Driver fatigue detection based on comprehensive facial features and gated recurrent unit

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

In recent years, deep learning-based driver fatigue detection algorithms have been increasingly used. However, traditional fatigue detection algorithms cannot effectively correlate contextual information of image frames. They perform better in individual image frames. Also, the accuracy and robustness of these algorithms are limited because they only consider particular frames. Therefore, a fatigue detection method based on integrated facial features and Gate Recurrent Unit (GRU) judgment neural network is proposed in this paper. We use a neural network including a GRU layer to efficiently distinguish the contextual information present in multiple image frames arranged in chronological order. Besides, we designed a multi-task convolutional neural network (MTCNN) model to extract comprehensive facial features. After obtaining the facial feature points’ positions, we can calculate the aspect ratio between the upper and lower eyelids, the upper and lower lips, and the eyebrows to the chin. In addition to the above three features, we can also obtain the subject's three head pose angles by comparing the facial features with the typical 3D face model. Finally, we input the change curves of 6 features in 20 consecutive frames into the judgment network to learn the change rule and create a judgment network. After the learning is completed, the judgment network model will judge the six feature curves in the newly input 20 frames in real-time and output the driver's fatigue status. This fatigue detection method can take a real-time detection at 55 FPS on the workstation platform (TensorFlow 2.3.0, RTX2070s). On the Nvidia Jetson Xavier AGX embedded platform (TensorFlow lite, ARM 8-cores CPU), the method can take a real-time detection at 26 FPS. The accuracy of this fatigue detection method can reach 97.47%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Rau, P. S.: Drowsy driver detection and warning system for commercial vehicle drivers: field operational test design, data analyses, and progress. In: 19th International Conference on Enhanced Safety of Vehicles, 6–9 (2005)

  2. Silveira, C.S., Cardoso, J.S., Lourenço, A.L., et al.: Importance of subject-dependent classification and imbalanced distributions in driver sleepiness detection in realistic conditions. IET Intel. Transp. Syst. 13(2), 347–355 (2018)

    Article  Google Scholar 

  3. Ma, J., Murphey, Y. L., Zhao, H.: Real time drowsiness detection based on lateral distance using wavelet transform and neural network. In: 2015 IEEE symposium series on computational intelligence. IEEE, pp 411–418 (2015)

  4. Pratama, B. G., Ardiyanto, I., Adji, T. B.: A review on driver drowsiness based on image, bio-signal, and driver behavior. In: 2017 3rd International Conference on Science and Technology-Computer (ICST). IEEE, pp 70–75 (2017)

  5. Li, Z., Li, S.E., Li, R., et al.: Online detection of driver fatigue using steering wheel angles for real driving conditions. Sensors 17(3), 495 (2017)

    Article  MathSciNet  Google Scholar 

  6. Alioua, N., Amine, A., Rziza, M.: Driver’s fatigue detection based on yawning extraction. Int. J. Veh Technol. 2014, 1–7 (2014)

    Article  Google Scholar 

  7. Gasser, T., Sroka, L., Möcks, J.: The transfer of EOG activity into the EEG for eyes open and closed. Electroencephalogr. Clin. Neurophysiol. 61(2), 181–193 (1985)

    Article  Google Scholar 

  8. Kim, J.Y., Jeong, C.H., Jung, M.J., et al.: Highly reliable driving workload analysis using driver electroencephalogram (EEG) activities during driving. Int. J. Automot. Technol. 14(6), 965–970 (2013)

    Article  Google Scholar 

  9. Fatourechi, M., Bashashati, A., Ward, R.K., et al.: EMG and EOG artifacts in brain computer interface systems: a survey. Clin. Neurophysiol. 118(3), 480–494 (2007)

    Article  Google Scholar 

  10. Guede-Fernández, F., Fernández-Chimeno, M., Ramos-Castro, J., et al.: Driver drowsiness detection based on respiratory signal analysis. IEEE access 7, 81826–81838 (2019)

    Article  Google Scholar 

  11. Jung, S.J., Shin, H.S., Chung, W.Y.: Driver fatigue and drowsiness monitoring system with embedded electrocardiogram sensor on steering wheel. IET Intell. Transp. Syst. 8(1), 43–50 (2014)

    Article  Google Scholar 

  12. Lee, S.J., Jo, J., Jung, H.G., et al.: Real-time gaze estimator based on driver’s head orientation for forward collision warning system. IEEE Trans. Intell. Transp. Syst. 12(1), 254–267 (2011)

    Article  Google Scholar 

  13. Mbouna, R.O., Kong, S.G., Chun, M.G.: Visual analysis of eye state and head pose for driver alertness monitoring. IEEE Trans. Intell. Transp. Syst. 14(3), 1462–1469 (2013)

    Article  Google Scholar 

  14. Dinges, D. F., Grace, R.: PERCLOS: a valid psychophysiological measure of alertness as assessed by psychomotor vigilance. US Department of Transportation, Federal Highway Administration, Publication Number FHWA-MCRT-98-006 (1998)

  15. Mandal, B., Li, L., Wang, G.S., et al.: Towards detection of bus driver fatigue based on robust visual analysis of eye state. IEEE Trans. Intell. Transp. Syst. 18(3), 545–557 (2016)

    Article  Google Scholar 

  16. You, F., Li, X., Gong, Y., et al.: A real-time driving drowsiness detection algorithm with individual differences consideration. IEEE Access 7, 179396–179408 (2019)

    Article  Google Scholar 

  17. Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 1527–1554 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  18. Bengio, Y.: Learning deep architectures for AI. Now Publishers Inc (2009)

    Book  MATH  Google Scholar 

  19. Arel, I., Rose, D.C., Karnowski, T.P.: Deep machine learning-a new frontier in artificial intelligence research [research frontier]. IEEE Comput. Intell. Mag. 5(4), 13–18 (2010)

    Article  Google Scholar 

  20. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  21. LeCun, Y., Bottou, L., Bengio, Y., et al.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  22. Chirra, V.R.R., ReddyUyyala, S., Kolli, V.K.K.: Deep CNN: a machine learning approach for driver drowsiness detection based on eye state. Revue d’Intelligence Artificielle 33(6), 461–466 (2019)

    Article  Google Scholar 

  23. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)

    Article  Google Scholar 

  24. Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: IEEE Conference on Computer Vision & Pattern Recognition. IEEE (2014)

  25. Zhang, K., Zhang, Z., Li, Z., et al.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process. Lett. 23(10), 1499–1503 (2016)

    Article  Google Scholar 

  26. Xiao, Z., Hu, Z., Geng, L., et al.: Fatigue driving recognition network: fatigue driving recognition via convolutional neural network and long short-term memory units. IET Intell. Transp. Syst. 13(9), 1410–1416 (2019)

    Article  Google Scholar 

  27. Geng, L.: Real-time driver fatigue detection based on morphology infrared features and deep learning. Hongwai Yu Jiguang Gongcheng/Infrared Laser Eng. 47(2), 203009 (2018)

    Article  Google Scholar 

  28. Guo, J., Markoni, H.: Driver drowsiness detection using hybrid convolutional neural network and long short-term memory. Multimed. Tools Appl. 78(20), 29059–29087 (2019)

    Article  Google Scholar 

  29. You, F., Li, X., Gong, Y., Wang, H., Li, H.: A real-time driving drowsiness detection algorithm with individual differences consideration. IEEE Access 7, 179396–179408 (2019). https://doi.org/10.1109/access.2019.2958667

    Article  Google Scholar 

Download references

Funding

Our work is supported by Tianjin graduate scientific research and innovation project (special project of artificial intelligence), and the funding number is 2020YJSZXS21.

Author information

Authors and Affiliations

  1. Tianjin University of Science & Technology, Tianjin, China

    Dan Li, Xin Zhang, Xiaofan Liu, Zhicheng Ma & Baolong Zhang

Authors
  1. Dan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaofan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhicheng Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Baolong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Baolong Zhang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, D., Zhang, X., Liu, X. et al. Driver fatigue detection based on comprehensive facial features and gated recurrent unit. J Real-Time Image Proc 20, 19 (2023). https://doi.org/10.1007/s11554-023-01260-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11554-023-01260-4

Keywords

Search

Navigation