Skip to main content

Advertisement

SpringerLink
Log in

Development and validation of a deep learning-based algorithm for drowsiness detection in facial photographs

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Drowsiness is a feeling of sleepiness before the sleep onset and has severe implications from a safety perspective for the individuals involved in industrial activities, mining, and driving. The state-of-the-art computer vision (CV) based drowsiness detection methods generally utilize multiple deep convolutional neural networks (DCNN) without investigating deep feature aggregation techniques for the drowsiness detection task. More importantly, the reported results are mostly based on acted drowsy data, making the utilization of models trained on such data highly arguable for detecting drowsiness in real-life situations. Towards ameliorating this, we first present a comprehensive real drowsy data curated from 50 subjects, where subjects are labeled as fresh or drowsy. Further, four DCNN models: Xception, ResNet101, InceptionV4, and ResNext101, are trained on our dataset using transfer learning to select a baseline model for our drowsiness detection method. Moreover, an experimental study is performed using five different pooling methods: global max, global average, generalized mean, region of interest, and Weibull activation, to compute a robust and discriminative global descriptor. Our results reveal that the parametric Weibull activation pooling is the best suited for aggregating deep convolutional features. Additionally, a low complexity model based on the MobileNetV2 is proposed for a deployable drowsiness detection solution in mobile devices. The detection accuracy of 93.80% and 90.50% is achieved using our proposed Weibull-based ResNext101 and MobileNetV2 models, respectively. Moreover, our results show that the proposed non-invasive method outperforms the polysomnography signals-based invasive drowsiness detection approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Abtahi S, Omidyeganeh M, Shirmohammadi S, Hariri B (2014) Yawdd: A yawning detection dataset. In: Proceedings of the 5th ACM Multimedia Systems Conference. ACM, pp 24–28

  2. Akin M, Kurt MB, Sezgin N, Bayram M (2008) Estimating vigilance level by using EEG and EMG signals. Neural Comput Applic 17(3):227–236

    Article  Google Scholar 

  3. Akrout B, Mahdi W (2013) Vision based approach for driver drowsiness detection based on 3d head orientation. In: Multimedia and ubiquitous engineering. Springer, pp 43–50

  4. Arefnezhad S, Samiee S, Eichberger A, Nahvi A (2019) Driver drowsiness detection based on steering wheel data applying adaptive neuro-fuzzy feature selection. Sensors 19(4):943

    Article  Google Scholar 

  5. Arora M, Kumar M (2021) AutoFER PCA and PSO based automatic facial emotion recognition. Multimed Tools Appl 80(2):3039–3049

    Article  Google Scholar 

  6. Azizpour H, Razavian AS, Sullivan J, Maki A, Carlsson S (2015) From generic to specific deep representations for visual recognition. In: 2015 IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 36–45

  7. Bansal M, Kumar M, Kumar M (2021) 2D object recognition: a comparative analysis of SIFT, SURF and ORB feature descriptors. Multimed Tools Appl 80(12):18839–18857

    Article  Google Scholar 

  8. Byrnes A, Sturton C (2018) On using drivers’ eyes to predict accident-causing drowsiness levels. In: 2018 21St international conference on intelligent transportation systems (ITSC). IEEE, pp 2092–2097

  9. Celona L, Mammana L, Bianco S, Schettini R (2018) A multi-task cnn framework for driver face monitoring. In: 2018 IEEE 8Th international conference on consumer electronics-berlin (ICCE-berlin). IEEE, pp 1–4

  10. Chen S, Wang Z, Chen W (2021) Driver drowsiness estimation based on factorized bilinear feature fusion and a long-short-term recurrent convolutional network. Information 12(1):3

    Article  Google Scholar 

  11. Chhabra P, Garg NK, Kumar M (2020) Content-based image retrieval system using ORB and SIFT features. Neural Comput Applic 32(7):2725–2733

    Article  Google Scholar 

  12. Chollet F (2017) Xception: Deep learning with depthwise separable convolutions. In: 2017 IEEE Conference on computer vision and pattern recognition (CVPR), pp 1800–1807

  13. Chowdhury A, Shankaran R, Kavakli M, Haque MM (2018) Sensor applications physiological features in drivers’ drowsiness detection: A review. IEEE Sensors J 18(8):3055–3067

    Article  Google Scholar 

  14. Donahue J, Hendricks LA, Guadarrama S, Rohrbach M, Venugopalan S, Saenko K, Darrell T (2015) Long-term recurrent convolutional networks for visual recognition and description. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2625–2634

  15. Dua M, Singla R, Raj S, Jangra A et al (2020) Deep cnn models-based ensemble approach to driver drowsiness detection. Neural Comput Applic, 1–14

  16. Gershon P, Shinar D, Oron-Gilad T, Parmet Y, Ronen A (2011) Usage and perceived effectiveness of fatigue countermeasures for professional and nonprofessional drivers. Accident Analysis & Prevention 43(3):797–803

    Article  Google Scholar 

  17. Ghoddoosian R, Galib M, Athitsos V (2019) A realistic dataset and baseline temporal model for early drowsiness detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 0–0

  18. Guo J-M, Markoni H (2019) Driver drowsiness detection using hybrid convolutional neural network and long short-term memory. Multimed Tools Appl 78 (20):29059–29087

    Article  Google Scholar 

  19. Hachisuka S (2013) Human and vehicle-driver drowsiness detection by facial expression. In: 2013 International conference on biometrics and kansei engineering. IEEE, pp 320–326

  20. Husain SS, Ong EJ, Bober M (2019) ACTNET: End-to-end learning of feature activations and multi-stream aggregation for effective instance image retrieval coRR

  21. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE Conference on computer vision and pattern recognition (CVPR), pp 770–778

  22. Ibrahim LF, Abulkhair M, AlShomrani AD, Manal A-G, Ameerah A-M, Fadiah A-G et al (2014) Using haar classifiers to detect driver fatigue and provide alerts. Multimedia Tools and Applications 71(3):1857–1877

    Article  Google Scholar 

  23. Jamshidi S, Azmi R, Sharghi M, Soryani M (2021) Hierarchical deep neural networks to detect driver drowsiness. Multimed Tools Appl, 1–14

  24. Kaida K, Takahashi M, Åkerstedt T, Nakata A, Otsuka Y, Haratani T, Fukasawa K (2006) Validation of the karolinska sleepiness scale against performance and eeg variables. Clin Neurophysiol 117(7):1574–1581

    Article  Google Scholar 

  25. Katyal Y, Alur S, Dwivedi S (2014) Safe driving by detecting lane discipline and driver drowsiness. In: 2014 IEEE International conference on advanced communications, control and computing technologies. IEEE, pp 1008–1012

  26. Khessiba S, Blaiech AG, Ben Khalifa K, Ben Abdallah A, Bedoui MH (2020) Innovative deep learning models for EEG-based vigilance detection. Neural Comput Applic, 1–17

  27. King DE (2009) Dlib-ml: A machine learning toolkit. The Journal of Machine Learning Research 10:1755–1758

    Google Scholar 

  28. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  29. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  30. Kumar M, Chhabra P, Garg NK (2018) An efficient content based image retrieval system using BayesNet and k-NN. Multimed Tools Appl 77 (16):21557–21570

    Article  Google Scholar 

  31. Kumar A, Kaur A, Kumar M (2019) Face detection techniques: a review. Artif Intell Rev 52(2):927–948

    Article  Google Scholar 

  32. Kumar A, Kumar M, Kaur A (2021) Face detection in still images under occlusion and non-uniform illumination. Multimed Tools Appl 80(10):14565–14590

    Article  Google Scholar 

  33. Kumar M, Kumar M et al (2021) XGBOost: 2D-object recognition using shape descriptors and extreme gradient boosting classifier. In: Computational methods and data engineering. Springer, pp 207–222

  34. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  35. Lin M, Chen Q, Yan S (2014) Network in network. In: International conference on learning representations, ICLR

  36. Maclean AW (2019) Sleep and driving. In: Handbook of behavioral neuroscience, vol 30. Elsevier, pp 611–622

  37. Malcangi M (2016) Applying evolutionary methods for early prediction of sleep onset. Neural Comput Applic 27(5):1165–1173

    Article  Google Scholar 

  38. Massoz Q, Langohr T, François C, Verly JG (2016) The ulg multimodality drowsiness database (called drozy) and examples of use. In: 2016 IEEE Winter conference on applications of computer vision (WACV). IEEE, pp 1–7

  39. Mehreen A, Anwar SM, Haseeb M, Majid M, Ullah MO (2019) A hybrid scheme for drowsiness detection using wearable sensors. IEEE Sensors J 19(13):5119–5126

    Article  Google Scholar 

  40. National safety Council. Accessed: 09-03-2020

  41. Ngxande M, Tapamo J-R, Burke M (2017) Driver drowsiness detection using behavioral measures and machine learning techniques: A review of state-of-art techniques. In: 2017 Pattern recognition association of South Africa and robotics and mechatronics (PRASA-robmech). IEEE, pp 156–161

  42. Park S, Pan F, Kang S, Yoo CD (2016) Driver drowsiness detection system based on feature representation learning using various deep networks. In: Asian conference on computer vision. Springer, pp 154–164

  43. Parkhi OM, Vedaldi A, Zisserman A (2015) Deep face recognition

  44. Radenovic F, Tolias G, Chum O (2018) Fine-tuning CNN image retrieval with no human annotation. IEEE Trans Pattern Anal Mach Intell, 1–1

  45. Ramzan M, Khan HU, Awan SM, Ismail A, Ilyas M, Mahmood A (2019) A survey on state-of-the-art drowsiness detection techniques. IEEE Access 7:61904–61919

    Article  Google Scholar 

  46. Reddy B, Kim Y-H, Yun S, Seo C, Jang J (2017) Real-time driver drowsiness detection for embedded system using model compression of deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 121–128

  47. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen L-C (2018) Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4510–4520

  48. Sharaff A, Nagwani NK (2020) ML-EC2: An algorithm for Multi-Label email classification using clustering. International Journal of Web-Based Learning and Teaching Technologies (IJWLTT) 15(2):19–33

    Article  Google Scholar 

  49. Sharaff A, Nagwani NK, Dhadse A (2016) Comparative study of classification algorithms for spam email detection. In: Emerging research in computing, information, communication and applications. Springer, pp 237–244

  50. Shih T-H, Hsu C-T (2016) Mstn: Multistage spatial-temporal network for driver drowsiness detection. In: Asian conference on computer vision. Springer, pp 146–153

  51. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  52. Singh S, Ahuja U, Kumar M, Kumar K, Sachdeva M (2021) Face mask detection using YOLOv3 and faster R-CNN models: COVID-19 environment. Multimed Tools Appl, 1–16

  53. Szegedy C, Ioffe S, Vanhoucke V, Alemi AA (2017) Inception-v4, inception-resnet and the impact of residual connections on learning. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pp 4278–4284

  54. Tolias G, Sicre R, Jégou H (2015) Particular object retrieval with integral max-pooling of CNN activations coRR

  55. Wang Y, Huang R, Guo L (2019) Eye gaze pattern analysis for fatigue detection based on gp-bcnn with esm. Pattern Recogn Lett 123:61–74

    Article  Google Scholar 

  56. Weng C-H, Lai Y-H, Lai S-H (2016) Driver drowsiness detection via a hierarchical temporal deep belief network. In: Asian conference on computer vision. Springer, pp 117–133

  57. Wijnands JS, Thompson J, Nice KA, Aschwanden GDPA, Stevenson M (2019) Real-time monitoring of driver drowsiness on mobile platforms using 3d neural networks. Neural Comput Applic, 1–13

  58. World Health Organization et al (2018) Global status report on road safety 2018. Technical report, World Health Organization

  59. Xie S, Girshick R, Dollár P, Tu Z, He K (2017) Aggregated residual transformations for deep neural networks. In: 2017 IEEE Conference on computer vision and pattern recognition (CVPR), pp 5987–5995

  60. Yu J, Park S, Lee S, Jeon M (2018) Driver drowsiness detection using condition-adaptive representation learning framework. IEEE Trans Intell Transp Syst 20(11):4206–4218

    Article  Google Scholar 

  61. Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters 23(10):1499–1503

    Article  Google Scholar 

  62. Zhao L, Wang Z, Zhang G, Gao H (2020) Driver drowsiness recognition via transferred deep 3d convolutional network and state probability vector. Multimed Tools Appl 79(35):26683–26701

    Article  Google Scholar 

  63. Zhenhai G, DinhDat L, Hongyu H, Ziwen Y, Xinyu W (2017) Driver drowsiness detection based on time series analysis of steering wheel angular velocity. In: 2017 9Th international conference on measuring technology and mechatronics automation (ICMTMA). IEEE, pp 99–101

Download references

Author information

Authors and Affiliations

  1. Centre for Vision, Speech, Signal Processing, University of Surrey, Guildford, UK

    Syed Sameed Husain

  2. Department of Electrical Engineering, University of Engineering and Technology Taxila, 47050, Taxila, Pakistan

    Junaid Mir & Muhammad Obaid Ullah

  3. Department of Computer Engineering, University of Engineering and Technology Taxila, 47050, Taxila, Pakistan

    Syed Muhammad Anwar

  4. Department of Engineering Science, University of Oxford, Oxford, UK

    Waqas Rafique

Authors
  1. Syed Sameed Husain
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Junaid Mir
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Syed Muhammad Anwar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Waqas Rafique
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Muhammad Obaid Ullah
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Junaid Mir.

Ethics declarations

Conflict of Interests

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Husain, S.S., Mir, J., Anwar, S.M. et al. Development and validation of a deep learning-based algorithm for drowsiness detection in facial photographs. Multimed Tools Appl 81, 20425–20441 (2022). https://doi.org/10.1007/s11042-022-12433-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12433-x

Keywords

Search

Navigation