Skip to main content
SpringerLink
Log in

Heart rate estimation from facial videos using nonlinear mode decomposition and improved consistency check

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Remote photoplethysmography (rPPG) is a non-contact and noninvasive way of measuring human physiological signals such as the heart rate using the subtle color changes of skin regions. Since the face of a person is generally visible, facial videos can be used for estimating the heart rate remotely. The rigid and non-rigid motions of the face and illumination variations are the main challenges that affect the accuracy of heart rate estimation. In this paper, we present a new method for estimating the heart rate of a person from the skin region of the facial video using nonlinear mode decomposition (NMD), which is a recently proposed blind source separation method and has been shown to be more robust to noise. We also propose a new method (history-based consistency check—HBCC) for selecting the best heart rate candidate after decomposition by minimizing a temporal cost function. Experiments on two datasets show that the proposed method (rPPG-NMD) achieves promising results as compared to several the state-of-the-art methods for rPPG-based heart rate estimation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Verkruysse, W., et al.: Remote plethysmographic imaging using ambient light. Opt. Exp. 16(26), 21434–21445 (2008)

    Article  Google Scholar 

  2. Poh, M., et al.: Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt. Exp. 18(10), 10762–10774 (2010)

    Article  Google Scholar 

  3. Hertzman, A.B.: Observations on the finger volume pulse recorded photoelectrically. Am. J. Physiol. 119, 334–335 (1937)

    Google Scholar 

  4. Iatsenko, D., et al.: Nonlinear mode decomposition: a noise-robust, adaptive decomposition method. Phys. Rev. E 92(3), 032916 (2015)

    Article  MathSciNet  Google Scholar 

  5. Demirezen, H., Erdem, C.E.: Remote photoplethysmography using nonlinear mode decomposition. In: IEEE ICASSP, pp. 1060–1064 (2018)

  6. Stricker, R., et al.: Non-contact video-based pulse rate measurement on a mobile service robot. In: IEEE Int. Symp, RHIC (2014)

  7. Bobbia, S., et al.: Remote photoplethysmography based on implicit living skin tissue segmentation. In: ICPR(2016)

  8. Li, X., et al.: Remote heart rate measurement from face videos under realistic situations. In: IEEE CVPR, pp. 4264–4271 (2014)

  9. Po, L.M., et al.: Block-based adaptive ROI for remote photoplethysmography. Multimed. Tools Appl. 77, 6503–6529 (2018)

    Article  Google Scholar 

  10. Bobbia, S., et al.: Real-time temporal superpixels for unsupervised remote photoplethysmography. In: IEEE CVPRW, pp. 1341–1348 (2018)

  11. Li, P., et al.: Model-based region of interest segmentation for remote photoplethysmography. In: CVTA (2019)

  12. Cardoso, J.F.: High-order contrasts for independent component analysis. Neural Comput. 11(1), 157–192 (1999)

    Article  Google Scholar 

  13. Poh, M., et al.: Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Trans. Biomed. Eng. 58, 7–11 (2010)

  14. Macwan, R., et al.: Remote photoplethysmography measurement using constrained ICA. In: IEEE EHB (2017)

  15. Haan, G.D., Jeanne, V.: Robust pulse rate from chrominance-based rPPG. In: IEEE TBE (2013)

  16. Balakrishnan, G., et al. Detecting pulse from head motions in video. In: IEEE CVPR (2013)

  17. Lewandowska, M., et al.: Measuring pulse rate with a webcam—a non-contact method for evaluating cardiac activity. In: Federated Conf. on Computer Sci. and Inf. Sys. (2011)

  18. Chen, D.Y., et al.: Image sensor-based heart rate evaluation from face reflectance using Hilberthuang transform. IEEE Sens. J. 15, 618–627 (2015)

    Article  Google Scholar 

  19. Wu, Z., Huang, N.E.: Ensemble empirical mode decomposition: a noise-assisted data analysis method. AADA 1(01), 1–41 (2009)

    Google Scholar 

  20. Song, R., et al.: New insights on super-high resolution for video-based heart rate estimation with a semi-blind source separation method. Comput. Biol. Med. 116, 103535 (2020)

    Article  Google Scholar 

  21. Huang, P.-W., et al.: A heart rate monitoring framework for real-world drivers using remote photoplethysmography. In: IEEE J-BHI (2020)

  22. Kumar, M., et al.: Distanceppg: robust non-contact vital signs monitoring using a camera. Biomed. Opt. Exp. 6, 1565–1588 (2015)

    Article  Google Scholar 

  23. Wang, W., et al.: A novel algorithm for remote photoplesthymograpy: spatial subspace rotation. IEEE Trans. Biomed. Eng. 63, 1974–1984 (2015)

    Article  Google Scholar 

  24. Monkaresi, H., et al.: A machine learning approach to improve contactless heart rate monitoring using a webcam. IEEE J. Biomed. Health Inf. 18(4), 1153–1160 (2013)

    Article  Google Scholar 

  25. Osman, A., et al.: Supervised learning approach to remote heart rate estimation from facial videos. In: IEEE FG, vol. 1 (2015)

  26. Chen, W., McDuff, D.: DeepPhys: video-based physiological measurement using convolutional attention networks. In: ECCV (2018)

  27. Niu, X., Han, H., Shan, S., Chen, X.: SynRhythm: learning a deep heart rate estimator from general to specific. In: ICPR (2018)

  28. Niu, X., et al.: VIPL-HR: A multi-modal database for pulse estimation from less-constrained face video. In: ACCV (2018)

  29. Hsu, G.-S., et al.: Deep learning with time-frequency representation for pulse estimation from facial videos. In: IEEE IJCB (2017)

  30. Song, R., et al.: Heart rate estimation from facial videos using a spatiotemporal representation with convolutional neural networks. In: IEEE TIM (2020)

  31. Botina-Monsalve, D., et al.: Long short-term memory deep-filter in remote photoplethysmography. In: IEEE CVPRW, pp. 306–307 (2020)

  32. Lee, E., et al.: Meta-rppg: remote heart rate estimation using a transductive meta-learner. In: ECCV, pp. 392–409. Springer, Berlin (2020)

  33. Huang, N.E., et al.: The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis. Proc. R. Soc. Lond. A Math. Phys. Eng, Sci. 454, 903–995 (1998)

  34. Viola, P., et al.: Rapid object detection using a boosted cascade of simple features. In: IEEE CVPR (2001)

  35. Bay, H., et al.: Speeded-up robust features (SURF). CVIU 110(3), 346–359 (2008)

    Google Scholar 

  36. Tomasi, C., Kanade, T.: Detection and tracking of point features. Carnegie Mellon University, Tech. Rep. MU-CS-91-132 (1991)

  37. Conaire, C.O., et al.: Detector adaptation by maximising agreement between independent data sources. In: IEEE CVPR, pp. 1–6 (2007)

  38. PURE dataset (2020). https://www.tu-ilmenau.de/en/neurob/data-sets-code/pulse/

  39. Bob’s rPGG library (2020). https://www.idiap.ch/software/bob/docs/bob/bob.rppg.base/master/index.html

  40. Špetlík, R., Franc, V., Matas, J.: Visual heart rate estimation with convolutional neural network. In: BMVC (2018)

  41. Bousefsaf, F., et al.: 3D convolutional neural networks for remote pulse rate measurement and mapping from facial video. Appl. Sci. 9, 4364 (2019)

Download references

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Marmara University, Istanbul, Turkey

    Halil Demirezen & Cigdem Eroglu Erdem

Authors
  1. Halil Demirezen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cigdem Eroglu Erdem
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Halil Demirezen.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Demirezen, H., Eroglu Erdem, C. Heart rate estimation from facial videos using nonlinear mode decomposition and improved consistency check. SIViP 15, 1415–1423 (2021). https://doi.org/10.1007/s11760-021-01873-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-021-01873-x

Keywords

Search

Navigation