Skip to main content
SpringerLink
Log in

An approach to occluded face recognition based on dynamic image-to-class warping using structural similarity index

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Face recognition in uncontrolled environments is a challenging problem in computer vision due to occlusion, pose, and illumination changes. While machine learning techniques address occluded face recognition, they require retraining when updating gallery images. The Dynamic Image-to-Class Warping (DICW) technique offers real-time recognition without training, maintaining the natural order of facial features (forehead, eyes, nose, mouth, and chin) to avoid disruptions caused by occlusion. DICW separates face image patches and integrates them into an ordered sequence through raster scanning. It computes the image-to-class distance between query and target faces using optimal warping paths along temporal and within-class dimensions. This paper proposes an improved face recognition approach using DICW and the Structural SIMilarity (SSIM) index, mitigating variations in illumination and contrast to match structural information. A technique for automatic face recognition from video sequences with DICW is also presented. Experiments on the AR Face Database, Chokepoint Database, and uncontrolled environment video sequences show that the proposed approach significantly improves the recognition rates for occluded images. The proposed approach achieved an improvement of around 5-6% in all considered cases compared to other state-of-the-art approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Algorithm 1
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data Availibility Statement

The datasets generated during and/or analyzed during the current study are available in the GitHub repository, https://github.com/NLPaSE2020/Video-Sequence-Data.

Notes

  1. http://www.imagemagick.org/Magick++/ Documentation.html

  2. AR dataset:https://web.mit.edu/emeyers/www/face _databases.html

  3. https://github.com/NLPaSE2020/Video-Sequence-Data

References

  1. Golwalkar R, N Mehendale (2022) Masked-face recognition using deep metric learning and facemasknet-21. Appl Intell 1–12

  2. R Szeliski (2022) Computer vision: algorithms and applications. Springer Nature

  3. Jin M, Yu L, Zhou K, Yi Q (2022) Occlusion tolerant object recognition using visual memory selection model. Appl Intell 1–25

  4. Li N, X Zhao (2022) A multi-modal dataset for gait recognition under occlusion. Appl Intell 1–18

  5. P Viola, M Jones (2001) Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001, volume 1. Ieee, p I–I

  6. M Storer, M Urschler, H Bischof (2010) Occlusion detection for icao compliant facial photographs. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops. IEEE, 122–129

  7. D Lin, X Tang (2007) Quality-driven face occlusion detection and recovery. In: 2007 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, p 1–7

  8. R Min, A Hadid, J-L Dugelay (2011) Improving the recognition of faces occluded by facial accessories. In: 2011 IEEE international conference on automatic face & gesture recognition (FG). IEEE, p 442–447

  9. H Jia, AM Martinez (2008) Face recognition with occlusions in the training and testing sets. In: 2008 8th IEEE international conference on automatic face & gesture recognition. IEEE, p 1–6

  10. X Wei, C-T Li, Y Hu (2012) Robust face recognition under varying illumination and occlusion considering structured sparsity. In: 2012 International Conference on Digital Image Computing Techniques and Applications (DICTA). IEEE, p 1–7

  11. Liao S, Jain AK, Li SZ (2012) Partial face recognition: Alignment-free approach. IEEE Trans Pattern Anal Mach Intell 35(5):1193–1205

    Article  Google Scholar 

  12. Tan X, Chen S, Zhou Z-H (2009) J Liu Face recognition under occlusions and variant expressions with partial similarity. IEEE Trans Inf Forensics Secur 4(2):217–230

    Article  Google Scholar 

  13. Martinez AM (2002) Recognizing imprecisely localized partially occluded, and expression variant faces from a single sample per class. IEEE Trans Pattern Anal Mach Intell 24(6):748–763

    Article  Google Scholar 

  14. Tan X, Chen S, Zhou Z-H (2005) F Zhang Recognizing partially occluded, expression variant faces from single training image per person with som and soft k-nn ensemble. IEEE Trans Neural Netw 16(4):875–886

    Article  Google Scholar 

  15. Wei X, Li C-T, Hu Y (2013) Face recognition with occlusion using dynamic image-to-class warping (dicw)

  16. X Wei, C-T Li, Y Hu (2013) Robust face recognition with occlusions in both reference and query images. In: 2013 International Workshop on Biometrics and Forensics(IWBF). IEEE p 1–4

  17. Tzimiropoulos G, Zafeiriou S, Pantic M (2012) Subspace learning from image gradient orientations. IEEE Trans Pattern Anal Mach Intell 34(12):2454–2466

    Article  Google Scholar 

  18. Zhu J, Cao D, Liu S, Lei Z, Li SZ (2012) Discriminant analysis with gabor phase for robust face recognition. In: 2012 5th IAPR International Conference on Biometrics (ICB). IEEE, p 13–18

  19. A Morelli Andrés, S Padovani, M Tepper, J Jacobo-Berlles. Face recognition on partially occluded images using compressed sensing. Pattern Recognit Lett 36:235–242, 2014

  20. Ramamonjisoa M, Du Y, Lepetit V (2020) Predicting sharp and accurate occlusion boundaries in monocular depth estimation using displacement fields. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. p 14648–14657

  21. Huang S-M, Yang J-F (2012) Subface hidden markov models coupled with a universal occlusion model for partially occluded face recognition. IET Biometrics 1(3):149–159

    Article  Google Scholar 

  22. G Suvarna Kumar, P Reddy, M Srinadh Swamy, S Gupta, et al. Skin based occlusion detection and face recognition using machine learning techniques. Int J Comput Appl 41(18), 2012

  23. Yuan Y, Chu J, Leng L, Miao J, Kim B-G (2020) A scale-adaptive object-tracking algorithm with occlusion detection. EURASIP J Image Video Process 2020:1–15

    Article  Google Scholar 

  24. He M, Zhang J, Shan S, Kan M, Chen X (2020) Deformable face net for pose invariant face recognition. Pattern Recognit 100:107113

    Article  Google Scholar 

  25. Amutha R et al. (2012) A novel approach to face recognition under various facial expressions, occlusion and tilt angles. In: 2012 International Conference on Emerging Trends in Science, Engineering and Technology (INCOSET). IEEE, p 143–149

  26. Venkat I, Khader AT, Subramanian KG, De Wilde P (2013) Recognizing occluded faces by exploiting psychophysically inspired similarity maps. Pattern Recognit Lett 34(8):903–911

    Article  Google Scholar 

  27. Lin J, Ming J, Crookes D (2011) Robust face recognition with partial occlusion, illumination variation and limited training data by optimal feature selection. IET Comput Vis 5(1):23–32

    Article  MathSciNet  Google Scholar 

  28. A Vijayalakshmi, P Raj. An efficient method to recognize human faces from video sequences with occlusion. World Comput Sci Inform Technol J 5(2), 2015

  29. Wei X, Li C-T, Hu Y (2013) Face recognition with occlusion using dynamic image-to-class warping (dicw). In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG). IEEE, p 1–6

  30. Ding R-X, Huang H, Shang J (2015) Patch-based locality-enhanced collaborative representation for face recognition. IET Image Process 9(3):211–217

    Article  Google Scholar 

  31. Martinez AM (2002) Recognizing imprecisely localized, partially occluded, and expression variant faces from a single sample per class. IEEE Trans Pattern Anal Mach Intell 24(6):748–763

    Article  Google Scholar 

  32. Jiang X, Lai J (2014) Sparse and dense hybrid representation via dictionary decomposition for face recognition. IEEE Trans Pattern Anal Mach Intell 37(5):1067–1079

    Article  Google Scholar 

  33. Liao S, Jain AK, Li SZ (2012) Partial face recognition: Alignment-free approach. IEEE Trans Pattern Anal Mach Intell 35(5):1193–1205

    Article  Google Scholar 

  34. Deng W, Hu J, Guo J (2012) Extended src: Undersampled face recognition via intraclass variant dictionary. IEEE Trans Pattern Anal Mach Intell 34(9):1864–1870

    Article  Google Scholar 

  35. He R, Zheng W-S, Hu B-G (2010) Maximum correntropy criterion for robust face recognition. IEEE Trans Pattern Anal Mach Intell 33(8):1561–1576

    Google Scholar 

  36. N Zhang, M Paluri, Y Taigman, R Fergus, L Bourdev (2015) Beyond frontal faces: Improving person recognition using multiple cues. In: Proceedings of the IEEE conference on computer vision and pattern recognition. p 4804–4813

  37. M Yang, L Zhang, J Yang, D Zhang (2011) Robust sparse coding for face recognition. In: CVPR 2011. IEEE, p 625–632

  38. Wu CY, Ding JJ (2018) Occluded face recognition using low-rank regression with generalized gradient direction. Pattern Recognit 80:256–268

    Article  Google Scholar 

  39. Cen F, Zhao X, Li W, Wang G (2021) Deep feature augmentation for occluded image classification. Pattern Recognit 111:107737

    Article  Google Scholar 

  40. Zheng W, Gou C, Wang F-Y (2020) A novel approach inspired by optic nerve characteristics for few-shot occluded face recognition. Neurocomputing 376:25–41

    Article  Google Scholar 

  41. Ou W, Luan X, Gou J, Zhou Q, Xiao W, Xiong X, Zeng W (2018) Robust discriminative nonnegative dictionary learning for occluded face recognition. Pattern Recognit Lett 107:41–49

    Article  Google Scholar 

  42. Cen F, Wang G (2019) Dictionary representation of deep features for occlusion-robust face recognition. IEEE Access 7:26595–26605

    Article  Google Scholar 

  43. H Qiu, D Gong, Z Li, W Liu, D Tao (2021) End2end occluded face recognition by masking corrupted features. IEEE Trans Pattern Anal Mach Intell

  44. Li W, Chai Z, Tang Z (2021) A decomposition-based multi-objective immune algorithm for feature selection in learning to rank. Knowl-Based Syst 234:107577

    Article  Google Scholar 

  45. Yu X, Tian X (2022) A fault detection algorithm for pipeline insulation layer based on immune neural network. Int J Press Vessels Pip 196:104611

    Article  Google Scholar 

  46. Yu X, Ye X, Zhang S (2022) Floating pollutant image target extraction algorithm based on immune extremum region. Dig Signal Process 123:103442

    Article  Google Scholar 

  47. R Spijkerman (2021) Immune inspired face recognition utilizing anomaly detection and symbiotic agents. University of Johannesburg (South Africa)

  48. R Zhang, P Isola, Efros AA, Shechtman E, O Wang (2018) The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE conference on computer vision and pattern recognition. p 586–595

  49. Li Q, He H, Lai H, Cai T, Wang Q, Gao QX (2022) Enhanced nuclear norm based matrix regression for occluded face recognition. Pattern Recognit 126:108585

    Article  Google Scholar 

  50. He M, Zhang J, Shan S, Liu X, Wu Z, Chen X (2021) Locality-aware channel-wise dropout for occluded face recognition. IEEE Trans Image Process 31:788–798

    Article  Google Scholar 

  51. Wei X, Li C-T, Lei Z, Yi D, Li SZ (2014) Dynamic image-to-class warping for occluded face recognition. IEEE Trans Inform Forens Sec 9(12):2035–2050

    Article  Google Scholar 

  52. Azimipour M, Zawadzki RJ, Gorczynska I, Migacz J, Werner JS, Jonnal RS (2018) Intraframe motion correction for raster-scanned adaptive optics images using strip-based cross-correlation lag biases. PLoS ONE 13(10):e0206052

    Article  Google Scholar 

  53. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  54. Ahonen T, Hadid A, Pietikainen M (2006) Face description with local binary patterns: Application to face recognition. IEEE Trans Pattern Anal Mach Intellig 28(12):2037–2041

    Article  MATH  Google Scholar 

  55. Y Wong, S Chen, S Mau, C Sanderson, BC Lovell (2011) Patch-based probabilistic image quality assessment for face selection and improved video-based face recognition. In: CVPR 2011 WORKSHOPS. IEEE, p 74–81

  56. A Martinez, R Benavente (1998) The ar face database: Cvc technical report, 24

  57. M Lal, K Kumar, RH Arain, A Maitlo, SA Ruk, H Shaikh. Study of face recognition techniques: A survey. Int J Adv Comput Sci Appl 9(6), 2018

  58. AL Machidon, OM Machidon, PL Ogrutan (2019) Face recognition using eigenfaces, geometrical pca approximation and neural networks. In: 2019 42nd international conference on telecommunications and signal processing (TSP). IEEE, p 80–83

  59. D Duisembayev (2022) Feasibility of face recognition algorithms on edge

  60. Hassaballah M, Alshazly HA, Ali AA (2019) Ear recognition using local binary patterns: A comparative experimental study. Expert Syst Appl 118:182–200

    Article  Google Scholar 

  61. Kiani K, Rezaeirad S, Rastgoo R (2021) Hmm-based face recognition using svd and half of the face image. Model Simul Electr Electron Eng 1(2):45–50

    Google Scholar 

  62. Hu L, Cui J (2019) Digital image recognition based on fractional-order-pca-svm coupling algorithm. Measurement 145:150–159

    Article  Google Scholar 

  63. Cheng E-J, Chou K-P, Rajora S, Jin B-H, Tanveer M, Lin C-T, Young K-Y, Lin W-C, Prasad M (2019) Deep sparse representation classifier for facial recognition and detection system. Pattern Recognit Lett 125:71–77

    Article  Google Scholar 

  64. Chen M-J, Bovik AC (2011) Fast structural similarity index algorithm. J Real Time Image Process 6(4):281–287

    Article  Google Scholar 

  65. Zeng D, Veldhuis R, Spreeuwers L (2021) A survey of face recognition techniques under occlusion. IET Biometric 10(6):581–606

    Article  Google Scholar 

Download references

Acknowledgements

The authors are thankful to the editor and the anonymous reviewers for their valuable comments that helped in the improvement of the paper.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Indian Institute of Technology, Roorkee, India

    Shadab Naseem, Sandeep Kumar, Sugata Gangopadhyay & Ankita Jain

  2. Department of Computer Science and Engineering, ABV-IIITM, Gwalior, India

    Santosh Singh Rathore

Authors
  1. Shadab Naseem
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Santosh Singh Rathore
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sandeep Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sugata Gangopadhyay
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ankita Jain
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sandeep Kumar.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

All authors contributed equally and are joint first authors.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Naseem, S., Rathore, S.S., Kumar, S. et al. An approach to occluded face recognition based on dynamic image-to-class warping using structural similarity index. Appl Intell 53, 28501–28519 (2023). https://doi.org/10.1007/s10489-023-05026-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-05026-0

Keywords

Search

Navigation