Skip to main content
SpringerLink
Log in

Deep Learning in Face Recognition Across Variations in Pose and Illumination

  • Chapter
Deep Learning in Object Detection and Recognition

Abstract

Even though face recognition in frontal view and normal lighting conditions works very well, the performance drops sharply in extreme conditions. Recently there is plenty of work dealing with pose and illumination problems, respectively. However both the lighting and pose variations always happen simultaneously in general conditions, and consequently we propose an end-to-end face recognition algorithm to deal with two variations at the same time based on convolutional neural networks. In order to achieve better performance, we extract discriminative nonlinear features that are invariant to pose and illumination. We propose to use the 1 × 1 convolutional kernels to extract the local features. Furthermore a parallel multi-stream convolutional neural network is developed to extract multi-hierarchy features which are more efficient than single-scale features. In the experiments we obtain the average face recognition rate of 96.9% on MultiPIE dataset. Even for profile position, the average recognition rate is also around 98.5% in different lighting conditions, which improves the state-of-the-art face recognition across poses and illumination by 7.5%.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 179.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 179.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Ahonen, T., Hadid, A., Pietikainen, M.: Face description with local binary patterns: Application to face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(12), 2037–2041 (2006). DOI 10.1109/TPAMI.2006.244

    Google Scholar 

  2. Ahonen, T., Hadid, A., Pietikainen, M.: Face description with local binary patterns: Application to face recognition. IEEE transactions on pattern analysis and machine intelligence 28(12), 2037–2041 (2006)

    Article  Google Scholar 

  3. Andrew, G., Arora, R., Bilmes, J., Livescu, K.: Deep canonical correlation analysis. In: International Conference on Machine Learning, pp. 1247–1255 (2013)

    Google Scholar 

  4. Ashraf, A.B., Lucey, S., Chen, T.: Learning patch correspondences for improved viewpoint invariant face recognition. In: Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, pp. 1–8. IEEE (2008)

    Google Scholar 

  5. Basri, R., Jacobs, D.W.: Lambertian reflectance and linear subspaces. IEEE Transactions on Pattern Analysis and Machine Intelligence 25(2), 218–233 (2003)

    Article  Google Scholar 

  6. Bay, H., Ess, A., Tuytelaars, T., Van Gool, L.: Speeded-up robust features (surf). Computer vision and image understanding 110(3), 346–359 (2008)

    Article  Google Scholar 

  7. Belhumeur, P.N., Kriegman, D.J.: What is the set of images of an object under all possible illumination conditions? International Journal of Computer Vision 28(3), 245–260 (1998)

    Article  Google Scholar 

  8. Biswas, S., Aggarwal, G., Flynn, P.J., Bowyer, K.W.: Pose-robust recognition of low-resolution face images. IEEE transactions on pattern analysis and machine intelligence 35(12), 3037–3049 (2013)

    Article  Google Scholar 

  9. Castillo, C.D., Jacobs, D.W.: Using stereo matching with general epipolar geometry for 2d face recognition across pose. IEEE Transactions on Pattern Analysis and Machine Intelligence 31(12), 2298–2304 (2009)

    Article  Google Scholar 

  10. Chen, D., Cao, X., Wen, F., Sun, J.: Blessing of dimensionality: High-dimensional feature and its efficient compression for face verification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3025–3032 (2013)

    Google Scholar 

  11. Ding, C., Choi, J., Tao, D., Davis, L.S.: Multi-directional multi-level dual-cross patterns for robust face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 38(3), 518–531 (2016). DOI 10.1109/TPAMI.2015.2462338

    Google Scholar 

  12. Ding, C., Tao, D.: Pose-invariant face recognition with homography-based normalization. Pattern Recognition 66, 144–152 (2017)

    Article  Google Scholar 

  13. Ding, C., Xu, C., Tao, D.: Multi-task pose-invariant face recognition. IEEE Transactions on Image Processing 24(3), 980–993 (2015)

    Article  MathSciNet  Google Scholar 

  14. Ding, C., Xu, C., Tao, D.: Multi-task pose-invariant face recognition. IEEE Transactions on Image Processing 24(3), 980–93 (2015)

    Article  MathSciNet  Google Scholar 

  15. Gao, Y., Leung, M.K.: Face recognition using line edge map. IEEE transactions on pattern analysis and machine intelligence 24(6), 764–779 (2002)

    Article  Google Scholar 

  16. Gross, R., Matthews, I., Cohn, J., Kanade, T., Baker, S.: Multi-pie. Image and Vision Computing 28(5), 807–813 (2010)

    Article  Google Scholar 

  17. Gunther, M., Costa-Pazo, A., Ding, C., Boutellaa, E.: The 2013 face recognition evaluation in mobile environment. 2013 International Conference on Biometrics (ICB) pp. 1–7 (2013)

    Google Scholar 

  18. He, X., Yan, S., Hu, Y., Niyogi, P., Zhang, H.J.: Face recognition using laplacian faces. IEEE transactions on pattern analysis and machine intelligence 27(3), 328–340 (2005)

    Article  Google Scholar 

  19. Ho, H.T., Chellappa, R.: Pose-invariant face recognition using markov random fields. IEEE transactions on image processing 22(4), 1573–1584 (2013)

    Article  MathSciNet  Google Scholar 

  20. Hu, J., Lu, J., Tan, Y.P.: Discriminative deep metric learning for face verification in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1875–1882 (2014)

    Google Scholar 

  21. Huang, S.C., Cheng, F.C., Chiu, Y.S.: Efficient contrast enhancement using adaptive gamma correction with weighting distribution. IEEE Transactions on Image Processing 22(3), 1032–1041 (2013)

    Article  MathSciNet  Google Scholar 

  22. Jiang, X., Cheng, Y., Xiao, R., Li, Y., Zhao, R.: Spherical harmonic based linear face de-lighting and compensation. Applied Mathematics and Computation 185(2), 857–868 (2007). https://doi.org/10.1016/j.amc.2006.06.090. http://www.sciencedirect.com/science/article/pii/S0096300306007673. Special Issue on Intelligent Computing Theory and Methodology

  23. Jiang, X., Feng, X., Wu, J., Peng, J.: Lighting alignment for image sequences. In: International Conference on Image and Graphics, pp. 462–474. Springer (2015)

    Google Scholar 

  24. Jiang, X., Kong, Y.O., Huang, J., Zhao, R., Zhang, Y.: Learning from real images to model lighting variations for face images. In: European Conference on Computer Vision (ECCV), pp. 284–297 (2008)

    Google Scholar 

  25. Kafai, M., An, L., Bhanu, B.: Reference face graph for face recognition. IEEE Transactions on Information Forensics and Security 9(12), 2132–2143 (2014)

    Article  Google Scholar 

  26. Kan, M., Shan, S., Chang, H., Chen, X.: Stacked progressive auto-encoders (spae) for face recognition across poses. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1883–1890 (2014)

    Google Scholar 

  27. Kan, M., Shan, S., Xilin., C.: Multi-view deep network for cross-view classification. In: IEEE Conference on Computer Vision and Pattern Recognition (2016)

    Google Scholar 

  28. Kan, M., Shan, S., Zhang, H., Lao, S., Chen, X.: Multi-view discriminant analysis. IEEE transactions on pattern analysis and machine intelligence 38(1), 188–194 (2016)

    Article  Google Scholar 

  29. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp. 1097–1105 (2012)

    Google Scholar 

  30. Lee, K.C., Ho, J., Kriegman, D.: Acquiring linear subspaces for face recognition under variable lighting. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(5), 684–698 (2005)

    Article  Google Scholar 

  31. Li, A., Shan, S., Chen, X., Gao, W.: Maximizing intra-individual correlations for face recognition across pose differences. In: Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pp. 605–611. IEEE (2009)

    Google Scholar 

  32. Li, A., Shan, S., Gao, W.: Coupled bias-variance tradeoff for cross-pose face recognition. IEEE Transactions on Image Processing 21(1), 305–15 (2012)

    Article  MathSciNet  Google Scholar 

  33. Li, H., Hua, G., Lin, Z., Brandt, J., Yang, J.: Probabilistic elastic matching for pose variant face verification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3499–3506 (2013)

    Google Scholar 

  34. Li, S., Liu, X., Chai, X., Zhang, H., Lao, S., Shan, S.: Morphable displacement field based image matching for face recognition across pose. European Conference on Computer Vision 2012 pp. 102–115 (2012)

    Google Scholar 

  35. Liao, Q., Leibo, J.Z., Poggio, T.: Learning invariant representations and applications to face verification. In: Advances in Neural Information Processing Systems, pp. 3057–3065 (2013)

    Google Scholar 

  36. Liao, S., Jain, A.K., Li, S.Z.: Partial face recognition: Alignment-free approach. IEEE Transactions on pattern analysis and machine intelligence 35(5), 1193–1205 (2013)

    Article  Google Scholar 

  37. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International journal of computer vision 60(2), 91–110 (2004)

    Article  Google Scholar 

  38. Majumdar, A., Singh, R., Vatsa, M.: Face recognition via class sparsity based supervised encoding. IEEE transactions on pattern analysis and machine intelligence (2016)

    Google Scholar 

  39. Masi, I., Rawls, S., Medioni, G., Natarajan, P.: Pose-aware face recognition in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4838–4846 (2016)

    Google Scholar 

  40. Parkhi, O.M., Vedaldi, A., Zisserman, A., et al.: Deep face recognition. In: British Machine Vision Conference, vol. 1, p. 6 (2015)

    Google Scholar 

  41. Peng, X., Yu, X., Sohn, K., Metaxas, D., Chandraker, M.: Reconstruction for feature disentanglement in pose-invariant face recognition. arXiv preprint arXiv:1702.03041 (2017)

    Google Scholar 

  42. Pentland, A., Moghaddam, B., Starner, T., et al.: View-based and modular eigenspaces for face recognition. In: CVPR, vol. 94, pp. 84–91 (1994)

    Google Scholar 

  43. Pizer, S.M., Amburn, E.P., Austin, J.D., Cromartie, R., Geselowitz, A., Greer, T., ter Haar Romeny, B., Zimmerman, J.B., Zuiderveld, K.: Adaptive histogram equalization and its variations. Computer vision, graphics, and image processing 39(3), 355–368 (1987)

    Article  Google Scholar 

  44. Prince, S.J., Elder, J.H., Warrell, J., Felisberti, F.M.: Tied factor analysis for face recognition across large pose differences. IEEE Transactions on pattern analysis and machine intelligence 30(6), 970–984 (2008)

    Article  Google Scholar 

  45. Ramamoorthi, R., Hanrahan, P.: On the relationship between radiance and irradiance: determining the illumination from images of a convex lambertian object. Journal of the Optical Society of America, A 18(10), 2448–2459 (2001)

    Article  MathSciNet  Google Scholar 

  46. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems (NIPS) pp. 91–99 (2015)

    Google Scholar 

  47. Rupnik, J., Shawe-Taylor, J.: Multi-view canonical correlation analysis. In: Conference on Data Mining and Data Warehouses (SiKDD 2010), pp. 1–4 (2010)

    Google Scholar 

  48. Rupnik, J., Shawe-Taylor, J.: Multi-view canonical correlation analysis. Conference on Data Mining and Data Warehouses(SiKDD 2010) pp. 1–4 (2010)

    Google Scholar 

  49. Schölkopf, B., Smola, A., Müller, K.R.: Nonlinear component analysis as a kernel eigenvalue problem. Neural computation 10(5), 1299–1319 (1998)

    Article  Google Scholar 

  50. Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: A unified embedding for face recognition and clustering. Computer Vision and Pattern Recognition pp. 815–823 (2015)

    Google Scholar 

  51. Schroff, F., Treibitz, T., Kriegman, D., Belongie, S.: Pose, illumination and expression invariant pairwise face-similarity measure via doppelgänger list comparison. In: Computer Vision (ICCV), 2011 IEEE International Conference on, pp. 2494–2501. IEEE (2011)

    Google Scholar 

  52. Sharma, A.: Generalized multiview analysis: A discriminative latent space. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2160–2167 (2012)

    Google Scholar 

  53. Sharma, A., Al Haj, M., Choi, J., Davis, L.S., Jacobs, D.W.: Robust pose invariant face recognition using coupled latent space discriminant analysis. Computer Vision and Image Understanding 116(11), 1095–1110 (2012)

    Article  Google Scholar 

  54. Sharma, A., Jacobs, D.W.: Bypassing synthesis: Pls for face recognition with pose, low-resolution and sketch. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 593–600 (2011)

    Google Scholar 

  55. Sharma, A., Kumar, A., Daume, H., Jacobs, D.W.: Generalized multiview analysis: A discriminative latent space. In: Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pp. 2160–2167. IEEE (2012)

    Google Scholar 

  56. Shashua, A., Riklin-Raviv, T.: The quotient image: Class-based re-rendering and recognition with varying illuminations. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(2), 129–139 (2001)

    Article  Google Scholar 

  57. Sun, Y., Wang, X., Tang, X.: Deeply learned face representations are sparse, selective, and robust. Computer Vision and Pattern Recognition

    Google Scholar 

  58. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

    Google Scholar 

  59. Taigman, Y., Yang, M., Ranzato, M., Wolf, L.: Deepface: Closing the gap to human-level performance in face verification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1701–1708 (2014)

    Google Scholar 

  60. Tan, X., Triggs, B.: Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE transactions on image processing 19(6), 1635–1650 (2010)

    Article  MathSciNet  Google Scholar 

  61. Tran, L., Yin, X., Liu, X.: Disentangled representation learning gan for pose-invariant face recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 3, p. 7 (2017)

    Google Scholar 

  62. Turk, M., Pentland, A.: Eigenfaces for recognition. Journal of cognitive neuroscience 3(1), 71–86 (1991)

    Article  Google Scholar 

  63. Wagner, A., Wright, J., Ganesh, A., Zhou, Z., Mobahi, H., Ma, Y.: Toward a practical face recognition system: robust alignment and illumination by sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence 34(2), 372–386 (2012)

    Article  Google Scholar 

  64. Wang, H., Li, S.Z., Wang, Y.: Generalized quotient image. In: Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, vol. 2, pp. II–II. IEEE (2004)

    Google Scholar 

  65. Wang, X., Han, T.X., Yan, S.: An hog-lbp human detector with partial occlusion handling. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 32–39 (2009). DOI 10.1109/ICCV.2009.5459207

    Google Scholar 

  66. Wen, Y., Zhang, K., Li, Z., Qiao, Y.: A discriminative feature learning approach for deep face recognition. In: B. Leibe, J. Matas, N. Sebe, M. Welling (eds.) Computer Vision – ECCV 2016, pp. 499–515. Springer International Publishing, Cham (2016)

    Chapter  Google Scholar 

  67. Wiskott, L., Krüger, N., Kuiger, N., Von Der Malsburg, C.: Face recognition by elastic bunch graph matching. IEEE Transactions on pattern analysis and machine intelligence 19(7), 775–779 (1997)

    Article  Google Scholar 

  68. Xie, X., Zheng, W.S., Lai, J., Yuen, P.C., Suen, C.Y.: Normalization of face illumination based on large-and small-scale features. IEEE Transactions on Image Processing 20(7), 1807–1821 (2011)

    Article  MathSciNet  Google Scholar 

  69. Yang, J., Frangi, A.F., Yang, J.y., Zhang, D., Jin, Z.: Kpca plus lda: a complete kernel fisher discriminant framework for feature extraction and recognition. IEEE Transactions on pattern analysis and machine intelligence 27(2), 230–244 (2005)

    Google Scholar 

  70. Zhang, Y., Shao, M., Wong, E.K., Fu, Y.: Random faces guided sparse many-to-one encoder for pose-invariant face recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2416–2423 (2013)

    Google Scholar 

  71. Zhao, W., Chellappa, R., Phillips, P.J., Rosenfeld, A.: Face recognition: A literature survey. ACM Computing Surveys 35(4), 399–458 (2003)

    Article  Google Scholar 

  72. Zhou, H., Sadka, A.H.: Combining perceptual features with diffusion distance for face recognition. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 41(5), 577–588 (2011)

    Google Scholar 

  73. Zhu, Z., Luo, P., Wang, X., Tang, X.: Deep learning identity-preserving face space. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 113–120 (2013)

    Google Scholar 

  74. Zhu, Z., Luo, P., Wang, X., Tang, X.: Multi-view perceptron: a deep model for learning face identity and view representations. Advances in Neural Information Processing Systems (NIPS) pp. 217–225 (2014)

    Google Scholar 

Download references

Acknowledgements

This chapter is partly supported by the National Natural Science Foundation of China (No.61502388), Ph.D. Programs Foundation of Ministry of Education of China (No. 20136102120041), the Fundamental Research Funds for the Central Universities (No. 3102015BJ (II)ZS016), and the Shaanxi Province International Science and Technology Cooperation and Exchange Program (2017KW002).

Author information

Authors and Affiliations

  1. School of Electronics and Information, Northwestern Polytechnical University, Xi’an, China

    Xiaoyue Jiang, Yaping Hou, Dong Zhang & Xiaoyi Feng

Authors
  1. Xiaoyue Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yaping Hou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaoyi Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoyue Jiang .

Editor information

Editors and Affiliations

  1. School of Electronics and Information, Northwestern Polytechnical University, Xi’an, Shaanxi, China

    Xiaoyue Jiang  & Xiaoyi Feng  & 

  2. Center for Machine Vision and Signal Analysis, University of Oulu, Oulu, Oulu, Finland

    Abdenour Hadid

  3. School of Electrical and Information Engineering, Tianjin University, Tianjin, Tianjin, China

    Yanwei Pang

  4. École de technologie supérieure, University of Québec, Montréal, QC, Canada

    Eric Granger

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Singapore Pte Ltd.

About this chapter

Cite this chapter

Jiang, X., Hou, Y., Zhang, D., Feng, X. (2019). Deep Learning in Face Recognition Across Variations in Pose and Illumination. In: Jiang, X., Hadid, A., Pang, Y., Granger, E., Feng, X. (eds) Deep Learning in Object Detection and Recognition. Springer, Singapore. https://doi.org/10.1007/978-981-10-5152-4_3

Download citation

  • DOI: https://doi.org/10.1007/978-981-10-5152-4_3

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-10-5151-7

  • Online ISBN: 978-981-10-5152-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation