Skip to main content
SpringerLink
Log in

Multiscale Fusion of Visible and Thermal IR Images for Illumination-Invariant Face Recognition

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

This paper describes a new software-based registration and fusion of visible and thermal infrared (IR) image data for face recognition in challenging operating environments that involve illumination variations. The combined use of visible and thermal IR imaging sensors offers a viable means for improving the performance of face recognition techniques based on a single imaging modality. Despite successes in indoor access control applications, imaging in the visible spectrum demonstrates difficulties in recognizing the faces in varying illumination conditions. Thermal IR sensors measure energy radiations from the object, which is less sensitive to illumination changes, and are even operable in darkness. However, thermal images do not provide high-resolution data. Data fusion of visible and thermal images can produce face images robust to illumination variations. However, thermal face images with eyeglasses may fail to provide useful information around the eyes since glass blocks a large portion of thermal energy. In this paper, eyeglass regions are detected using an ellipse fitting method, and replaced with eye template patterns to preserve the details useful for face recognition in the fused image. Software registration of images replaces a special-purpose imaging sensor assembly and produces co-registered image pairs at a reasonable cost for large-scale deployment. Face recognition techniques using visible, thermal IR, and data-fused visible-thermal images are compared using a commercial face recognition software (FaceIt®) and two visible-thermal face image databases (the NIST/Equinox and the UTK-IRIS databases). The proposed multiscale data-fusion technique improved the recognition accuracy under a wide range of illumination changes. Experimental results showed that the eyeglass replacement increased the number of correct first match subjects by 85% (NIST/Equinox) and 67% (UTK-IRIS).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Adini, Y., Moses, Y., and Ullman, S. 1997. Face recognition: The problem of compensating for changes in illumination direction. IEEE Trans. Pattern Analysis and Machine Intelligence, 19(7):721–732.

    Article  Google Scholar 

  • Ben-Yacoub, S., Abdeljaoued, Y., and Mayoraz, E. 1999. Fusion of face and speech data for person identity verification. IEEE Trans. on Neural Networks, 10(5):1065–1074.

    Article  Google Scholar 

  • Besl, P.J. and McKay, N.D. 1992. A Method for Registration of 3-D shapes. IEEE Trans. Pattern Analysis and Machine Intelligence, 14(2):239–256.

    Article  Google Scholar 

  • Blackburn, D.M., Bone, J.M., and Phillips, P.J. 2001. Face Recognition Vendor Test 2000. Evaluation Report, National Institute of Standards and Technology, pp. 1–70.

  • Bone, M. and Blackburn, D. 2002. Face Recognition at a Chokepoint: Scenario Evaluation Results. Evaluation Report, Department of Defense.

  • Bookstein, F.L. 1979. Fitting conic sections to scattered data. Computer Graphics and Image Processing, 9(1):56–71.

    Google Scholar 

  • Boughorbel, F., Koschan, A., Abidi, B., and Abidi, M. 2004a. Gaussian fields: A New Criterion for 3D Rigid Registration. Pattern Recognition, 37(7):1567–1571.

    Article  Google Scholar 

  • Boughorbel, F., Koschan, A., Abidi, B., and Abidi, M. 2004b. Gaussian Energy Functions for Registration without Correspondences. In Proc. of 17th Int’l Conf. on Pattern Recognition, pp. 24–27 Cambridge, UK.

  • Brunelli R. and Poggio, T. 1993. Face recognition: Features versus templates. IEEE Trans. Pattern Analysis and Machine Intelligence, 15(10):1042–1052.

    Article  Google Scholar 

  • Brunelli, R. and Falavigna D. 1995. Personal Identification Using Multiple Cues. IEEE Trans. Pattern Analysis and Machine Intelligence, 17(10):955–966.

    Article  Google Scholar 

  • Burton, A.M., Bruce, V., and Hancock, P.J.B. 1999. From pixels to people: A model of familiar face recognition. Cognitive Science, 23(1):1–31.

    Article  Google Scholar 

  • Chang, K., Bowyer, K.W., Sarkar, S., and Victor, B. 2003. Comparison and Combination of Ear and Face Image in Appearance-Based Biometrics. IEEE Trans. Pattern Analysis and Machine Intelligence, 25(9):1160–1165.

    Article  Google Scholar 

  • Chellappa, R., Wilson, C.L., and Sirohey, S. 1995. Human and Machine Recognition of Faces: A survey. In Proceedings of the IEEE, Vol. 83, no. 5, pp. 705–740.

    Article  Google Scholar 

  • Cox, I.J., Ghosn, J., and Yianilos, P.N. 1996. Feature-Based Face Recognition Using Mixture-Distance. In Proc. Int’l Conf. Computer Vision and Pattern Recognition, pp. 209–216.

  • Charpiat, G., Faugeras, O., and Keriven, R. 2003. Shape Metrics, Warping and Statistics. In Proc. Int’l Conf. on Image Processing, Barcelona, Vol. 2, pp. 627–630.

  • Chen, X., Flynn, P. and Bowyer, K. 2003. Visible-light and infrared face recognition. In Proc. of Workshop on Multimodal User Authentication, pp. 48–55.

  • Craw, I., Costen, N., Kato, T., and Akamatsu, S. 1999. How should we represent faces for automatic recognition?. IEEE Trans. Pattern Analysis and Machine Intelligence, 21(8):725–736.

    Article  Google Scholar 

  • Dalley, G. and Flynn, P. 2002. Pair-Wise Range Image Registration: A Study in Outlier Classification. Computer Vision and Image Understanding, 87(1–3):104–115.

    Article  MATH  Google Scholar 

  • Dasarathy, B.V. 1994. Decision Fusion, Washington. DC: IEEE Computer Society Press.

    Google Scholar 

  • Daubechies, I. 1992. Ten lectures on wavelets. CBMS-NSF Lecture Notes, No. 61, Society for Industrial and Applied Mathematics.

  • Elgammal, A., Duraiswami, R., and Davis, L. 2003. Efficient Kernel Density Estimation using the Fast Gauss Transform with Applications to Color Modeling and Tracking. IEEE Trans. Pattern Analysis and Machine Intelligence, 25(11):1499–1504.

    Article  Google Scholar 

  • Fang, Y., Tan, T., and Wang, Y. 2002. Fusion of global and local features for face verification. In Proc. of Int’l Conf. on Pattern Recognition, pp. 382–385.

  • Fitzgibbon, A.W. 2003. Robust registration of 2D and 3D Point Sets. Image and Vision Computing, 21(13):1145–1153.

    Article  Google Scholar 

  • Fitzgibbon, A., Pilu, M., and Fisher, R.B. 1999. Direct least square fitting of ellipses. IEEE Trans. on Pattern Analysis and Machine Intelligence, 21(5):476–480.

    Article  Google Scholar 

  • Freeman, H. and Davis, L.S. 1977. A Corner-Finding Algorithm for Chain-Coded Curves. IEEE Trans. on Computer, 26(3):297–303.

    Google Scholar 

  • Greengard, L. 1988. The Rapid Evaluation of Potential Fields in Particle Systems. Cambridge, MA: MIT Press.

    MATH  Google Scholar 

  • Greengard, L. and Strain, J. 1991. The Fast Gauss Transform. SIAM Journal on Scientific Computing, 12:79–94.

    Article  MATH  MathSciNet  Google Scholar 

  • Gutta, S., Huang, J.R.J., Jonathon, P., and Wechsler, H. 2000. Mixture of experts for classification of gender, ethnic origin, and pose of human faces. IEEE Trans. Neural Networks, 11(4):948–960.

    Article  Google Scholar 

  • Hall, D.L. and Llinas, J. 2001. Handbook of Multisensor Data Fusion. CRC Press.

  • Heo, J., Abidi, B., Kong, S.G., and Abidi, M. 2003. Performance Comparison of Visual and Thermal Signatures for Face Recognition. Biometric Consortium Conference, Arlington, VA.

  • Ho, T.K., Hull, J.J., and Srihari, S.N. 1994. Decision combination in multiple classifier systems. IEEE Trans. on Pattern Analysis and Machine Intelligence, 16(1):66–75.

    Article  Google Scholar 

  • Hong, L. and Jain, A. 1998. Integrating faces and fingerprints for personal identification. IEEE Trans. Pattern Analysis and Machine Intelligence, 20(12):1295–1307.

    Article  Google Scholar 

  • http://www.cse.ohio-state.edu/otcbvs-bench/

  • http://www.equinoxsensors.com/products/HID.html

  • Huttenlocher, D.P., Klanderman, G.A., and Rucklidge, W.J. 1993. Comparing images using the Hausdorff distance. IEEE Trans. Pattern Analysis and Machine Intelligence, 15(9):850–863.

    Article  Google Scholar 

  • Irani, M. and Anandan, P. 1998. Robust multi-sensor image alignment. In Proc. of the 6th Int’l Conf. on Computer Vision, pp. 959–965.

  • Jain, A.K., Ross, A., and Prabhakar, S. 2004. An introduction to biometric recognition. IEEE Trans. Circuits and Systems for Video Technology, 14(1):4–20.

    Article  Google Scholar 

  • Kanade, T. 1973. Picture Processing by Computer Complex and Recognition of Human Faces. Technical Report, Kyoto University.

  • Kong, S.G., Heo, J., Abidi, B.R., Paik, J., and Abidi, M.A. 2005. Recent Advances in Visual and Infrared Face Recognition—A review. Computer Vision and Image Understanding, 97(1):103–135.

    Article  Google Scholar 

  • Manjunath, B.S., Chellappa, R., and von der Malsburg, C. 1992. A feature Based Approach to Face Recognition. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 373–378.

  • Murio, D.A. 1993. The Mollification Method and the Numerical Solution of Ill-Posed Problems. New York, NY: John Wiley & Sons.

    Google Scholar 

  • Pavlidis, I. and Symosek, P. 2000. The imaging issue in an automatic face/disguise detection system. In Proc. IEEE Workshop on Computer Vision Beyond the Visible Spectrum: Methods and Applications, pp. 15–24.

  • Penev, P.S. 1998. Local Feature Analysis: A Statistical Theory for Information Representation and Transmission. Ph.D. Thesis, The Rockefeller University.

  • Penev, P.S. 1999. Dimensionality reduction by sparsification in a local-features representation of human faces. Technical Report, The Rockefeller University.

  • Phillips, P.J., Grother, P., Micheals, R.J., Blackburn, D.M., Tabassi, E., and Bone, M. 2003. Face Recognition Vendor Test 2002. Evaluation Report, National Institute of Standards and Technology, pp. 1–56.

  • Phillips, P.J., Moon, H., Rizvi, S.A., and Rauss, P.J. 2000. The FERET Evaluation Methodology for Face-Recognition Algorithms. IEEE Trans. Pattern Analysis and Machine Intelligence, 22(10):1090–1104.

    Article  Google Scholar 

  • Press, W.H., Teukolsky, S.A., Vetterling, W.T., and Flannery, B.P. 1992. Numerical Recipes in C: The Art of Scientific Computing. Second edition. Cambridge University Press.

  • Prokoski, F. 2000. History, current status, and future of infrared identification. In Proc. IEEE Workshop on Computer Vision Beyond the Visible Spectrum: Methods and Applications, pp. 5–14.

  • Ross, A. and Jain, A. 2003. Information fusion in biometrics. Pattern Recognition Letters, 24(13):2115–2125.

    Article  Google Scholar 

  • Selinger, A. and Socolinsky, D.A. 2001. Appearance-Based Facial Recognition Using Visible and Thermal Imagery: A Comparative Study. Technical Report 02-01, Equinox Corporation.

  • Sharp, G.C., Lee, S.W., and Wehe, D.K. 2002. ICP Registration using Invariant Features. IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 24(1):90–102.

    Article  Google Scholar 

  • Singh, S., Gyaourova, A., Bebis, G., and Pavlidis, I. 2004. Infrared and Visible Image Fusion for Face Recognition. In Proc. SPIE Defense and Security Symposium (Biometric Technology for Human Identification), pp. 585–596.

  • Socolinsky, D.A., Selinger, A., and Neuheisel, J.D. 2003. Face recognition with visible and thermal infrared imagery. Computer Vision and Image Understanding, 91(1–2):72–114.

    Article  Google Scholar 

  • Snelick, R., Uludag, U., Mink, A., Indovina, M., and Jain, A. 2005. Large-scale evaluation of multimodal biometric authentication using state-of-the-art systems. IEEE Trans. Pattern Analysis and Machine Intelligence, 27(3):450–455.

    Article  Google Scholar 

  • Störring, M., Andersen, H.J., and Granum, E. 2001. Physics-based modelling of human skin colour under mixed illuminants. Journal of Robotics and Autonomous Systems, 35(3/4):131–142.

    Article  MATH  Google Scholar 

  • Turk, M. and Pentland, A. 1991. Eigenfaces for recognition. Journal of Cognitive Neuroscience, 3(1):72–86.

    Article  Google Scholar 

  • Wilder, J., Phillips, P.J., Jiang, C. and Wiener, S. 1996. Comparison of visible and infrared imagery for face recognition. In Proc. Int. Conf. Automatic Face and Gesture Recognition, pp. 182–187.

  • Wolff, L.B., Socolinsky, D.A., and Eveland, C.K. 2001. Quantitative measurement of illumination invariance for face recognition using thermal infrared imagery. In Proc. IEEE Workshop on Computer Vision Beyond the Visible Spectrum.

  • Yang, C., Duraiswami, R., Gumerov, N.A., and Davis, L. 2003. Improved fast Gauss transform and efficient kernel density estimation. In Proc. 9th Int’l Conf. on Computer Vision, Nice, France, pp. 464–471.

  • Yoshitomi, Y., Miyaura, T., Tomita, S., and Kimura, S. 1997. Face identification using thermal image processing. In Proc. IEEE Int’l Workshop on Robot and Human Communication, pp. 374–379.

  • Zitova, B. and Flusser, J. 2003. Image registration methods: a survey. Image and Vision Computing, 21(11):977–1000.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Imaging, Robotics, and Intelligent Systems Laboratory, Department of Electrical and Computer Engineering, The University of Tennessee, Knoxville, TN, 37996-2100

    Seong G. Kong, Faysal Boughorbel, Yue Zheng, Besma R. Abidi, Andreas Koschan, Mingzhong Yi & Mongi A. Abidi

  2. Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, PA, 15213-3890

    Jingu Heo

Authors
  1. Seong G. Kong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jingu Heo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Faysal Boughorbel
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yue Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Besma R. Abidi
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Andreas Koschan
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Mingzhong Yi
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Mongi A. Abidi
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kong, S.G., Heo, J., Boughorbel, F. et al. Multiscale Fusion of Visible and Thermal IR Images for Illumination-Invariant Face Recognition. Int J Comput Vision 71, 215–233 (2007). https://doi.org/10.1007/s11263-006-6655-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-006-6655-0

Keywords

Search

Navigation