TR-GAN: thermal to RGB face synthesis with generative adversarial network for cross-modal face recognition

Landry Kezebou; Victor Oludare; Karen Panetta; Sos Agaian

doi:10.1117/12.2558166

21 April 2020 TR-GAN: thermal to RGB face synthesis with generative adversarial network for cross-modal face recognition

Landry Kezebou, Victor Oludare, Karen Panetta, Sos Agaian

Author Affiliations +

Proceedings Volume 11399, Mobile Multimedia/Image Processing, Security, and Applications 2020; 113990P (2020) https://doi.org/10.1117/12.2558166
Event: SPIE Defense + Commercial Sensing, 2020, Online Only

Abstract

Unlike RBG cameras, thermal cameras perform well under very low lighting conditions and can capture information beyond the human visible spectrum. This provides many advantages for security and surveillance applications. However, performing face recognition tasks in the thermal domain is very challenging given the limited visual information embedded in thermal images and the inherent similarities among facial heat maps. Attempting to perform recognition across modalities, such as recognizing a face captured in the thermal domain given the corresponding visible light domain ground truth database or vice versa is also a challenge. In this paper, a Thermal to RGB Generative Adversarial Network (TRGAN) to automatically synthesize face images captured in the thermal domain, to their RBG counterparts, with a goal of reducing current inter-domain gaps and significantly improving cross-modal facial recognition capabilities is proposed. Experimental results on the TUFTS Face Dataset using the VGG-Face recognition model without retraining, demonstrates that performing image translation with the proposed TR-GAN model almost doubles the cross-modal recognition accuracy and also performs better than other state-of-the-art GAN models on the same task. The generator in our network uses a UNET like architecture with cascaded-in-cascaded blocks to reuse features from earlier convolutions, which helps generate high quality images. To further guide the generator to synthesize images with fine details, we optimize a training loss as the weighted sum of the perceptual, adversarial, and cycle-consistent loss. Simulation results demonstrate that the proposed model generates more realistic and more visually appealing images, with finer details and better reconstruction of intricate details such sunglasses and facial emotions, than similar GAN models.

Citation Download Citation

Landry Kezebou, Victor Oludare, Karen Panetta, and Sos Agaian "TR-GAN: thermal to RGB face synthesis with generative adversarial network for cross-modal face recognition", Proc. SPIE 11399, Mobile Multimedia/Image Processing, Security, and Applications 2020, 113990P (21 April 2020); https://doi.org/10.1117/12.2558166

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available