Skip to main content
SpringerLink
Log in

Transfer deep feature learning for face sketch recognition

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Sketch-to-photo recognition is an important challenge in face recognition because it requires matching face images in different domains. Even the deep learning, which has recently been deployed in face recognition, is not efficient for face sketch recognition due to the limited sketch datasets. In this paper, we propose a novel face sketch recognition approach based on transfer learning. We design a three-channel convolutional neural network architecture in which the triplet loss is adopted in order to learn discriminative features and reduce intra-class variations. Moreover, we propose a hard triplet sample selection strategy to augment the number of training samples and avoid slow convergence. With the proposed method, facial features from digital photos and from sketches taken from the same person are closer; the opposite occurs if the digital photo and sketch are from different identities. Experimental results on multiple public datasets indicate that the proposed face sketch recognition method outperforms the existing approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Peng C, Wang N, Gao X, Li J (2018) Face recognition from multiple stylistic sketches: scenarios, datasets, and evaluation. Pattern Recognit 84:262–272

    Article  Google Scholar 

  2. Wang N, Gao X, Sun J, Li J (2017) Anchored neighborhood index for face sketch synthesis. IEEE Trans Circuits Syst Video Technol 28:2154–2163

    Article  Google Scholar 

  3. Samma H, Suandi SA, Mohamad-Saleh J (2018) Face sketch recognition using a hybrid optimization model. Neural Comput Appl. https://doi.org/10.1007/s00521-018-3475-4

    Article  Google Scholar 

  4. Liu D, Li J, Wang N, Peng C, Gao X (2018) Composite components-based face sketch recognition. Neurocomputing 302:46–54

    Article  Google Scholar 

  5. Zhang W, Wang X, Tang X (2011) Coupled information-theoretic encoding for face photo-sketch recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 513–520

  6. Klare B, Li Z, Jain A (2011) Matching forensic sketches to mug shot photos. IEEE Trans Pattern Anal Mach Intell 33:639–646

    Article  Google Scholar 

  7. Galoogahi H, Sim T (2012) Inter-modality face sketch recognition. In: Proceedings of the IEEE international conference on multimedia and expo, pp 224–229

  8. Alex A, Asari V, Mathew A (2013) Local difference of gaussian binary pattern: robust features for face sketch recognition. In: IEEE international conference on systems, man, and cybernetics, pp 1211–1216

  9. Klare BF, Jain AK (2013) Heterogeneous face recognition using kernel prototype similarities. IEEE Trans Pattern Anal Mach Intell 35:1410–1422

    Article  Google Scholar 

  10. Tang X, Wang X (2003) Face sketch synthesis and recognition. In: Proceedings of the IEEE conference on computer vision, pp 687–694

  11. Liu Q, Tang X, Jin H, Lu H, Ma S (2005) A nonlinear approach for face sketch synthesis and recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1005–1010

  12. Li J, Yu X, Peng C, Wang N (2017) Adaptive representation-based face sketch-photo synthesis. Neurocomputing 269:152–159

    Article  Google Scholar 

  13. Wang N, Gao X, Li J (2018) Random sampling for fast face sketch synthesis. Pattern Recognit 76:215–227

    Article  Google Scholar 

  14. Zhang L, Lin L, Wu X, Ding S, Zhang L (2015) End-to-end photo-sketch generation via fully convolutional representation learning. In: Proceedings of the 5th ACM on international conference on multimedia retrieval, pp 627–634

  15. Wang N, Zha W, Li J, Gao X (2018) Back projection: an effective postprocessing method for gan-based face sketch synthesis. Pattern Recognit Lett 107:59–65

    Article  Google Scholar 

  16. Jiao L, Zhang S, Li L, Liu F, Ma W (2018) A modified convolutional neural network for face sketch synthesis. Pattern Recognit 76:125–136

    Article  Google Scholar 

  17. Jiang J, Yu Y, Wang Z, Liu X, Ma J (2019) Graph-regularized locality-constrained joint dictionary and residual learning for face sketch synthesis. IEEE Trans Image Process 28:628–641

    Article  MathSciNet  Google Scholar 

  18. Lin D, Tang X (2006) Inter-modality face recognition. In: Proceedings of the 9th European conference on computer vision, pp 13–26

    Chapter  Google Scholar 

  19. Lei Z, Li S (2009) Coupled spectral regression for matching heterogeneous faces. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1123–1128

  20. Lei Z, Zhou C, Yi D, Jain A, Li S (2012) An improved coupled spectral regression for heterogeneous face recognition. In: Proceedings of the international conference on biomedical, pp 7–12

  21. Sharma A, Jacobs D (2011) Bypass synthesis: PLS for face recognition with pose, low-resolution and sketch. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 593–600

  22. Mignon A, Jurie F (2012) CMML: a new metric learning approach for cross modal matching. In: Proceedings of the Asian conference on computer vision, pp 1–14

  23. Kan M, Shan S, Zhang H, Lao S, Chen X (2012) Multi-view discriminant analysis. In: Proceedings of the 12th European conference on computer vision, pp 808–821

    Chapter  Google Scholar 

  24. Huo J, Gao Y, Shi Y, Wang Y, Yin H (2018) Heterogeneous face recognition by margin-based cross-modality metric learning. IEEE Trans Cybern 48:1814–1826

    Article  Google Scholar 

  25. Schroff F, Kalenichenko D, Philbin J (2015) Facenet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 815–823

  26. Parkhi OM, Andrea V, Andrew Z (2015) Deep face recognition. Br Mach Vis Conf 1:1–6

    Google Scholar 

  27. Galea C, Farrugia RA (2017) Forensic face photo-sketch recognition using a deep learning-based architecture. IEEE Signal Process Lett 24:1586–1590

    Article  Google Scholar 

  28. Mittal P, Vatsa M, Singh R (2015) Composite sketch recognition via deep network—a transfer learning approach. In: Biometrics (ICB), 2015 IEEE international conference on, pp 251–256

  29. Zhang H, Ji Y, Huang W, Liu L (2018) Sitcom-star-based clothing retrieval for video advertising: a deep learning framework. Neural Comput Appl. https://doi.org/10.1007/s00521-018-3579-x

    Article  Google Scholar 

  30. Zhang H, Cao X, Ho JKL, Chow TWS (2017) Object-level video advertising: an optimization framework. IEEE Trans Ind Inf 13:520–531

    Article  Google Scholar 

  31. Wang X, Tang X (2009) Face photo-sketch synthesis and recognition. IEEE Trans Pattern Anal Mach Intell 31:196–1955

    Google Scholar 

  32. Bhatt HS, Bharadwaj S, Singh R, Vatsa M (2011) Memetic approach for matching sketches with digital face images. Technical Report TR-2011-006, Indraprastha Institute of Information Technology, New Delhi, India

  33. King DE (2009) Dlib-ml: a machine learning toolkit. J Mach Learn Res 10:1755–1758

    Google Scholar 

  34. Bolle RM, Connell JH, Pankanti S et al (2005) The relation between the ROC curve and the CMC. In: Fourth IEEE workshop on automatic identification advanced technologies, pp 15–20

  35. Lenc L, Pavel K (2015) Automatic face recognition system based on the SIFT features. Comput Electr Eng 46:256–272

    Article  Google Scholar 

  36. Ding C, Choi J, Tao D, Davis LS (2016) Multi-directional multi-level dual-cross patterns for robust face recognition. IEEE Trans Pattern Anal Mach Intell 38:518–531

    Article  Google Scholar 

  37. Wu X, He R, Sun Z, Tan T (2018) A light CNN for deep face representation with noisy labels. IEEE Trans Inf Forens Secur 13:2884–2896

    Article  Google Scholar 

  38. Yang S, Luo P, Loy CC, Tang X (2015) From facial parts responses to face detection: a deep learning approach. In: IEEE international conference on computer vision, pp 3676–3684

Download references

Acknowledgements

This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (GR 2016R1D1A3B03931911). This study was also financially supported by the grants of China Scholarship Council (CSC No. 201708260057). This work was also supported in part by the National Natural Science Foundation of China under Grants 61802253, in part by the Collaborative Innovation Center for Economic Crime Investigation and Prevention Technology of Jiangxi Province under Grant JXJZXTCX-027, in part by the Shanghai Chenguang Talented Program under Grant 17CG59.

Author information

Authors and Affiliations

  1. Division of Computer Science and Engineering, Chonbuk National University, Jeonju, Korea

    Weiguo Wan & Hyo Jong Lee

  2. School of Electrics and Electronic Engineering, Shanghai University of Engineering Science, Shanghai, China

    Yongbin Gao

  3. Center for Advanced Image and Information Technology, Chonbuk National University, Jeonju, Korea

    Hyo Jong Lee

Authors
  1. Weiguo Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yongbin Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hyo Jong Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hyo Jong Lee.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest. We declare that we do not have any commercial or associative interest that represents a conflict of interest in connection with the work entitled “Transfer Deep Feature Learning for Face Sketch Recognition.”

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wan, W., Gao, Y. & Lee, H.J. Transfer deep feature learning for face sketch recognition. Neural Comput & Applic 31, 9175–9184 (2019). https://doi.org/10.1007/s00521-019-04242-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-019-04242-5

Keywords

Search

Navigation