Skip to main content

Advertisement

SpringerLink
Log in

Learning to sketch human facial portraits using personal styles by case-based reasoning

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper employs case-based reasoning (CBR) to capture the personal styles of individual artists and generate the human facial portraits from photos accordingly. For each human artist to be mimicked, a series of cases are firstly built-up from her/his exemplars of source facial photo and hand-drawn sketch, and then its stylization for facial photo is transformed as a style-transferring process of iterative refinement by looking-for and applying best-fit cases in a sense of style optimization. Two models, fitness evaluation model and parameter estimation model, are learned for case retrieval and adaptation respectively from these cases. The fitness evaluation model is to decide which case is best-fitted to the sketching of current interest, and the parameter estimation model is to automate case adaptation. The resultant sketch is synthesized progressively with an iterative loop of retrieval and adaptation of candidate cases until the desired aesthetic style is achieved. To explore the effectiveness and advantages of the novel approach, we experimentally compare the sketch portraits generated by the proposed method with that of a state-of-the-art example-based facial sketch generation algorithm as well as a couple commercial software packages. The comparisons reveal that our CBR based synthesis method for facial portraits is superior both in capturing and reproducing artists’ personal illustration styles to the peer methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

References

  1. Aamodt A, Plaza E (1994) Case-based reasoning: foundational issues, methodological variations, and system approaches. AI Commun 7(1):39–59

    Google Scholar 

  2. AKVIS: Akvis sketch v.13.0 (2012) http://akvis.com/en/sketch/index.php

  3. Bay H, Tuytelaars T, Van Gool L (2006) Surf: speeded up robust features. ECCV. Springer, pp 404–417

  4. Belongie S, Malik J, Puzicha J (2002) Shape matching and object recognition using shape contexts. PAMI 24(4):509–522

    Article  Google Scholar 

  5. Berger I, Shamir A, Mahler M, Carter E, Hodgins J (2013) Style and abstraction in portrait sketching. TOG 32(4):55

    Article  Google Scholar 

  6. Bradski G (2000) The opencv library. Dr Dobb’s Journal of Software Tools

  7. Breiman L (1996) Bagging predictors. Mach Learn 24:123–140

    MathSciNet  MATH  Google Scholar 

  8. Burgess CM (2005) Cosmetic dermatology. Springer

  9. Chen CT (1998) Linear system theory and design, 3rd edn. Oxford University Press Inc, Oxford

    Google Scholar 

  10. Chen H, Liu Z, Rose C, Xu Y, Shum HY, Salesin D (2004) Example-based composite sketching of human portraits. In: NPAR. ACM, pp 95–153

  11. Gonzalez RC, Woods RE (2002) Digital image processing, 2nd edn. Prentice Hall, SL

    Google Scholar 

  12. Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The weka data mining software: an update. SIGKDD Explor Newsl 11(1):10–18

    Article  Google Scholar 

  13. Hertzmann A, Jacobs CE, Oliver N, Curless B, Salesin DH (2001) Image analogies. In: SIGGRAPH. ACM, pp 327–340

  14. Kalogerakis E, Nowrouzezahrai D, Breslav S, Hertzmann A (2012) Learning hatching for pen-and-ink illustration of surfaces. ACM Trans Graph 31(1):1:1–1:17

    Article  Google Scholar 

  15. Kim SY, Maciejewski R, Isenberg T, Andrews WM, Chen W, Sousa MC, Ebert DS (2009) Stippling by example. In: NPAR. ACM, pp 41–50

  16. Kohavi R, John GH (1997) Wrappers for feature subset selection. Artif Intell 97(1):273–324

    Article  MATH  Google Scholar 

  17. Kyprianidis J, Collomosse J, Wang T, Isenberg T (2013) State of the art: a taxonomy of artistic stylization techniques for images and video. TVCG 19(5):866–885

    Google Scholar 

  18. Leake DB, Kinley A, Wilson D (1995) Learning to improve case adaptation by introspective reasoning and cbr. In: Case-based reasoning research and development. Springer, pp 229–240

  19. Lee H, Seo S, Ryoo S, Yoon K (2010) Directional texture transfer. In: NPAR. ACM, pp 43–48

  20. Lee SY, Chwa KY, Hahn J, Shin SY (1996) Image morphing using deformation techniques. JVCA 7(1):3–23

    Google Scholar 

  21. Liang L, Chen H, Xu YQ, Shum HY (2002) Example-based caricature generation with exaggeration. In: PG. IEEE Computer Society, p 386

  22. Liu Q, Tang X, Jin H, Lu H, Ma S (2005) A nonlinear approach for face sketch synthesis and recognition. In: CVPR. IEEE Computer Society, pp 1005–1010

  23. Lu C, Xu L, Jia J (2012) Combining sketch and tone for pencil drawing production. In: NPAR. Eurographics Association, pp 65–73

  24. Lu J, Sander PV, Finkelstein A (2010) Interactive painterly stylization of images, videos and 3d animations. In: Proceedings of the 2010 ACM SIGGRAPH symposium on interactive 3d graphics and games. ACM, pp 127–134

  25. Maes F, Collignon A, Vandermeulen D, Marchal G, Suetens P (1997) Multimodality image registration by maximization of mutual information. T-MI 16 (2):187–198

    Google Scholar 

  26. Microsoft: Microsoft office homestyle + trial edition (2003). http://www.microsoft.com/zh-tw/download/details.aspx?id=4851

  27. Milborrow S, Nicolls F (2008) Locating facial features with an extended active shape model ECCV

  28. Min F, Suo JL, Zhu SC, Sang N (2007) An automatic portrait system based on and-or graph representation. In: EMMCVPR. Springer, pp 184–197

  29. Mish FC (1994) Merriam webster’s collegiate dictionary. Merriam Webster Inc

  30. Modat M, Ridgway GR, Taylor ZA, Lehmann M, Barnes J, Hawkes DJ, Fox NC, Ourselin S (2010) Fast free-form deformation using graphics processing units. Comput Methods Prog Biomed 98:278– 284

    Article  Google Scholar 

  31. Papari G, Petkov N (2009) Continuous glass patterns for painterly rendering. IEEE Trans Image Process 18(3):652–664

    Article  MathSciNet  MATH  Google Scholar 

  32. Papari G, Petkov N, Campisi P (2007) Artistic edge and corner enhancing smoothing. IEEE Trans Image Process 16(10):2449–2462

    Article  MathSciNet  Google Scholar 

  33. Parke FI (1972) Computer generated animation of faces. In: Proceedings of the ACM annual conference, vol 1. ACM, pp 451–457

  34. Peng H, Long F, Ding C (2005) Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy. PAMI 27(8):1226–1238

    Article  Google Scholar 

  35. Powell MJD (1998) Direct search algorithms for optimization calculations. Acta Numerica:287–336

  36. Redman L (2012) How to draw caricatures. McGraw Hill Professional

  37. Reinhard E, Ashikhmin M, Gooch B, Shirley P (2001) Color transfer between images. IEEE Comput Graph Appl 21(5):34–41

    Article  Google Scholar 

  38. Salesin DH (2002) Non-Photorealistic animation & rendering: 7 grand challenges. In: Keynote talk at second international symposium on non-photorealistic animation and rendering (NPAR 2002, Annecy, France, June 3–5, 2002). http://www.research.microsoft.com/salesin/NPAR.ppt

  39. Schaffer C (1993) Selecting a classification method by cross-validation. Mach Learn 13(1):135–143

    Google Scholar 

  40. Schaefer S, McPhail T, Warren J (2006) Image deformation using moving least squares. TOG 25:533–540

    Article  Google Scholar 

  41. Semillon (2013) Semillon’s homepage on sina weibo. http://weibo.com/semillon

  42. Song Y, Bao L, Yang Q, Yang MH (2014) Real-time exemplar-based face sketch synthesis. Springer, pp 800–813

  43. Sonka M, Hlavac V, Boyle R (1993) Image processing, analysis and machine vision. Chapman and Hall, London

    Book  Google Scholar 

  44. Thévenaz P, Unser M (2000) Optimization of mutual information for multiresolution image registration. IEEE Trans Image Process 9(12):2083–2099

    Article  MATH  Google Scholar 

  45. Tu CT, Lien JJJ (2010) Automatic location of facial feature points and synthesis of facial sketches using direct combined model. Trans Sys Man Cyber Part B 40:1158–1169

    Article  Google Scholar 

  46. Wang N (2013) Results on cuhk face sketch database. https://nannanwang.github.io/Result_CUFS.htm

  47. Wang N, Tao D, Gao X, Li X, Li J (2013) Transductive face sketch-photo synthesis. IEEE Transactions on Neural Networks and Learning Systems 24(9):1364–1376

    Article  Google Scholar 

  48. Wang X, Tang X (2009) Face photo-sketch synthesis and recognition. PAMI 31:1955–1967

    Article  Google Scholar 

  49. Wei G, Rui M, Lei W, Yi Z, Zhenyun P, Yaohui Z (2009) Template-based portrait caricature generation with facial components analysis. In: ICIS, pp 219–223

  50. Wu C, Liu C, Shum HY, Xy YQ, Zhang Z (2004) Automatic eyeglasses removal from face images. IEEE Trans Pattern Anal Mach Intell 26(3):322–336

    Article  Google Scholar 

  51. Zhang Y, Dong W, Deussen O, Huang F, Li K, Hu BG (2014) Data-driven face cartoon stylization. In: SIGGRAPH Asia 2014 technical briefs. ACM, pp 14:1–14:4

  52. Zhao M, Zhu SC (2010) Sisley the abstract painter. In: NPAR. ACM, pp 99–107

  53. Zhao M, Zhu SC (2011) Portrait painting using active templates. In: NPAR. ACM, pp 117–124

  54. Zhou H, Kuang Z, Wong KYK (2012) Markov weight fields for face sketch synthesis. In: IEEE conference on computer vision and pattern recognition (CVPR), 2012, pp 1091–1097

Download references

Acknowledgments

This work was supported by a grant from National Program on Key Basic Research Project of China (973 Program, 2013CB329504), National High Technology Research and Development Program of China (863 Program, 2013AA013705), National Natural Science Foundation of China (N0. 61379067), and National Key Technology R & D Program of the Ministry of Science and Technology (2012BAH03F03).

Author information

Authors and Affiliations

  1. Department of Computer Science and Technology, Zhejiang University, Hangzhou, China

    Bingwen Jin & Weidong Geng

  2. Department of Information Systems, New Jersey Institute of Technology, Newark, NJ, USA

    Songhua Xu

Authors
  1. Bingwen Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Songhua Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Weidong Geng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bingwen Jin.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jin, B., Xu, S. & Geng, W. Learning to sketch human facial portraits using personal styles by case-based reasoning. Multimed Tools Appl 77, 5417–5441 (2018). https://doi.org/10.1007/s11042-017-4457-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-4457-8

Keywords

Search

Navigation