Skip to main content
SpringerLink
Log in

StencilTorch: An Iterative and User-Guided Framework for Anime Lineart Colorization

  • Conference paper
  • First Online:
Image and Vision Computing (IVCNZ 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13836))

Included in the following conference series:

  • 767 Accesses

Abstract

Automatic lineart colorization is a challenging task for Computer Vision. Contrary to grayscale images, linearts lack semantic information such as shading and texture, making the task even more difficult. Modern approaches train a Generative Adversarial Network (GAN) to generate illustrations from user inputs such as color hints. While such approaches can generate high-quality outputs in real-time, the user only interacts with the pipeline once at the beginning of the process. This paper presents StencilTorch, an interactive and user-guided framework for anime lineart colorization motivated by digital artist workflows. StencilTorch generates illustrations from a given lineart, color hints, and a mask allowing for iterative workflows where the output of the first pass becomes the input of a second. Our method improves previous work on both objective and subjective evaluations.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Anonymous, community, D., Branwen, G.: Danbooru 2020: A large-scale crowdsourced and tagged anime illustration dataset, January 2021. https://www.gwern.net/Danbooru2020

  2. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: Proceedings of the 34th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 70, pp. 214–223. PMLR, International Convention Centre, Sydney, Australia, 06–11 Aug 2017. https://proceedings.mlr.press/v70/arjovsky17a.html

  3. Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning, pp. 41–48. ICML 2009, Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/1553374.1553380

  4. Chollet, F.: Xception: Deep learning with depthwise separable convolutions. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1800–1807 (2017). https://doi.org/10.1109/CVPR.2017.195

  5. Ci, Y., Ma, X., Wang, Z., Li, H., Luo, Z.: User-guided deep anime line art colorization with conditional adversarial networks. In: Proceedings of the 26th ACM International Conference on Multimedia, pp. 1536–1544. MM 2018, Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3240508.3240661

  6. Elman, J.L.: Learning and development in neural networks: the importance of starting small. Cognition 48(1), 71–99 (1993)

    Google Scholar 

  7. Frans, K.: Outline colorization through tandem adversarial networks. CoRR abs/1704.08834 (2017). arxiv:1704.08834

  8. Furusawa, C., Hiroshiba, K., Ogaki, K., Odagiri, Y.: Comicolorization: semi-automatic manga colorization. In: SIGGRAPH Asia 2017 Technical Briefs, SA 2017, Association for Computing Machinery, New York, NY, USA (2017). https://doi.org/10.1145/3145749.3149430

  9. Goodfellow, I.J., et al.: Generative adversarial nets. In: Proceedings of the 27th International Conference on Neural Information Processing Systems, vol. 2, pp. 2672–2680. NIPS 2014, MIT Press, Cambridge, MA, USA (2014). https://dl.acm.org/doi/10.5555/2969033.2969125

  10. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of Wasserstein GANs. In: Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017), https://proceedings.neurips.cc/paper/2017/file/892c3b1c6dccd52936e27cbd0ff683d6-Paper.pdf

  11. Hacohen, G., Weinshall, D.: On the power of curriculum learning in training deep networks. In: Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9–15 June 2019, Long Beach, California, USA. Proceedings of Machine Learning Research, vol. 97, pp. 2535–2544. PMLR (2019). https://proceedings.mlr.press/v97/hacohen19a.html

  12. Hati, Y., Jouet, G., Rousseaux, F., Duhart, C.: PaintsTorch: a user-guided anime line art colorization tool with double generator conditional adversarial network. In: European Conference on Visual Media Production. CVMP 2019, Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3359998.3369401

  13. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016

    Google Scholar 

  14. Hensman, P., Aizawa, K.: CGAN-based manga colorization using a single training image. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), vol. 3, pp. 72–77. IEEE Computer Society, Los Alamitos, CA, USA, Nov 2017. https://doi.org/10.1109/ICDAR.2017.295, https://doi.ieeecomputersociety.org/10.1109/ICDAR.2017.295

  15. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6629–6640. NIPS’17, Curran Associates Inc., Red Hook, NY, USA (2017)

    Google Scholar 

  16. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)

    Google Scholar 

  17. Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Adv. Neural. Inf. Process. Syst. 33, 6840–6851 (2020)

    Google Scholar 

  18. Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Bach, F., Blei, D. (eds.) Proceedings of the 32nd International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 37, pp. 448–456. PMLR, Lille, France, 07–09 July 2015. https://proceedings.mlr.press/v37/ioffe15.html

  19. Kandinsky, W., Sadleir, M.: Concerning the Spiritual in Art. Dover Publications, New York (1977). (oCLC: 3042682)

    Google Scholar 

  20. Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. CoRR abs/1710.10196 (2017). arxiv.org:1710.10196

  21. Kim, H., Jhoo, H.Y., Park, E., Yoo, S.: Tag2pix: line art colorization using text tag with Secat and changing loss. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9055–9064 (2019). https://doi.org/10.1109/ICCV.2019.00915

  22. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. In: 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14–16, 2014, Conference Track Proceedings (2014). arxiv.org:1312.6114

  23. Kingma, D.P., Dhariwal, P.: Glow: Generative flow with invertible 1 \(\times \) 1 convolutions. In: Advances in Neural Information Processing Systems, vol. 31 (2018)

    Google Scholar 

  24. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)

    Google Scholar 

  25. Lim, J.H., Ye, J.C.: Geometric GAN (2017)

    Google Scholar 

  26. Liu, Y., Qin, Z., Wan, T., Luo, Z.: Auto-painter: cartoon image generation from sketch by using conditional Wasserstein generative adversarial networks. Neurocomputing 311, 78–87 (2018)

    Google Scholar 

  27. Loshchilov, I., Hutter, F.: Fixing weight decay regularization in adam. CoRR abs/1711.05101 (2017). arxiv.org:1711.05101

  28. Zhang, L., Ji, Y., Liu, C.: DanbooRegion: an illustration region dataset. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12358, pp. 137–154. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58601-0_9

    Chapter  Google Scholar 

  29. Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)

  30. Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks. CoRR abs/1802.05957 (2018). arxiv.org:1802.05957

  31. Pixiv: Pelica Paint. https://petalica-paint.pixiv.dev/index_en.html (2017)

  32. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip LATENTs. arXiv preprint arXiv:2204.06125 (2022)

  33. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  34. Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vision (IJCV) 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y

    Article  MathSciNet  Google Scholar 

  35. Saito, M., Matsui, Y.: Illustration2vec: a semantic vector representation of illustrations. In: SIGGRAPH Asia 2015 Technical Briefs, pp. 5:1–5:4. SA 2015, ACM, New York, NY, USA (2015). https://doi.org/10.1145/2820903.2820907

  36. Sangkloy, P., Lu, J., Fang, C., Yu, F., Hays, J.: Scribbler: controlling deep image synthesis with sketch and color. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, 21–26 July 2017, pp. 6836–6845. IEEE Computer Society (2017). https://doi.org/10.1109/CVPR.2017.723

  37. Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)

    Google Scholar 

  38. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015, Conference Track Proceedings (2015). arxiv.org:1409.1556

  39. Szegedy, C., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)

    Google Scholar 

  40. Vaswani, A., et al.: Attention is all you need. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017). https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

  41. Winnemöller, H., Kyprianidis, J.E., Olsen, S.C.: XDOG: an extended difference-of-Gaussians compendium including advanced image stylization. Comput. Graph. 36(6), 740–753 (2012). https://doi.org/10.1016/j.cag.2012.03.004, www.sciencedirect.com/science/article/pii/S009784931200043X, 2011 Joint Symposium on Computational Aesthetics (CAe), Non-Photorealistic Animation and Rendering (NPAR), and Sketch-Based Interfaces and Modeling (SBIM)

  42. Xie, S., Girshick, R.B., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5987–5995 (2017)

    Google Scholar 

  43. Zhang, L., Ji, Y., Lin, X., Liu, C.: Style transfer for anime sketches with enhanced residual u-net and auxiliary classifier GAN. In: 2017 4th IAPR Asian Conference on Pattern Recognition (ACPR), pp. 506–511 (2017). https://doi.org/10.1109/ACPR.2017.61

  44. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR (2018)

    Google Scholar 

  45. Zhang, R., et al.: Real-time user-guided image colorization with learned deep priors. ACM Trans. Graph. 36(4), 1–11 (2017). https://doi.org/10.1145/3072959.3073703

Download references

Author information

Authors and Affiliations

  1. University of Reims Champagne-Ardenne, Laboratory CReSTIC, 51 100, Reims, France

    Yliess Hati, Florent Nolot & Francis Rousseaux

  2. Léonard de Vinci Pôle Universitaire, Research Center, 92 916, Paris La Défense, France

    Yliess Hati, Vincent Thevenin & Clement Duhart

Authors
  1. Yliess Hati
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vincent Thevenin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Florent Nolot
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Francis Rousseaux
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Clement Duhart
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yliess Hati .

Editor information

Editors and Affiliations

  1. Auckland University of Technology, Auckland, New Zealand

    Wei Qi Yan

  2. Auckland University of Technology, Auckland, New Zealand

    Minh Nguyen

  3. Auckland University of Technology, Auckland, New Zealand

    Martin Stommel

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hati, Y., Thevenin, V., Nolot, F., Rousseaux, F., Duhart, C. (2023). StencilTorch: An Iterative and User-Guided Framework for Anime Lineart Colorization. In: Yan, W.Q., Nguyen, M., Stommel, M. (eds) Image and Vision Computing. IVCNZ 2022. Lecture Notes in Computer Science, vol 13836. Springer, Cham. https://doi.org/10.1007/978-3-031-25825-1_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25825-1_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25824-4

  • Online ISBN: 978-3-031-25825-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation