Skip to main content
SpringerLink
Log in

Twin Deep Convolutional Neural Network for Example-Based Image Colorization

  • Conference paper
  • First Online:
Computer Analysis of Images and Patterns (CAIP 2017)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 10424))

Included in the following conference series:

Abstract

This paper deals with the colorization of grayscale images. Recent papers have shown remarkable results on image colorization utilizing various deep architectures. Unlike previous methods, we perform colorization using a deep architecture and a reference image. Our architecture utilizes two parallel Convolutional Neural Networks which have the same structure. One CNN, which uses the reference image, helps the other CNN in color prediction for the input image. On the other hand, the second CNN, which uses the input image, helps to identify the areas which holds essential information about the color scheme of the scene. Comprehensive experiments and qualitative and quantitative evaluations were conducted on the images of SUN database and on other images. Quantitative evaluations are based on Peak Signal-to-Noise Ratio (PSNR) and on Quaternion Structural Similarity (QSSIM).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Krizhevsky, A., Sutskever, I., Hinton, G.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)

    Google Scholar 

  2. Nagy, B., Benedek, C.: 3D CNN based phantom object removing from mobile laser scanning data. In: International Joint Conference on Neural Networks, pp. 4429–4435 (2017)

    Google Scholar 

  3. Bochinski, E., Eiselein, V., Sikora, T.: Training a convolutional neural network for multi-class object detection using solely virtual world data. In: IEEE International Conference on Advanced Video and Signal Based Surveillance, pp. 278–285 (2016)

    Google Scholar 

  4. Lawrence, S., Giles, C.L., Tsoi, A.C., Back, A.D.: Face recognition: a Convolutional Neural Network approach. IEEE Trans. Neural Networks 8(1), 98–113 (1997)

    Article  Google Scholar 

  5. Ciresan, D., Meier, U.: Multi-column deep neural networks for offline handwritten Chinese character classification. In: Proceedings of the International Joint Conference on Neural Networks, pp. 1–6 (2015)

    Google Scholar 

  6. Yan, Z., Zhang, H., Wang, B., Paris, S., Yu, Y.: Automatic photo adjustment using deep learning. CoRR, abs/1412.7725 (2014)

    Google Scholar 

  7. Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. ACM Trans. Graph. 23(3), 689–694 (2004)

    Article  Google Scholar 

  8. Huang, Y.C., Tung, Y.S., Chen, J.C., Wang, S.W., Wu, J.L.: An adaptive edge detection based colorization algorithm and its applications. In: Proceedings of the 13th Annual ACM International Conference on Multimedia, pp. 351–354 (2005)

    Google Scholar 

  9. Yatziv, L., Sapiro, G.: Fast image and video colorization using chrominance blending. IEEE Trans. Image Process. 15(5), 1120–1129 (2006)

    Article  Google Scholar 

  10. Reinhard, E., Ashikhmin, M., Gooch, B., Shirley, P.: Color transfer between images. IEEE Comput. Graph. Appl. 21(5), 34–41 (2001)

    Article  Google Scholar 

  11. Welsh, T., Ashikhmin, M., Mueller, K.: Transfering color to greyscale images. ACM Trans. Graph. 21(3), 277–280 (2002)

    Article  Google Scholar 

  12. Irony, R., Cohen-Or, D., Lischinski, D.: Colorization by example. In: Eurographics Symposium on Rendering (2005)

    Google Scholar 

  13. Charpiat, G., Hofmann, M., Schölkopf, B.: Automatic image colorization via multimodal predictions. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5304, pp. 126–139. Springer, Heidelberg (2008). doi:10.1007/978-3-540-88690-7_10

    Chapter  Google Scholar 

  14. Gupta, R.K., Chia, A.Y.S., Rajan, D., Ng, E.S., Zhiyong, H.: Image colorization using similar images. In: Proceedings of the 20th ACM International Conference on Multimedia, pp. 369–378 (2012)

    Google Scholar 

  15. Bugeau, A., Ta, V.T.: Patch-based image colorization. In: Proceedings of the IEEE International Conference on Pattern Recognition, pp. 3058–3061 (2012)

    Google Scholar 

  16. Deshpande, A., Rock, J., Forsyth, D.: Learning large-scale automatic image colorization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 567–575 (2015)

    Google Scholar 

  17. Cheng, Z., Yang, Q., Sheng, B.: Deep colorization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 415–423 (2015)

    Google Scholar 

  18. Tola, E., Lepetit, V., Fua, P.: DAISY: an efficient dense descriptor applied to wide-baseline stereo. IEEE Trans. Pattern Anal. Mach. Intell. 32(5), 815–830 (2010)

    Article  Google Scholar 

  19. Iizuka, S., Simo-Serra, E., Ishikawa, H.: Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans. Graph. (TOG) 35(4), 110 (2016)

    Article  Google Scholar 

  20. Varga, D., Szirányi, T.: Fully automatic image colorization based on Convolutional Neural Network. In: International Conference on Pattern Recognition (2016)

    Google Scholar 

  21. Simonyan, K., Zisserman, A.: Very Deep Convolutional Networks for Large-Scale Image Recognition. CoRR, abs/1409.1556 (2014)

    Google Scholar 

  22. Kolaman, A., Yadid-Pecht, O.: Quaternion structural similarity: a new quality index for color images. IEEE Trans. Image Process. 21(4), 1526–1536 (2012)

    Article  MathSciNet  Google Scholar 

  23. Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 649–666. Springer, Cham (2016). doi:10.1007/978-3-319-46487-9_40

    Chapter  Google Scholar 

  24. Liang, X., Su, Z., Xiao, Y., Guo, J., Luo, X., Deep patch-wise colorization model for grayscale images. SIGGRAPH ASIA 2016 Technical Briefs 13 (2016)

    Google Scholar 

  25. He, K., Sun, J., Tang, X.: Guided image filtering. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6311, pp. 1–14. Springer, Heidelberg (2010). doi:10.1007/978-3-642-15549-9_1

    Chapter  Google Scholar 

  26. Larsson, G., Maire, M., Shakhnarovich, G.: Learning representations for automatic colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 577–593. Springer, Cham (2016). doi:10.1007/978-3-319-46493-0_35

    Chapter  Google Scholar 

  27. Hariharan, B., Arbeláez, P., Girshick, R., Malik, J.: Hypercolumns for object segmentation and fine-grained localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 447–456 (2015)

    Google Scholar 

  28. Deshpande, A., Lu, J., Yeh, M.C., Forsyth, D.: Learning Diverse Image Colorization. arXiv preprint arXiv:1612.01958 (2016)

  29. Cao, Y., Zhou, Z., Zhang, W., Yu, Y.: Unsupervised Diverse Colorization via Generative Adversarial Networks. arXiv preprint arXiv:1702.06674 (2017)

  30. Limmer, M., Lensch, H.: Infrared Colorization Using Deep Convolutional Neural Networks. arXiv preprint arXiv:1604.02245 (2016)

  31. Chollet, F.: Keras (2015). https://github.com/fchollet/keras

  32. Xiao, J., Hays, J., Ehinger, K., Oliva, A., Torralba, A.: SUN database: large-scale scene recognition from abbey to zoo. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 3485–3492 (2010)

    Google Scholar 

  33. Kingma, D., Adam, J.B.: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  34. Girosi, F., Jones, M., Poggio, T.: Regularization theory and neural networks architectures. Neural Comput. 7(2), 219–269 (1995)

    Article  Google Scholar 

Download references

Acknowledgment

The research was supported by the Hungarian Scientific Research Fund (No. OTKA 120499). We are very thankful to Levente Kovács for helping us with professional advices in high-performance computing.

Author information

Authors and Affiliations

  1. MTA SZTAKI, Institute for Computer Science and Control, Budapest, Hungary

    Domonkos Varga & Tamás Szirányi

  2. Department of Networked Systems and Services, Budapest University of Technology and Economics, Budapest, Hungary

    Domonkos Varga

  3. Department of Material Handling and Logistics Systems, Budapest University of Technology and Economics, Budapest, Hungary

    Tamás Szirányi

Authors
  1. Domonkos Varga
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tamás Szirányi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Domonkos Varga .

Editor information

Editors and Affiliations

  1. Linköping University, Linköping, Sweden

    Michael Felsberg

  2. Lund University, Lund, Sweden

    Anders Heyden

  3. University of Southern Denmark, Odense, Denmark

    Norbert Krüger

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Varga, D., Szirányi, T. (2017). Twin Deep Convolutional Neural Network for Example-Based Image Colorization. In: Felsberg, M., Heyden, A., Krüger, N. (eds) Computer Analysis of Images and Patterns. CAIP 2017. Lecture Notes in Computer Science(), vol 10424. Springer, Cham. https://doi.org/10.1007/978-3-319-64689-3_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-64689-3_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-64688-6

  • Online ISBN: 978-3-319-64689-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation