Skip to main content
SpringerLink
Log in

ParaColorizer-Realistic image colorization using parallel generative networks

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Image colorization is a fascinating application of AI for information restoration. The inherently ill-posed nature of the problem increases the challenge since the outputs could be multimodal. Existing learning-based methods produce acceptable results for straightforward cases but usually fail to restore the contextual information without clear figure-ground separation. Also, the images suffer from color bleeding and desaturated backgrounds since a single model trained on full-image features is insufficient for learning the diverse data modes. This work presents a parallel generative adversarial network (GAN)-based colorization framework to address these issues. The proposed framework uses parallel GANs tailored to colorize the foreground (using object-level features) and the background (using full-image features) independently and performs unbalanced GAN training. We develop a DenseFuse-based fusion network to obtain the final colorized image by feature-based fusion of the parallelly generated intermediate outputs. We conduct extensive performance evaluations and ablation studies of our framework with multiple perceptual metrics, including human evaluation. Our approach outperforms most existing learning-based methods and produces results comparable to the state of the art. The runtime analysis experiments revealed an average inference time of 24 milliseconds (ms) per image, and thus the proposed framework can colorize the grayscale images in real time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data availability

The datasets were obtained from public domain resources available at https://cocodataset.org/, http://places.csail.mit.edu/, and https://image-net.org/.

References

  1. Charpiat, G., Hofmann, M., Schölkopf, B.: Automatic image colorization via multimodal predictions. In: European Conference on Computer Vision, pp. 126–139. Springer, (2008)

  2. Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. In ACM SIGGRAPH 2004 Papers, pp. 689–694. (2004)

  3. Huang, Y. C., Tung, Y. S., Chen, J. C., Wang, S. W., Wu, J. L.: An adaptive edge detection based colorization algorithm and its applications. In: Proceedings of the 13th Annual ACM International Conference on Multimedia, pp. 351–354. (2005)

  4. Yatziv, L., Sapiro, G.: Fast image and video colorization using chrominance blending. Trans. Img. Proc. 15(5), 1120–1129 (2006)

    Article  Google Scholar 

  5. Qu, Y., Wong, T.T., Heng, P.A.: Manga colorization. ACM Trans. Graph. 25(3), 1214–1220 (2006)

    Article  Google Scholar 

  6. Luan, Q., Wen, F., Cohen-Or, D., Liang, L., Xu, Y.Q., Shum, H.Y.: Natural image colorization. In: Proceedings of the 18th Eurographics Conference on Rendering Techniques, EGSR’07, pp. 309-320, Goslar, DEU, (2007). Eurographics Association

  7. Sýkora, D., Dingliana, J., Collins, S.: Lazybrush: Flexible painting tool for hand–drawn cartoons. Computer Graphics Forum, 28, (2009)

  8. Welsh, T., Ashikhmin, M., Mueller, K.: Transferring color to greyscale images. ACM Trans. Graph. 21(3), 277–280 (2002)

    Article  Google Scholar 

  9. Ironi, R., Cohen-Or, D., Lischinski, D.: Colorization by example. Render. Tech. 29, 201–210 (2005)

    Google Scholar 

  10. Gupta, R.K., Chia, A.Y.S., Rajan, D., Ng, E.S., Zhiyong, H.: Image colorization using similar images. In Proceedings of the 20th ACM international conference on Multimedia, pp. 369–378. (2012)

  11. Liu, X., Wan, L., Qu, Y., Wong, T.T., Lin, S., Leung, C.S., Heng, P.A.: Intrinsic colorization. In: ACM SIGGRAPH Asia 2008 papers, pp. 1–9. (2008)

  12. Iizuka, S., Simo-Serra, E., Ishikawa, H.: Let there be color! joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans. Graph.(ToG) 35(4), 1–11 (2016)

    Article  Google Scholar 

  13. Larsson, G., Maire, M., Shakhnarovich, G.: Learning representations for automatic colorization. In: European Conference on Computer Vision, pp. 577–593. Springer, (2016)

  14. Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.), Computer Vision—ECCV 2016, pp. 649–666, Springer International Publishing, Cham, (2016)

  15. Zhang, R., Zhu, J.Y., Isola, P., Geng, X., Lin, A.S., Yu, T., Efros, A.A.: Real-time user-guided image colorization with learned deep priors, (2017)

  16. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134. (2017)

  17. Zhao, J., Liu, L., Snoek, C. G., Han, J., Shao, L.: Pixel-level semantics guided image colorization, (2018)

  18. Mingming, H., Dongdong, C., Jing, L., Sander, P.V., Yuan, L.: Deep exemplar-based colorization. ACM Trans. Graph. (TOG) 37(4), 1–16 (2018)

    Google Scholar 

  19. Guadarrama, S., Dahl, R., Bieber, D. Norouzi, M., Shlens, J., Murphy, K.: Pixcolor: Pixel recursive colorization. arXiv preprint arXiv:1705.07208, (2017)

  20. Kolesnikov, A., Royer, A., Lampert, C.: Probabilistic image colorization. In: Brostow, G., Kim, T.K., Zafeiriou, S., Mikolajczyk, K. (eds.) Proceedings of the British Machine Vision Conference (BMVC), pp. 85.1–85.12. BMVA Press, September (2017)

  21. Deshpande, A., Lu, J., Yeh, M.C., Jin Chong, M., Forsyth, D.: Learning diverse image colorization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6837–6845. (2017)

  22. Messaoud, S., Forsyth, D., Schwing, A.G.: Structural consistency and controllability for diverse colorization. In: Proceedings of the European Conference on Computer Vision (ECCV), September (2018)

  23. Antic, J: jantic/deoldify: A deep learning based project for colorizing and restoring old images (and video!), (2019)

  24. Su, J.W., Chu., H.K., Huang, J.B.: Instance-aware image colorization. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7965–7974. (2020)

  25. Sun, Q., Chen, Y., Tao, W., Han Jiang, M., Zhang, K.C., Erdt, M.: A gan-based approach toward architectural line drawing colorization prototyping. Vis. Comput. 38(4), 1283–1300 (2022)

    Article  Google Scholar 

  26. Min, W., Jin, X., Jiang, Q., Lee, S., Liang, W., Lin, G., Yao, S.: Remote sensing image colorization using symmetrical multi-scale dcgan in yuv color space. Vis. Comput. 37, 1707–1729 (2021)

    Article  Google Scholar 

  27. Xiong, K., Hong, K., Li, J., Li, W., Liao, W., Liu, Q.: Joint intensity-gradient guided generative modeling for colorization. Vis. Comput. pp. 1432–2315, (2022)

  28. Nazeri, K., Ng, E., Ebrahimi, M: Image colorization using generative adversarial networks. In: International Conference on Articulated Motion and Deformable Objects, pp. 85–94. Springer, (2018)

  29. Vitoria, P., Raad, L., Ballester, C.: Chromagan: adversarial picture colorization with semantic class distribution. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2445–2454. (2020)

  30. Zhang, L., Ji, Y., Lin, X., Liu, C.: Style transfer for anime sketches with enhanced residual u-net and auxiliary classifier gan. In: 2017 4th IAPR Asian Conference on Pattern Recognition (ACPR), pp. 506–511. IEEE, (2017)

  31. O’Hare, N., Smeaton, A.F.: Context-aware person identification in personal photo collections. IEEE Trans. Multimed. 11(2), 220–228 (2009)

    Article  Google Scholar 

  32. Ham, H.G., Jun, T.J., Kim, D.: Unbalanced gans: pre-training the generator of generative adversarial network using variational autoencoder. arXiv:2002.02112, (2020)

  33. Li, H., Xiao-Jun, W.: Densefuse: A fusion approach to infrared and visible images. IEEE Trans. Image Process. 28(5), 2614–2623 (2019)

    Article  MathSciNet  Google Scholar 

  34. Hoang, Q.M., Nguyen, T.D., Le, T., Phung, D.Q.: Mgan: training generative adversarial nets with multiple generators. In ICLR, (2018)

  35. An, X., Pellacini, F.: Appprop: all-pairs appearance-space edit propagation. ACM Trans. Graph., 27(3), (2008)

  36. Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2414–2423. (2016)

  37. Freedman, D., Kisilev, P.: Object-to-object color transfer: optimal flows and smsp transformations. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 287–294, (2010)

  38. Pitie, F., Kokaram, A.C., Dahyot, R.: N-Dimensional probability density function transfer and its application to color transfer. In: Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1, vol. 2, pp. 1434–1439. (2005)

  39. Laffont, P.Y., Ren, Z., Tao, X., Qian, C., Hays, J.: Transient attributes for high-level understanding and editing of outdoor scenes. ACM Trans. Graph. 33(4), 1–11 (2014)

    Article  Google Scholar 

  40. Shih, Y.C., Paris, S., Barnes, C., Freeman, W.T., Frédo, D.: Style transfer for headshot portraits. ACM Trans. Graph., 33, (2014)

  41. Wu, F., Dong, W., Kong, Y., Mei, X., Paul, J.C., Zhang, X.:: Content-based colour transfer. In: Computer Graphics Forum, pp. 190–203. Wiley Online Library, 2013

  42. Zhang, B., He, M., Liao, J., Sander, P. V., Yuan, L., Bermak, A., Chen, D.: Deep exemplar-based video colorization, (2019)

  43. Cheng, Z., Yang, Q., Sheng, B.: Deep colorization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 415–423, (2015)

  44. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255, (2009)

  45. He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969. (2017)

  46. Zhang, H., Goodfellow, I., Metaxas, D., Odena, A.: Self-attention generative adversarial networks. In: International Conference on Machine Learning, pp. 7354–7363. PMLR, (2019)

  47. Mao, X., Li, Q., Xie, H., Lau, R.Y.K., Wang, Z., Paul Smolley, S.: Least squares generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2794–2802, (2017)

  48. Xia, J., Tan, G., Xiao, Y., Xu, F., Leung, C.S.: Edge-aware multi-scale progressive colorization. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1655–1659. IEEE, (2021)

  49. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: common objects in context. In: European Conference on Computer Vision, pp. 740–755. Springer, (2014)

  50. Kastryulin, S., Zakirov, J., Prokopenko, D., Dylov, D.V.: Pytorch image quality: Metrics for image quality assessment, (2022)

  51. Maximilian Seitzer. pytorch-fid: Fid score for pytorch. https://github.com/mseitzer/pytorch-fid, (2020)

  52. Wu, Y., Wang, X., Li, Y., Zhang, H., Zhao, X., Shan, Y.: Towards vivid and diverse image colorization with generative color prior. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14377–14386, (2021)

  53. Hasler, D., Suesstrunk, S.E.: Measuring colorfulness in natural images. In: Human Vision and Electronic Imaging VIII, vol. 5007, pp. 87–95. International Society for Optics and Photonics, (2003)

  54. Zhou, B., Khosla, A., Lapedriza, A., Torralba, A., Oliva, A.: Places: an image database for deep scene understanding. arXiv preprint arXiv:1610.02055, (2016)

  55. Kumar, M., Weissenborn, D., Kalchbrenner, N.: Colorization transformer. arXiv preprint arXiv:2102.04432, (2021)

Download references

Acknowledgements

The authors extend their gratitude to the CSIR-CEERI Director for supporting AI-related research and to the volunteers for participating in the human evaluation test. All computations were performed using the GPU resources provided by the AI Computing Facility, CSIR-CEERI.

Author information

Author notes

  1. Himanshu Kumar and Abeer Banerjee have contributed equally to this work.

Authors and Affiliations

  1. CSIR-Central Electronics Engineering Research Institute (CSIR-CEERI), Pilani, 333031, India

    Himanshu Kumar, Abeer Banerjee, Sumeet Saurav & Sanjay Singh

  2. Academy of Scientific and Innovative Research (AcSIR), Ghaziabad, 201002, India

    Himanshu Kumar, Abeer Banerjee, Sumeet Saurav & Sanjay Singh

Authors
  1. Himanshu Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Abeer Banerjee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sumeet Saurav
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sanjay Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Abeer Banerjee.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kumar, H., Banerjee, A., Saurav, S. et al. ParaColorizer-Realistic image colorization using parallel generative networks. Vis Comput (2023). https://doi.org/10.1007/s00371-023-03067-7

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00371-023-03067-7

Keywords

Search

Navigation