Mingming He
Skip Abstract Section
Skip Supplemental Material SectionAbstract
We propose the first deep learning approach for exemplar-based local colorization. Given a reference color image, our convolutional neural network directly maps a grayscale image to an output colorized image. Rather than using hand-crafted rules as in traditional exemplar-based methods, our end-to-end colorization network learns how to select, propagate, and predict colors from the large-scale data. The approach performs robustly and generalizes well even when using reference images that are unrelated to the input grayscale image. More importantly, as opposed to other learning-based colorization methods, our network allows the user to achieve customizable results by simply feeding different references. In order to further reduce manual effort in selecting the references, the system automatically recommends references with our proposed image retrieval algorithm, which considers both semantic and luminance information. The colorization can be performed fully automatically by simply picking the top reference suggestion. Our approach is validated through a user study and favorable quantitative comparisons to the-state-of-the-art methods. Furthermore, our approach can be naturally extended to video colorization. Our code and models are freely available for public use.
Supplemental Material
- Artem Babenko and Victor Lempitsky. 2015. Aggregating local deep features for image retrieval. In Proc. ICCV. 1269--1277. Google Scholar
Digital Library
- Artem Babenko, Anton Slesarev, Alexandr Chigorin, and Victor Lempitsky. 2014. Neural codes for image retrieval. In Proc. ECCV. Springer, 584--599.Google Scholar
- Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. 2015. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. arXiv preprint arXiv:1511.00561 (2015).Google Scholar
- Nicolas Bonneel, James Tompkin, Kalyan Sunkavalli, Deqing Sun, Sylvain Paris, and Hanspeter Pfister. 2015. Blind video temporal consistency. ACM Trans. Graph. (Proc. of SIGGRAPH Asia) 34, 6 (2015), 196. Google ScholarDigital Library
- Aurélie Bugeau and Vinh-Thong Ta. 2012. Patch-based image colorization. In Pattern Recognition (ICPR), 2012 21st International Conference on. IEEE, 3058--3061.Google Scholar
- Aurélie Bugeau, Vinh-Thong Ta, and Nicolas Papadakis. 2014. Variational exemplar-based image colorization. IEEE Trans. on Image Processing 23, 1 (2014), 298--307. Google ScholarDigital Library
- Guillaume Charpiat, Matthias Hofmann, and Bernhard Schölkopf. 2008. Automatic image colorization via multimodal predictions. (2008), 126--139. Google ScholarDigital Library
- Dongdong Chen, Jing Liao, Lu Yuan, Nenghai Yu, and Gang Hua. 2017a. Coherent online video style transfer. In Proc. ICCV.Google ScholarCross Ref
- Dongdong Chen, Lu Yuan, Jing Liao, Nenghai Yu, and Gang Hua. 2017b. Stylebank: An explicit representation for neural image style transfer. In Proc. CVPR.Google ScholarCross Ref
- Dongdong Chen, Lu Yuan, Jing Liao, Nenghai Yu, and Gang Hua. 2018. Stereoscopic Neural Style Transfer. In Proc. CVPR.Google ScholarCross Ref
- Qifeng Chen and Vladlen Koltun. 2017. Photographic image synthesis with cascaded refinement networks. In Proc. ICCV, Vol. 1.Google ScholarCross Ref
- Zezhou Cheng, Qingxiong Yang, and Bin Sheng. 2015. Deep colorization. In Proc. ICCV. 415--423. Google ScholarDigital Library
- Alex Yong-Sang Chia, Shaojie Zhuo, Raj Kumar Gupta, Yu-Wing Tai, Siu-Yeung Cho, Ping Tan, and Stephen Lin. 2011. Semantic colorization with internet images. ACM Trans. Graph. (Proc. of SIGGRAPH Asia) 30, 6 (2011), 156. Google ScholarDigital Library
- Aditya Deshpande, Jason Rock, and David Forsyth. 2015. Learning large-scale automatic image colorization. In Proc. ICCV. 567--575. Google ScholarDigital Library
- Yunchao Gong, Liwei Wang, Ruiqi Guo, and Svetlana Lazebnik. 2014. Multi-scale orderless pooling of deep convolutional activation features. In Proc. ECCV. Springer, 392--407.Google Scholar
- Raj Kumar Gupta, Alex Yong-Sang Chia, Deepu Rajan, Ee Sin Ng, and Huang Zhiyong. 2012. Image colorization using similar images. In Proc. of the 20th ACM international conference on Multimedia. ACM, 369--378. Google ScholarDigital Library
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385 (2015).Google Scholar
- Mingming He, Jing Liao, Lu Yuan, and Pedro V Sander. 2017. Neural Color Transfer between Images. arXiv preprint arXiv:1710.00756 (2017).Google Scholar
- Yi-Chin Huang, Yi-Shin Tung, Jun-Cheng Chen, Sung-Wen Wang, and Ja-Ling Wu. 2005. An adaptive edge detection based colorization algorithm and its applications. In Proc. of the 13th annual ACM international conference on Multimedia. ACM, 351--354. Google ScholarDigital Library
- Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2016. Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans. Graph. (Proc. of SIGGRAPH) 35,4 (2016), 110. Google ScholarDigital Library
- Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning. 448--456. Google ScholarDigital Library
- Revital Irony, Daniel Cohen-Or, and Dani Lischinski. 2005. Colorization by Example. In Rendering Techniques. 201--210. Google ScholarDigital Library
- Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-image translation with conditional adversarial networks. In Proc. CVPR.Google ScholarCross Ref
- Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In Proc. ECCV. Springer, 694--711.Google Scholar
- Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097--1105. Google ScholarDigital Library
- Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. 2016. Learning representations for automatic colorization. In Proc. ECCV. 577--593.Google ScholarCross Ref
- Anat Levin, Dani Lischinski, and Yair Weiss. 2004. Colorization using optimization. ACM Trans. Graph. (Proc. of SIGGRAPH) 23, 3 (2004), 689--694. Google ScholarDigital Library
- Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, and Sing Bing Kang. 2017. Visual Attribute Transfer through Deep Image Analogy. arXiv preprint arXiv:1705.01088 36, 4 (2017), 120. Google ScholarDigital Library
- Ce Liu, Jenny Yuen, and Antonio Torralba. 2011. Sift flow: Dense correspondence across scenes and its applications. IEEE Trans. Pattern Anal. Mach. Intell. 33, 5 (2011), 978--994. Google ScholarDigital Library
- Xiaopei Liu, Liang Wan, Yingge Qu, Tien-Tsin Wong, Stephen Lin, Chi-Sing Leung, and Pheng-Ann Heng. 2008. Intrinsic colorization. ACM Trans. Graph. (Proc. of SIGGRAPH Asia) 27, 5 (2008), 152. Google ScholarDigital Library
- Qing Luan, Fang Wen, Daniel Cohen-Or, Lin Liang, Ying-Qing Xu, and Heung-Yeung Shum. 2007. Natural image colorization. In Proc. of the 18th Eurographics conference on Rendering Techniques. Eurographics Association, 309--320. Google ScholarDigital Library
- Yingge Qu, Tien-Tsin Wong, and Pheng-Ann Heng. 2006. Manga colorization. ACM Trans. Graph. (Proc. of SIGGRAPH Asia) 25, 3 (2006), 1214--1220. Google ScholarDigital Library
- Alec Radford, Luke Metz, and Soumith Chintala. 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015).Google Scholar
- Ali Sharif Razavian, Josephine Sullivan, Stefan Carlsson, and Atsuto Maki. 2016. A baseline for visual instance retrieval with deep convolutional networks. arXiv preprint arXiv:1412.6574 (2016).Google Scholar
- Erik Reinhard, Michael Adhikhmin, Bruce Gooch, and Peter Shirley. 2001. Color transfer between images. IEEE Computer graphics and applications 21, 5 (2001), 34--41. Google ScholarDigital Library
- Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, and others. 2015. Imagenet large scale visual recognition challenge. International Journal of Computer Vision 115, 3 (2015), 211--252. Google ScholarDigital Library
- Mehdi SM Sajjadi, Bernhard Scholkopf, and Michael Hirsch. 2017. Enhancenet: Single image super-resolution through automated texture synthesis. In Proc. CVPR. 4491--4500.Google ScholarCross Ref
- Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, and James Hays. 2016. Scribbler: Controlling deep image synthesis with sketch and color. In Proc. CVPR.Google Scholar
- Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going Deeper with Convolutions. In Proc. CVPR.Google ScholarCross Ref
- Yu-Wing Tai, Jiaya Jia, and Chi-Keung Tang. 2005. Local color transfer via probabilistic segmentation by expectation-maximization. In Proc. CVPR, Vol. 1. IEEE, 747--754. Google ScholarDigital Library
- Engin Tola, Vincent Lepetit, and Pascal Fua. 2010. Daisy: An efficient dense descriptor applied to wide-baseline stereo. IEEE Trans. Pattern Anal. Mach. Intell. 32, 5 (2010), 815--830. Google ScholarDigital Library
- Giorgos Tolias, Ronan Sicre, and Hervé Jégou. 2015. Particular object retrieval with integral max-pooling of CNN activations. arXiv preprint arXiv:1511.05879 (2015).Google Scholar
- Philippe Weinzaepfel, Jerome Revaud, Zaid Harchaoui, and Cordelia Schmid. 2013. DeepFlow: Large displacement optical flow with deep matching. In Proc. ICCV. 1385--1392. Google ScholarDigital Library
- Tomihisa Welsh, Michael Ashikhmin, and Klaus Mueller. 2002. Transferring color to greyscale images. ACM Trans. Graph. (Proc. of SIGGRAPH Asia) 21, 3 (2002), 277--280. Google ScholarDigital Library
- Liron Yatziv and Guillermo Sapiro. 2006. Fast image and video colorization using chrominance blending. IEEE Trans. on Image Processing 15, 5 (2006), 1120--1129. Google ScholarDigital Library
- Fisher Yu and Vladlen Koltun. 2015. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122 (2015).Google Scholar
- Richard Zhang, Phillip Isola, and Alexei A Efros. 2016. Colorful image colorization. In Proc. ECCV. 649--666.Google ScholarCross Ref
- Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S Lin, Tianhe Yu, and Alexei A Efros. 2017. Real-time user-guided image colorization with learned deep priors. ACM Trans. Graph. (Proc. of SIGGRAPH) 36, 4 (2017), 119. Google ScholarDigital Library
Index Terms
- Deep exemplar-based colorization
Recommendations
Real-time user-guided image colorization with learned deep priors
We propose a deep learning approach for user-guided image colorization. The system directly maps a grayscale image, along with sparse, local user "hints" to an output colorization with a Convolutional Neural Network (CNN). Rather than using hand-defined ...
A Novel Approach for Colorization of a Grayscale Image using Soft Computing Techniques
Colorization of grayscale image is a process to convert a grayscale image into a color one. Few research works reported in literature on this but there is hardly any generalized method that successfully colorizes all types of grayscale image. This study ...
Semantic-Sparse Colorization Network for Deep Exemplar-Based Colorization
Computer Vision – ECCV 2022AbstractExemplar-based colorization approaches rely on reference image to provide plausible colors for target gray-scale image. The key and difficulty of exemplar-based colorization is to establish an accurate correspondence between these two images. ...
Comments