Abstract
We introduce a method to stylize photographs with auxiliary textures, by means of the Laplacian pyramid. Laplacian pyramid coefficients from a synthetic texture are combined with the coefficients from the original image by means of a smooth maximum function. The final result is a stylized image which maintains the structural characteristics from the input, including edges, color, and existing texture, while enhancing the image with additional fine-scale details. Further, we extend patch-based texture synthesis to include a guidance channel so that texture structures are aligned with an orientation field, obtained through the image structure tensor.
Similar content being viewed by others
References
Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Susstrunk, S.: SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2274–2282 (2012)
Akl, A., Yaacoub, C., Donias, M., Da Costa, J.P., Germain, C.: Texture synthesis using the structure tensor. IEEE Trans. Image Process. 24, 4082–4095 (2015)
Ashikhmin, M.: Synthesizing natural textures. In: Proceedings of the 2001 Symposium on Interactive 3D Graphics, I3D ’01, pp. 217–226. ACM, New York, NY, USA (2001)
Aubry, M., Paris, S., Hasinoff, S.W., Kautz, J., Durand, F.: Fast local Laplacian filters: theory and applications. ACM Trans. Graph. 33(5), 167:1–167:14 (2014)
Bae, S., Paris, S., Durand, F.: Two-scale tone management for photographic look. ACM Trans. Graph. 25(3), 637–645 (2006)
Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.: Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28(3), 24 (2009)
Brox, T., van den Boomgaard, R., Lauze, F., van de Weijer, J., Weickert, J., Mrázek, P., Kornprobst, P.: Adaptive structure tensors and their applications. In: J. Weickert, H. Hagen (eds.) Visualization and Processing of Tensor Fields. Mathematics and Visualization, chap. 2, pp. 17–47. Springer, Berlin (2006)
Burt, P., Adelson, E.: The laplacian pyramid as a compact image code. IEEE Trans. Commun. 31(4), 532–540 (1983)
Burt, P.J., Adelson, E.H.: A multiresolution spline with application to image mosaics. ACM Trans. Graph. 2(4), 217–236 (1983)
Cabral, B., Leedom, L.C.: Imaging vector fields using line integral convolution. In: Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’93, pp. 263–270. ACM, New York, NY, USA (1993)
Cook, J.: Basic properties of the soft maximum. UT MD Anderson Cancer Center Department of Biostatistics Working Paper Series. Working Paper 70 (2011). http://biostats.bepress.com/cgi/viewcontent.cgi?article=1073&context=mdandersonbiostat
Criminisi, A., Sharp, T., Rother, C., Pérez, P.: Geodesic image and video editing. ACM Trans. Graph. 29(5), 134:1–134:15 (2010)
Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)
Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. In: Proceedings of SIGGRAPH 98, SIGGRAPH ’01, pp. 341–346. ACM, New York, NY, USA (2001)
Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: Proceedings of the International Conference on Computer Vision, ICCV ’99, vol. 2, p. 1033. IEEE Computer Society, Washington, DC, USA (1999)
Elad, M., Milanfar, P.: Style transfer via texture synthesis. IEEE Trans. Image Process. 26(5), 2338–2351 (2017). https://doi.org/10.1109/TIP.2017.2678168
Fang, H., Hart, J.C.: Textureshop: texture synthesis as a photograph editing tool. ACM Trans. Graph. 23(3), 354–359 (2004)
Fattal, R., Agrawala, M., Rusinkiewicz, S.: Multiscale shape and detail enhancement from multi-light image collections. ACM Trans. Graph. 26(3), 51 (2007)
Fišer, J., Jamriška, O., Simons, D., Shechtman, E., Lu, J., Asente, P., Lukáč, M., Sýkora, D.: Example-based synthesis of stylized facial animations. ACM Trans. Graph. 36(4), 155:1–155:11 (2017)
Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2414–2423 (2016)
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Z. Ghahramani, M. Welling, C. Cortes, N.D. Lawrence, K.Q. Weinberger (eds.) Advances in Neural Information Processing Systems 27, pp. 2672–2680. Curran Associates, Inc. (2014)
Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B., Salesin, D.H.: Image analogies. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’01, pp. 327–340. ACM, New York, NY, USA (2001)
Ignatov, A., Kobyshev, N., Timofte, R., Vanhoey, K., Van Gool, L.: DSLR-quality photos on mobile devices with deep convolutional networks. In: The IEEE International Conference on Computer Vision (ICCV) (2017)
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision—ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part II, pp. 694–711. Springer, Cham (2016)
Khan, E.A., Reinhard, E., Fleming, R.W., Bülthoff, H.H.: Image-based material editing. ACM Trans. Graph. 25(3), 654–663 (2006)
Kwatra, V., Essa, I., Bobick, A., Kwatra, N.: Texture optimization for example-based synthesis. ACM Trans. Graph. 24(3), 795–802 (2005)
Kwatra, V., Schödl, A., Essa, I., Turk, G., Bobick, A.: Graphcut textures: image and video synthesis using graph cuts. ACM Trans. Graph. 22(3), 277–286 (2003)
Lee, H., Seo, S., Yoon, K.: Extended papers from NPAR 2010: directional texture transfer with edge enhancement. Comput. Graph. 35(1), 81–91 (2011)
Liang, L., Liu, C., Xu, Y.Q., Guo, B., Shum, H.Y.: Real-time texture synthesis by patch-based sampling. ACM Trans. Graph. 20(3), 127–150 (2001)
Okura, F., Vanhoey, K., Bousseau, A., Efros, A.A., Drettakis, G.: Unifying color and texture transfer for predictive appearance manipulation. Comput. Graph. Forum 34(4), 53–63 (2015)
Paris, S., Hasinoff, S.W., Kautz, J.: Local Laplacian filters: edge-aware image processing with a Laplacian pyramid. ACM Trans. Graph. 30(4), 68:1–68:12 (2011)
Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. In: ACM Trans. Graph., vol. 22, pp. 313–318. ACM (2003)
Rother, C., Kolmogorov, V., Blake, A.: Grabcut: Interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. 23(3), 309–314 (2004)
Sajjadi, M.S., Schölkopf, B., Hirsch, M.: Enhancenet: Single image super-resolution through automated texture synthesis. arXiv preprint arXiv:1612.07919 (2016)
Semmo, A., Limberger, D., Kyprianidis, J.E., Döllner, J.: Image stylization by oil paint filtering using color palettes. In: Proceedings of the Workshop on Computational Aesthetics, CAE ’15, pp. 149–158. Eurographics Association, Goslar Germany, Germany (2015)
Shih, Y., Paris, S., Barnes, C., Freeman, W.T., Durand, F.: Style transfer for headshot portraits. ACM Trans. Graph. 33(4), 148:1–148:14 (2014)
Shih, Y., Paris, S., Durand, F., Freeman, W.T.: Data-driven hallucination of different times of day from a single outdoor photo. ACM Trans. Graph. 32(6), 200:1–200:11 (2013)
Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: Computer Vision, pp. 839–846. IEEE (1998)
Wang, B., Wang, W., Yang, H., Sun, J.: Efficient example-based painting and synthesis of 2d directional texture. IEEE Trans. Vis. Comput. Graph. 10(3), 266–277 (2004)
Wei, L.Y., Lefebvre, S., Kwatra, V., Turk, G.: State of the Art in Example-based Texture Synthesis. Eurographics 2009. State of the Art Report, EG-STAR, pp. 93–117. Eurographics Association, Munich, Germany (2009)
Wei, L.Y., Levoy, M.: Fast texture synthesis using tree-structured vector quantization. In: Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp. 479–488. ACM Press/Addison-Wesley Publishing Co. (2000)
Wexler, Y., Shechtman, E., Irani, M.: Space–time completion of video. IEEE Trans. Pattern Anal. Mach. Intell. 29(3), 463–476 (2007). https://doi.org/10.1109/TPAMI.2007.60
Zhou, Y., Shi, H., Lischinski, D., Gong, M., Kopf, J., Huang, H.: Analysis and controlled synthesis of inhomogeneous textures. Comput. Graph. Forum 36(2), 199–212 (2017)
Acknowledgements
We would like to thank the anonymous reviewers for many insightful comments. Thanks also to Eric Paquette and members of the Graphics, Imaging and Games Lab for productive comments and discussions. Funding for this work was provided by NSERC and by Carleton University. We used many images from Flickr under a Creative Commons license. Thanks to the numerous photographers who provided material: Bill Showalter (barn), Jim Sorbie (cabin), Alana Sise (rocky hill), Sérgio Sakakibara (cameraman), Harry Rose (grass), delta ! (girl), bananaana04 (parrot), Maria Morri (earring), Donald Lammers (ruin), Tim Adams (walkway), Ryan Basilio (berries), Gábor Lengyel (portrait), Stephanie Kroos (lion), and Martin Pettitt (car).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Doyle, L., Mould, D. Augmenting photographs with textures using the Laplacian pyramid. Vis Comput 35, 1489–1500 (2019). https://doi.org/10.1007/s00371-018-1513-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00371-018-1513-y