Abstract
We propose a discrete variational approach for image smoothing consisting of nonlocal data and smoothness constraints that penalise general dissimilarity measures defined on image patches. One of such dissimilarity measures is the weighted L 2 distance between patches. In such a case we derive an iterative neighbourhood filter that induces a new similarity measure in the photometric domain. It can be regarded as an extended patch similarity measure that evaluates not only the patch similarity of two chosen pixels, but also the similarity of their corresponding neighbours. This leads to a more robust smoothing process since the pixels selected for averaging are more coherent with the local image structure. By slightly modifying the way the similarities are computed we obtain two related filters: The NL-means filter of Buades et al. (SIAM Multiscale Model. Simul. 4(2):490–530, 2005b) and the NDS filter of Mrázek et al. (Geometric Properties for Incomplete Data, Computational Imaging and Vision, vol. 31, pp. 335–352, Springer, Dordrecht, 2006). In fact, the proposed approach can be considered as a generalisation of the latter filter to the space of patches. We also provide novel insights into relations of the NDS filter with diffusion/regularisation methods as well as with some recently proposed graph regularisation techniques. We evaluate our method for the task of denoising greyscale and colour images degraded with Gaussian and salt-and-pepper noise, demonstrating that it compares very well to other more sophisticated approaches.
Similar content being viewed by others
References
Aubert, G., & Kornprobst, P. (2006). Mathematical problems in image processing: partial differential equations and the calculus of variations (2nd ed.). Berlin: Springer.
Aurich, V., & Weule, J. (1995). Non-linear Gaussian filters performing edge preservating diffusion. In G. Sagerer, S. Posch, & F. Kummert (Eds.), Informatik Aktuell, Mustererkennung (pp. 538–545). Berlin: Springer.
Awate, S. P., & Whitaker, R. T. (2005). Higher-order image statistics for unsupervised, information-theoretic, adaptive, image filtering. In Proc. of the 2005 IEEE computer society conference on computer vision and pattern recognition (Vol. 2, pp. 44–51). Los Alamitos: IEEE Comput. Soc.
Awate, S. P., & Whitaker, R.T. (2006). Unsupervised, information-theoretic, adaptive image filtering for image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(3), 364–376.
Azzabou, N., Paragios, N., Guichard, F., & Cao, F. (2007). Variable bandwidth image denoising using image-based noise models. In Proc. of the 2007 IEEE computer society conference on computer vision and pattern recognition (pp. 1–7). Los Alamitos: IEEE Comput. Soc.
Bar, L., Kiryati, N., & Sochen, N. (2006). Image deblurring in the presence of impulsive noise. International Journal of Computer Vision, 70(3), 279–298.
Barash, D., & Comaniciu, D. (2004). A common framework for nonlinear diffusion, adaptive smoothing, bilateral filtering and mean shift. Image and Video Computing, 22(1), 73–81.
Bertero, M., Poggio, T. A., & Torre, V. (1988). Ill-posed problems in early vision. Proceedings of the IEEE, 76, 869–889.
Bilcu, R. C., & Vehvilainen, M. (2007). Fast nonlocal means for image denoising. In R. A. Martin, J. M. DiCarlo, & N. Sampat (Eds.), Proc. of SPIE : Vol. 6502. Digital photography III (p. 65020R). Bellingham: SPIE.
Bougleux, S., Elmoataz, A., & Melkemi, M. (2007). Discrete regularization on weighted graphs for image and mesh filtering. In F. Sgallari, A. Murli, & N. Paragios (Eds.), Lecture notes in computer science : Vol. 4485. Scale space and variational methods in computer vision (pp. 128–139). Berlin: Springer.
Bougleux, S., Elmoataz, A., & Melkemi, M. (2009). Local and nonlocal discrete regularization on weighted graphs for image and mesh processing. International Journal of Computer Vision, 84(2), 220–236.
Brouwer, L. E. J. (1911). Über Abbildungen von Mannigfaltigkeiten. Mathematische Annalen, 71(1), 97–115.
Brox, T., & Cremers, D. (2007). Iterated nonlocal means for texture restoration. In F. Sgallari, A. Murli, & N. Paragios (Eds.), Lecture notes in computer science : Vol. 4485. Scale space and variational methods in computer vision (pp. 13–24). Berlin: Springer.
Brox, T., Kleinschmidt, O., & Cremers, D. (2008). Efficient nonlocal means for denoising of textural patterns. IEEE Transactions on Image Processing, 17(7), 1083–1092.
Buades, A., Coll, B., & Morel, J.-M. (2005a). A non-local algorithm for image denoising. In Proc. of the 2005 IEEE computer society conference on computer vision and pattern recognition (Vol. 2, pp. 60–65). Los Alamitos: IEEE Comput. Soc.
Buades, A., Coll, B., & Morel, J.-M. (2005b). A review of image denoising algorithms, with a new one. SIAM Multiscale Modeling and Simulation, 4(2), 490–530.
Buades, A., Coll, B., & Morel, J.-M. (2006). Neighborhood filters and PDEs. Numerische Mathematik, 105(1), 1–34.
Cai, J.-F., Chan, R. H., & Nikolova, M. (2008). Two-phase approach for deblurring images corrupted by impulse plus Gaussian noise. Inverse Problems and Imaging, 2(2), 187–204.
Chan, R. H., Ho, C. W., & Nikolova, M. (2005). Salt-and-pepper noise removal by median-type noise detectors and detail-preserving regularization. IEEE Transactions on Image Processing, 14(10), 1479–1485.
Chan, T. F., & Shen, J. (2005). Image processing and analysis: variational, PDE, wavelet, and stochastic methods. Philadelphia: SIAM.
Chan, T. F., Osher, S., & Shen, J. (2001). The digital TV filter and nonlinear denoising. IEEE Transactions on Image Processing, 10(2), 231–241.
Charbonnier, P., Blanc-Féraud, L., Aubert, G., & Barlaud, M. (1997). Deterministic edge-preserving regularization in computed imaging. IEEE Transactions on Image Processing, 6(2), 298–311.
Chatterjee, P., & Milanfar, P. (2008). A generalization of non-local means via kernel regression. In C. A. Bouman, E. L. Miller, & I. Pollak (Eds.), Proc. of SPIE : Vol. 6814. Computational imaging VI (p. 68140P). Bellingham: SPIE.
Chen, Y., Ma, J., Feng, Q., Luo, L., Shi, P., & Chen, W. (2008). Nonlocal prior Bayesian tomographic reconstruction. Journal of Mathematical Imaging and Vision, 30(2), 133–146.
Cheng, Y. (1995). Mean shift, mode seeking, and clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(8), 790–799.
Chu, C. K., Glad, I. K., Godtliebsen, F., & Marron, J. S. (1998). Edge-preserving smoothers for image processing. Journal of the American Statistical Association, 93(442), 526–541.
Comaniciu, D., & Meer, P. (2002). Mean shift: A robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(5), 603–619.
Coupé, P., Hellier, P., Kervrann, C., & Barillot, C. (2008a). Bayesian non local means-based speckle filtering. In Proc. of the 2008 IEEE international symposium on biomedical imaging: from nano to macro (pp. 1291–1294).
Coupé, P., Yger, P., Prima, S., Hellier, P., Kervrann, C., & Barillot, C. (2008b). An optimized blockwise nonlocal means denoising filter for 3-D magnetic resonance images. IEEE Transactions on Medical Imaging, 27(4), 425–441.
Dabov, K., Foi, A., Katkovnik, V., & Egiazarian, K. (2007). Image denoising by sparse 3D transform-domain collaborative filtering. IEEE Transactions on Image Processing, 16(8), 2080–2095.
Darbon, J., Cunha, A., Chan, T. F., Osher, S., & Jensen, G. J. (2008). Fast nonlocal filtering applied to electron cryomicroscopy. In Proc. of the 2008 IEEE international symposium on biomedical imaging: from nano to macro (pp. 1331–1334).
Didas, S., & Weickert, J. (2006). From adaptive averaging to accelerated nonlinear diffusion filtering. In K. Franke, K.-R. Müller, B. Nicolay, & R. Schäfer (Eds.), Lecture notes in computer science : Vol. 4174. Pattern recognition (pp. 101–110). Berlin: Springer.
Didas, S., & Weickert, J. (2007). Integrodifferential equations for continuous multiscale wavelet shrinkage. Inverse Problems and Imaging, 1(1), 47–62.
Didas, S., Mrázek, P., & Weickert, J. (2007). Energy-based image simplification with nonlocal data and smoothness terms. In A. Iske, & J. Levesley (Eds.), Algorithms for approximation (pp. 51–60). Berlin: Springer.
Elad, M. (2002). On the origin of the bilateral filter and ways to improve it. IEEE Transactions on Image Processing, 11(10), 1141–1151.
Elmoataz, A., Lézoray, O., & Bougleux, S. (2008a). Nonlocal discrete regularization on weighted graphs: A framework for image and manifold processing. IEEE Transactions on Image Processing, 17(7), 1047–1060.
Elmoataz, A., Lézoray, O., Bougleux, S., & Ta, V.-T. (2008b). Unifying local and nonlocal processing with partial difference operators on weighted graphs. In A. Foi & A. Gotchev (Eds.), Proc. International workshop on local and non-local approximation in image processing (pp. 11–26).
Geman, S., & Geman, D. (1984). Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6, 721–741.
Gilboa, G., & Osher, S. (2007). Nonlocal linear image regularization and supervised segmentation. SIAM Multiscale Modeling and Simulation, 6(2), 595–630.
Gilboa, G., & Osher, S. (2008). Nonlocal operators with applications to image processing. SIAM Multiscale Modeling and Simulation, 7(3), 1005–1028.
Gilboa, G., Darbon, J., Osher, S., & Chan, T. F. (2006). Nonlocal convex functionals for image regularization. Technical Report CAM-06-57, Department of Mathematics, University of California at Los Angeles, CA, USA.
Goossens, B., Luong, H., Pizurica, A., & Philips, W. (2008). An improved non-local denoising algorithm. In A. Foi & A. Gotchev (Eds.), Proc. international workshop on local and non-local approximation in image processing (pp. 143–156).
Hampel, F. R., Ronchetti, E. M., Rousseeuw, P. J., & Stahel, W. A. (1986). Robust statistics. Probability and mathematical statistics. New York: Wiley.
Hein, M., Audibert, J.-Y., & von Luxburg, U. (2007). Graph Laplacians and their convergence on random neighborhood graphs. Journal of Machine Learning Research, 8, 1325–1370.
Holland, P. W., & Welsch, R. E. (1977). Robust regression using iteratively reweighted least-squares. Communications in Statistics, 6, 813–827.
Huber, P. J. (1981). Robust statistics. New York: Wiley.
Jung, M., & Vese, L. (2009). Image restoration via nonlocal Mumford-Shah regularizers. Technical Report CAM-09-09, Department of Mathematics, University of California at Los Angeles, CA, USA.
Kervrann, C., & Boulanger, J. (2006). Optimal spatial adaptation for patch-based image denoising. IEEE Transactions on Image Processing, 15(10), 2866–2878.
Kervrann, C., & Boulanger, J. (2008). Local adaptivity to variable smoothness for exemplar-based image regularization and representation. International Journal of Computer Vision, 79(1), 45–69.
Kervrann, C., Boulanger, J., & Coupé, P. (2007). Bayesian non-local means filter, image redundancy and adaptive dictionaries for noise removal. In F. Sgallari, A. Murli, & N. Paragios (Eds.), Lecture notes in computer science : Vol. 4485. Scale space and variational methods in computer vision (pp. 520–532). Berlin: Springer.
Kindermann, S., Osher, S., & Jones, P. W. (2005). Deblurring and denoising of images by nonlocal functionals. SIAM Multiscale Modeling and Simulation, 4(4), 1091–1115.
Kleinschmidt, O., Brox, T., & Cremers, D. (2008). Nonlocal texture filtering with efficient tree structures and invariant patch similarity measures. In A. Foi & A. Gotchev (Eds.), Proc. international workshop on local and non-local approximation in image processing (pp. 103–113).
Koenderink, J. J., & Van Doorn, A. L. (1999). The structure of locally orderless images. International Journal of Computer Vision, 31(2/3), 159–168.
Leclerc, Y. G. (1989). Constructing simple stable descriptions for image partitioning. International Journal of Computer Vision, 3(1), 73–102.
Lee, J.-S. (1983). Digital image smoothing and the sigma filter. Computer Vision, Graphics, and Image Processing, 24, 255–269.
Liu, J., Huan, Z., Huang, H., & Zhang, H. (2009). An adaptive method for recovering image from mixed noisy data. International Journal of Computer Vision, 85(2), 182–191.
Liu, Y.-L., Wang, J., Chen, X., & Guo, Y.-W. (2008). A robust and fast non-local means algorithm for image denoising. Journal of Computer Science and Technology, 23(2), 270–279.
Lou, Y., Favaro, P., Soatto, S., & Bertozzi, A. (2009). Nonlocal similarity image filtering. In P. Foggia, C. Sansone, & M. Vento (Eds.), Lecture notes in computer science : Vol. 5716. Image analysis and processing—ICIAP 2009 (pp. 62–71). Berlin: Springer.
Loupas, T., McDicken, W. N., & Allan, P. L. (1989). An adaptive weighted median filter for speckle suppression in medical ultrasonic images. IEEE Transactions on Circuits and Systems, 36(1), 129–135.
Mahmoudi, M., & Sapiro, G. (2005). Fast image and video denoising via nonlocal means of similar neighborhoods. IEEE Signal Processing Letters, 12(12), 839–842.
Mrázek, P., & Navara, M. (2003). Selection of optimal stopping time for nonlinear diffusion filtering. International Journal of Computer Vision, 52(2/3), 189–203.
Mrázek, P., Weickert, J., & Bruhn, A. (2006). On robust estimation and smoothing with spatial and tonal kernels. In R. Klette, R. Kozera, L. Noakes, & J. Weickert (Eds.), Computational imaging and vision : Vol. 31. Geometric properties for incomplete data (pp. 335–352). Dordrecht: Springer.
Mumford, D. (1994). The Bayesian rationale for energy functionals. In B. M. ter Haar Romeny (Ed.), Computational imaging and vision : Vol. 1. Geometry-driven diffusion in computer vision (pp. 141–153). Dordrecht: Kluwer.
Nikolova, M. (2005). Analysis of the recovery of edges in images and signals by minimizing nonconvex regularized least-squares. SIAM Multiscale Modeling and Simulation, 4(3), 960–991.
Nordström, N. (1990). Biased anisotropic diffusion—a unified regularization and diffusion approach to edge detection. Image and Vision Computing, 8, 318–327.
Orchard, J., Ebrahimi, M., & Wong, A. (2008). Efficient non-local-means denoising using the SVD. In Proc. of the 15th IEEE international conference on image processing (pp. 1732–1735). Los Alamitos: IEEE Comput. Soc.
Perona, P., & Malik, J. (1990). Scale space and edge detection using anisotropic diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12, 629–639.
Peter, J. D., Govindan, V. K., & Mathew, A. T. (2008). Robust estimation approach for NL-means filter. In G. Bebis, R. Boyle, B. Parvin, D. Koracin, P. Remagnino, F. Porikli, J. Peters, J. Klosowski, L. Arns, K. C. Yu, T.-M. Rhyne, & L. Monroe (Eds.), Lecture notes in computer science : Vol. 5359. Advances in visual computing (pp. 571–580). Berlin: Springer.
Peyré, G., Bougleux, S., & Cohen, L. (2008). Non-local regularization of inverse problems. In D. Forsyth, P. Torr, & A. Zisserman (Eds.), Lecture notes in computer science : Vol. 5304. Computer vision—ECCV 2008, part III (pp. 57–68). Berlin: Springer.
Pizarro, L., Didas, S., Bauer, F., & Weickert, J. (2007). Evaluating a general class of filters for image denoising. In B. K. Ersbøll & K. S. Pedersen (Eds.), Lecture notes in computer science : Vol. 4522. Image analysis (pp. 601–610). Berlin: Springer.
Portilla, J., Strela, V., Wainwright, M. J., & Simoncelli, E. P. (2003). Image denoising using Gaussian scale mixtures in the wavelet domain. IEEE Transactions on Image Processing, 12(11), 1338–1351.
Rudin, L. I., Osher, S., & Fatemi, E. (1992). Nonlinear total variation based noise removal algorithms. Physica D, 60, 259–268.
Saint-Marc, P., Chen, J.-S., & Medioni, G. (1991). Adaptive smoothing: A general tool for early vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(6), 514–529.
Scherzer, O., & Weickert, J. (2000). Relations between regularization and diffusion filtering. Journal of Mathematical Imaging and Vision, 12, 43–63.
Schnörr, C. (1994). Unique reconstruction of piecewise smooth images by minimizing strictly convex non-quadratic functionals. Journal of Mathematical Imaging and Vision, 4, 189–198.
Smith, S. M., & Brady, J. M. (1997). SUSAN—A new approach to low level image processing. International Journal of Computer Vision, 23(1), 43–78.
Steidl, G., Weickert, J., Brox, T., Mrázek, P., & Welk, M. (2004). On the equivalence of soft wavelet shrinkage, total variation diffusion, total variation regularization, and SIDEs. SIAM Journal on Numerical Analysis, 42(2), 686–713.
Stevenson, R. L., Schmitz, B. E., & Delp, E. J. (1994). Discontinuity preserving regularization of inverse visual problems. IEEE Transactions on Systems, Man and Cybernetics, 24, 455–469.
Ta, V.-T., Lézoray, O., Elmoataz, A., & Schüpp, S. (2009). Graph-based tools for microscopic cellular image segmentation. Pattern Recognition, 42(6), 1113–1125.
Tikhonov, A. N., & Arsenin, V. Y. (1977). Solutions of ill-posed problems. W.H. Washington: Winston.
Tomasi, C., & Manduchi, R. (1998). Bilateral filtering for gray and colour images. In Proc. of the 1998 IEEE international conference on computer vision (pp. 839–846). Bombay, India, January 1998. New Delhi: Narosa Publishing House.
van de Weijer, J., & van den Boomgaard, R. (2001). Local mode filtering. In Proc. 2001 IEEE computer society conference on computer vision and pattern recognition (Vol. 2, pp. 428–433). Kauai, HI, December 2001. Los Alamitos: IEEE Comput. Soc.
van den Boomgaard, R., & van de Weijer, J. (2002). On the equivalence of local-mode finding, robust estimation and mean-shift analysis as used in early vision tasks. In Proc. 16th international conference on pattern recognition (Vol. 3, pp. 927–930). Quebec City, Canada, August 2002.
van Ginneken, B., & ter Haar Romeny, B. M. (2000). Applications of locally orderless images. Journal of Visual Communication and Image Representation, 11, 196–208.
Vrscay, E. R. (2008). A simple model for affine self-similarity of images and its applications. In A. Foi & A. Gotchev (Eds.), Proc. international workshop on local and non-local approximation in image processing (pp. 61–76).
Weickert, J. (1994). Anisotropic diffusion filters for image processing based quality control. In A. Fasano, & M. Primicerio (Eds.), Proc. seventh European conf. on mathematics in industry (pp. 355–362). Stuttgart: Teubner.
Weickert, J. (1997). A review of nonlinear diffusion filtering. In B. ter Haar Romeny, L. Florack, J. Koenderink, & M. Viergever (Eds.), Lecture notes in computer science : Vol. 1252. Scale-space theory in computer vision (pp. 3–28). Berlin: Springer.
Weickert, J. (1998). Anisotropic diffusion in image processing. Stuttgart: Teubner.
Weickert, J., & Schnörr, C. (2001). A theoretical framework for convex regularizers in PDE-based computation of image motion. International Journal of Computer Vision, 45(3), 245–264.
Winkler, G. (2003). Image analysis, random fields and Markov chain Monte Carlo methods (2nd ed.). Berlin: Springer.
Winkler, G., Aurich, V., Hahn, K., & Martin, A. (1999). Noise reduction in images: Some recent edge-preserving methods. Pattern Recognition and Image Analysis, 9(4), 749–766.
Yaroslavsky, L. P. (1985). Digital picture processing. New York: Springer.
Zeidler, E. (1986). Nonlinear functional analysis and applications I: fixed-point theorems. New York: Springer.
Zhou, D., & Burges, C. (2008). High-order regularization on graphs. In International workshop on mining and learning with graphs.
Zhou, D., & Schölkopf, B. (2004). A regularization framework for learning from graph data. In Proc. ICML workshop on statistical relational learning and its connections to other fields (pp. 132–137).
Zhou, D., & Schölkopf, B. (2005). Regularization on discrete spaces. In W. G. Kropatsch, R. Sablatnig, & A. Hanbury (Eds.), Lecture notes in computer science : Vol. 3663. Pattern recognition (pp. 361–368). Berlin: Springer.
Zimmer, S., Didas, S., & Weickert, J. (2008). A rotationally invariant block matching strategy improving image denoising with non-local means. In A. Foi & A. Gotchev (Eds.), Proc. international workshop on local and non-local approximation in image processing (pp. 135–142).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Pizarro, L., Mrázek, P., Didas, S. et al. Generalised Nonlocal Image Smoothing. Int J Comput Vis 90, 62–87 (2010). https://doi.org/10.1007/s11263-010-0337-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-010-0337-7