Convergence of the linearized Bregman iteration for ℓ₁-norm minimization

Cai, Jian-Feng; Osher, Stanley; Shen, Zuowei

doi:10.1090/S0025-5718-09-02242-X

Convergence of the linearized Bregman iteration for $\ell _1$-norm minimization
HTML articles powered by AMS MathViewer

by Jian-Feng Cai, Stanley Osher and Zuowei Shen PDF

Math. Comp. 78 (2009), 2127-2136 Request permission

Abstract:

One of the key steps in compressed sensing is to solve the basis pursuit problem $\min _{u\in \mathbb {R}^n}\{\|u\|_1:Au=f\}$. Bregman iteration was very successfully used to solve this problem in [40]. Also, a simple and fast iterative algorithm based on linearized Bregman iteration was proposed in [40], which is described in detail with numerical simulations in [35]. A convergence analysis of the smoothed version of this algorithm was given in [11]. The purpose of this paper is to prove that the linearized Bregman iteration proposed in [40] for the basis pursuit problem indeed converges.

References

Dimitri P. Bertsekas, Constrained optimization and Lagrange multiplier methods, Computer Science and Applied Mathematics, Academic Press, Inc. [Harcourt Brace Jovanovich, Publishers], New York-London, 1982. MR 690767
D. P. Bertsekas, Nonlinear Programming, Athena Scientific, Belmont, Massachussets, 1999.
L. M. Brègman, A relaxation method of finding a common point of convex sets and its application to the solution of problems in convex programming, Ž. Vyčisl. Mat i Mat. Fiz. 7 (1967), 620–631 (Russian). MR 215617
M. Burger, E. Resmerita, and L. He, Error estimation for Bregman iterations and inverse scale space methods in image restoration, Computing 81 (2007), no. 2-3, 109–135. MR 2354192, DOI 10.1007/s00607-007-0245-z
J.-F. Cai, E.J. Candès, and Z. Shen, A singular value thresholding algorithm for matrix completion, 2008, preprint.
Jian-Feng Cai, Raymond Chan, Lixin Shen, and Zuowei Shen, Restoration of chopped and nodded images by framelets, SIAM J. Sci. Comput. 30 (2008), no. 3, 1205–1227. MR 2398862, DOI 10.1137/040615298
J.-F. Cai, R. H. Chan, L. Shen, and Z. Shen, Convergence analysis of tight framelet approach for missing data recovery, Adv. Comput. Math. (to appear), 2008.
—, Simultaneously inpainting in image and transformed domains, 2008, preprint.
Jian-Feng Cai, Raymond H. Chan, and Zuowei Shen, A framelet-based image inpainting algorithm, Appl. Comput. Harmon. Anal. 24 (2008), no. 2, 131–149. MR 2393979, DOI 10.1016/j.acha.2007.10.002
—, Simultaneous cartoon and texture inpainting, 2008, preprint.
J.-F. Cai, S. Osher, and Z. Shen, Linearized Bregman iterations for compressed sensing, 2008, Math. Comp., to appear; see also UCLA CAM Report (08-06).
—, Linearized Bregman iterations for frame-based image deblurring, 2008, preprint.
J.-F. Cai and Z. Shen, Deconvolution: A wavelet frame approach, II, 2008, preprint.
Emmanuel J. Candès, Compressive sampling, International Congress of Mathematicians. Vol. III, Eur. Math. Soc., Zürich, 2006, pp. 1433–1452. MR 2275736
Emmanuel J. Candès and Justin Romberg, Quantitative robust uncertainty principles and optimally sparse decompositions, Found. Comput. Math. 6 (2006), no. 2, 227–254. MR 2228740, DOI 10.1007/s10208-004-0162-x
Emmanuel J. Candès, Justin Romberg, and Terence Tao, Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information, IEEE Trans. Inform. Theory 52 (2006), no. 2, 489–509. MR 2236170, DOI 10.1109/TIT.2005.862083
Emmanuel J. Candes and Terence Tao, Near-optimal signal recovery from random projections: universal encoding strategies?, IEEE Trans. Inform. Theory 52 (2006), no. 12, 5406–5425. MR 2300700, DOI 10.1109/TIT.2006.885507
A. Enis Çetin, Reconstruction of signals from Fourier transform samples, Signal Process. 16 (1989), no. 2, 129–148 (English, with French and German summaries). MR 991553, DOI 10.1016/0165-1684(89)90092-3
—, An iterative algorithm for signal reconstruction from bispectrum, IEEE Trans. Signal Proc., 39 (1991), 2621–2628.
Anwei Chai and Zuowei Shen, Deconvolution: a wavelet frame approach, Numer. Math. 106 (2007), no. 4, 529–587. MR 2317925, DOI 10.1007/s00211-007-0075-0
Patrick L. Combettes and Jean-Christophe Pesquet, Proximal thresholding algorithm for minimization over orthonormal bases, SIAM J. Optim. 18 (2007), no. 4, 1351–1376. MR 2373305, DOI 10.1137/060669498
Patrick L. Combettes and Valérie R. Wajs, Signal recovery by proximal forward-backward splitting, Multiscale Model. Simul. 4 (2005), no. 4, 1168–1200. MR 2203849, DOI 10.1137/050626090
J. Darbon and S. Osher, Fast discrete optimization for sparse approximations and deconvolutions, 2007, preprint.
Ingrid Daubechies, Michel Defrise, and Christine De Mol, An iterative thresholding algorithm for linear inverse problems with a sparsity constraint, Comm. Pure Appl. Math. 57 (2004), no. 11, 1413–1457. MR 2077704, DOI 10.1002/cpa.20042
Ingrid Daubechies, Gerd Teschke, and Luminita Vese, Iteratively solving linear inverse problems under general convex constraints, Inverse Probl. Imaging 1 (2007), no. 1, 29–46. MR 2262744, DOI 10.3934/ipi.2007.1.29
David L. Donoho, De-noising by soft-thresholding, IEEE Trans. Inform. Theory 41 (1995), no. 3, 613–627. MR 1331258, DOI 10.1109/18.382009
David L. Donoho, Compressed sensing, IEEE Trans. Inform. Theory 52 (2006), no. 4, 1289–1306. MR 2241189, DOI 10.1109/TIT.2006.871582
Michel Fortin and Roland Glowinski, Augmented Lagrangian methods, Studies in Mathematics and its Applications, vol. 15, North-Holland Publishing Co., Amsterdam, 1983. Applications to the numerical solution of boundary value problems; Translated from the French by B. Hunt and D. C. Spicer. MR 724072
T. Goldstein and S. Osher, The split Bregman algorithm for l1 regularized problems, 2008, UCLA CAM Reports (08-29).
Magnus R. Hestenes, Multiplier and gradient methods, J. Optim. Theory Appl. 4 (1969), 303–320. MR 271809, DOI 10.1007/BF00927673
Jean-Baptiste Hiriart-Urruty and Claude Lemaréchal, Convex analysis and minimization algorithms. I, Grundlehren der mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences], vol. 305, Springer-Verlag, Berlin, 1993. Fundamentals. MR 1261420
Peter J. Huber, Robust regression: asymptotics, conjectures and Monte Carlo, Ann. Statist. 1 (1973), 799–821. MR 356373
K. Ito and K. Kunisch, The augmented Lagrangian method for equality and inequality constraints in Hilbert spaces, Math. Programming 46 (1990), no. 3, (Ser. A), 341–360. MR 1054143, DOI 10.1007/BF01585750
Stanley Osher, Martin Burger, Donald Goldfarb, Jinjun Xu, and Wotao Yin, An iterative regularization method for total variation-based image restoration, Multiscale Model. Simul. 4 (2005), no. 2, 460–489. MR 2162864, DOI 10.1137/040605412
S. Osher, Y. Mao, B. Dong, and W. Yin, Fast linearized bregman iteration for compressed sensing and sparse denoising, 2008, UCLA CAM Reports (08-37).
M. J. D. Powell, A method for nonlinear constraints in minimization problems, Optimization (Sympos., Univ. Keele, Keele, 1968) Academic Press, London, 1969, pp. 283–298. MR 0272403
F. Schöpfer, A. K. Louis, and T. Schuster, Nonlinear iterative methods for linear ill-posed problems in Banach spaces, Inverse Problems 22 (2006), no. 1, 311–329. MR 2194197, DOI 10.1088/0266-5611/22/1/017
Y. Tsaig and D. L. Donono, Extensions of compressed sensing, Signal Processing 86 (2005), 533–548.
W. Yin, On linearized Bregman iterative algorithms, private communication.
W. Yin, S. Osher, D. Goldfarb, and J. Darbon, Bregman iterative algorithms for $\ell _1$-minimization with applications to compressed sensing, SIAM J. Imaging Sci. 1 (2008), no. 1, 143–168.