Skip to main content
SpringerLink
Log in

Heavy-Ball-Based Optimal Thresholding Algorithms for Sparse Linear Inverse Problems

  • Published:
Journal of Scientific Computing Aims and scope Submit manuscript

Abstract

Linear inverse problems arise in diverse engineering fields especially in signal and image reconstruction. The development of computational methods for linear inverse problems with sparsity is one of the recent trends in this field. The so-called optimal k-thresholding is a newly introduced method for sparse optimization and linear inverse problems. Compared to other sparsity-aware algorithms, the advantage of optimal k-thresholding method lies in that it performs thresholding and error metric reduction simultaneously and thus works stably and robustly for solving medium-sized linear inverse problems. However, the runtime of this method is generally high when the size of the problem is large. The purpose of this paper is to propose an acceleration strategy for this method. Specifically, we propose a heavy-ball-based optimal k-thresholding algorithm and its relaxed variants for sparse linear inverse problems. The convergence of these algorithms is shown under the restricted isometry property. In addition, the numerical performance of the heavy-ball-based relaxed optimal k-thresholding pursuit (HBROTP) has been evaluated, and simulations indicate that HBROTP admits robustness for signal and image reconstruction even in noisy environments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data availability

The data are available from the corresponding or the first author upon reasonable request.

References

  1. Alotaibi, M., Buccini, A., Reichel, L.: Krylov subspace split Bregman methods. Appl. Numer. Math. 184, 371–390 (2023)

    MathSciNet  MATH  Google Scholar 

  2. Andersen, E..D., Andersen, K..D.: The MOSEK Interior Point Optimizer for Linear Programming: An Implementation of the Homogeneous Algorithm. High Performance Optimization, pp. 197–232. Springer, Boston, MA (2000)

    MATH  Google Scholar 

  3. Aujol, J.F., Dossal, C., Rondepierre, A.: Convergence rates of the heavy-ball method under the Łojasiewicz property. Math. Prog. 198, 195–254 (2023)

    MATH  Google Scholar 

  4. Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imag. Sci. 2(1), 183–202 (2009)

    MathSciNet  MATH  Google Scholar 

  5. Blanchard, J.D., Tanner, J.: Performance comparisons of greedy algorithms in compressed sensing. Numer. Linear Algebra Appl. 22(2), 254–282 (2015)

    MathSciNet  MATH  Google Scholar 

  6. Blanchard, J.D., Tanner, J., Wei, K.: CGIHT: conjugate gradient iterative hard thresholding for compressed sensing and matrix completion. IMA J. Inf. Inference 4(4), 289–327 (2015)

    MathSciNet  MATH  Google Scholar 

  7. Blumensath, T.: Accelerated iterative hard thresholding. Signal Process. 92(3), 752–756 (2012)

    Google Scholar 

  8. Blumensath, T., Davies, M.E.: Iterative thresholding for sparse approximations. J. Fourier Anal. Appl. 14, 629–654 (2008)

    MathSciNet  MATH  Google Scholar 

  9. Blumensath, T., Davies, M.E.: Normalized iterative hard thresholding: Guaranteed stability and performance. IEEE J. Sel. Top. Signal Process. 4(2), 298–309 (2010)

    Google Scholar 

  10. Borgerding, M., Schniter, P., Rangan, S.: AMP-Inspired deep networks for sparse linear inverse problems. IEEE Trans. Signal Process. 65(16), 4293–4308 (2017)

    MathSciNet  MATH  Google Scholar 

  11. Buccini, A., Pasha, M., Reichel, L.: Linearized Krylov subspace Bregman iteration with nonnegativity constraint. Numer. Algo. 87, 1177–1200 (2021)

    MathSciNet  MATH  Google Scholar 

  12. Buchheim, C., Traversi, E.: Quadratic combinatorial optimization using separable underestimators. INFORMS J. Comput. 30(3), 424–437 (2018)

    MathSciNet  MATH  Google Scholar 

  13. Cai, Y., Donatelli, M., Bianchi, D., Huang, T.Z.: Regularization preconditioners for frame-based image deblurring with reduced boundary artifacts. SIAM J. Sci. Comput. 38(1), B164–B189 (2016)

    MathSciNet  MATH  Google Scholar 

  14. Candès, E.J., Tao, T.: Decoding by linear programming. IEEE Trans. Inform. Theory 51(12), 4203–4215 (2005)

    MathSciNet  MATH  Google Scholar 

  15. Candès, E.J., Wakin, M.B., Boyd, S.P.: Enhancing sparsity by reweighted \(\ell _1\)-minimization. J. Fourier Anal. Appl. 14, 877–905 (2008)

    MathSciNet  MATH  Google Scholar 

  16. Chaovalitwongse, W.A., Androulakis, I.P., Pardalos, P.M.: Quadratic integer programming: Complexity and equivalent forms. In: Floudas C.A., Pardalos P.M.: (eds) Encyclopedia of Optimization. Springer, Boston, MA (2008)

  17. Chartrand, R.: Exact reconstruction of sparse signals via nonconvex minimization. IEEE Signal Process. Lett. 14(10), 707–710 (2007)

    Google Scholar 

  18. Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20(1), 33–61 (1998)

    MathSciNet  MATH  Google Scholar 

  19. Chen, W., Zhang, B., Jin, S., Ai, B., Zhong, Z.: Solving sparse linear inverse problems in communication systems: A deep learning approach with adaptive depth. IEEE J. Sel. Areas Commun. 39(1), 4–17 (2021)

    Google Scholar 

  20. Dai, W., Milenkovic, O.: Subspace pursuit for compressive sensing signal reconstruction. IEEE Trans. Inform. Theory 55(5), 2230–2249 (2009)

    MathSciNet  MATH  Google Scholar 

  21. Daubechies, I., Defrise, M., De, Mol C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Comm. Pure Appl. Math. 57(11), 1413–1457 (2004)

    MathSciNet  MATH  Google Scholar 

  22. Donoho, D.L.: De-noising by soft-thresholding. IEEE Trans. Inform. Theory 41(3), 613–627 (1995)

    MathSciNet  MATH  Google Scholar 

  23. Donoho, D.L., Johnstone, I.M.: Ideal spatial adaptation by wavelet shrinkage. Biometrika 81(3), 425–455 (1994)

    MathSciNet  MATH  Google Scholar 

  24. Elad, M.: Why simple shrinkage is still relevant for redundant representations? IEEE Trans. Inform. Theory 52(12), 5559–5569 (2006)

    MathSciNet  MATH  Google Scholar 

  25. Elad, M.: Sparse and Redundant Representations: From Theory to Applications in Signal and Image Processing. Springer, New York (2010)

    MATH  Google Scholar 

  26. Eldar, Y.C., Kutyniok, G.: Compressed Sensing: Theory and Applications. Cambridge University Press, Cambridge (2012)

    Google Scholar 

  27. Foucart, S.: Hard thresholding pursuit: An algorithm for compressive sensing. SIAM J. Numer. Anal. 49(6), 2543–2563 (2011)

    MathSciNet  MATH  Google Scholar 

  28. Foucart, S., Rauhut, H.: A Mathematical Introduction to Compressive Sensing. Springer, New York (2013)

    MATH  Google Scholar 

  29. Golub, G.H., Van Loan, C.F.: Matrix Computations, 4th edn. Johns Hopkins University Press, Baltimore (2013)

    MATH  Google Scholar 

  30. Grant, M., Boyd, S.: CVX: matlab software for disciplined convex programming. Version 1.21 (2017)

  31. Gürbüzbalaban, M., Ozdaglar, A., Parrilo, P.A.: On the convergence rate of incremental aggregated gradient algorithms. SIAM J. Opt. 27(2), 1035–1048 (2017)

    MathSciNet  MATH  Google Scholar 

  32. Kuru, N., Birbil, Ş.İ., Gürbüzbalaban, M., Yildirim, S.: Differentially private accelerated optimization algorithms. SIAM J. Opt. 32(2), 795–821 (2022)

    MathSciNet  MATH  Google Scholar 

  33. Kyrillidis, A., Cevher, V.: Matrix recipes for hard thresholding methods. J. Math. Imag. Vis. 48, 235–265 (2014)

    MathSciNet  MATH  Google Scholar 

  34. Lessard, L., Recht, B., Packard, A.: Analysis and design of optimization algorithms via integral quadratic constraints. SIAM J. Opt. 26(1), 57–95 (2016)

    MathSciNet  MATH  Google Scholar 

  35. Li, S., Amin, M., Zhao, G., Sun, H.: Radar imaging by sparse optimization incorporating MRF clustering prior. IEEE Geosci. Remote Sens. Lett. 17(7), 1139–1143 (2020)

    Google Scholar 

  36. Li, H., Cheng, H., Wang, Z., Wu, G.C.: Distributed Nesterov gradient and heavy-ball double accelerated asynchronous optimization. IEEE Trans. Neural Netw. Learn. Syst. 32(12), 5723–5737 (2021)

    MathSciNet  Google Scholar 

  37. Liu, Y., Zhan, Z., Cai, J.F., Guo, D., Chen, Z., Qu, X.: Projected iterative soft-thresholding algorithm for tight frames in compressed sensing magnetic resonance imaging. IEEE Trans. Med. Imag. 35(9), 2130–2140 (2016)

    Google Scholar 

  38. Meng, N., Zhao, Y.B.: Newton-step-based hard thresholding algorithms for sparse signal recovery. IEEE Trans. Signal Process. 68, 6594–6606 (2020)

    MathSciNet  MATH  Google Scholar 

  39. Meng, N., Zhao, Y.B.: Newton-type optimal thresholding algorithms for sparse optimization problems. J. Oper. Res. Soc. China 10, 447–469 (2022)

    MathSciNet  MATH  Google Scholar 

  40. Meng, N., Zhao, Y.B., Kočvara, M., Sun, Z.F.: Partial gradient optimal thresholding algorithms for a class of sparse optimization problems. J. Global Opt. 84, 393–413 (2022)

    MathSciNet  MATH  Google Scholar 

  41. Mohammadi, H., Razaviyayn, M., Jovanović, M.R.: Robustness of accelerated first-order algorithms for strongly convex optimization problems. IEEE Trans. Auto. Control 66(6), 2480–2495 (2021)

    MathSciNet  MATH  Google Scholar 

  42. Needell, D., Tropp, J.A.: CoSaMP: Iterative signal recovery from incomplete and inaccurate samples. Appl. Comput. Harmon. Anal. 26(3), 301–321 (2009)

    MathSciNet  MATH  Google Scholar 

  43. Oymak, S., Recht, B., Soltanolkotabi, M.: Sharp time-data tradeoffs for linear inverse problems. IEEE Trans. Inform. Theory 64(6), 4129–4158 (2018)

    MathSciNet  MATH  Google Scholar 

  44. Polyak, B.T.: Some methods of speeding up the convergence of iteration methods. USSR Comput. Math. Math. Phys. 4(5), 1–17 (1964)

    Google Scholar 

  45. Schniter P., Potter L.C., Ziniel J.: Fast Bayesian matching pursuit. In: Proc. Inform. Theory Appl. Workshop 326–333 (2008)

  46. Sun, Z.F., Zhou, J.C., Zhao, Y.B., Meng, N.: Heavy-ball-based hard thresholding algorithms for sparse signal recovery. J. Comput. Appl. Math. 430, 115264 (2023)

    MathSciNet  Google Scholar 

  47. Tirer, T., Giryes, R.: Back-projection based fidelity term for ill-posed linear inverse problems. IEEE Trans. Image Process. 29, 6164–6179 (2020)

    MathSciNet  MATH  Google Scholar 

  48. Tropp, J.A., Gilbert, A.C.: Signal recovery from random measurements via orthogonal matching pursuit. IEEE Trans. Inform. Theory 53(12), 4655–4666 (2007)

    MathSciNet  MATH  Google Scholar 

  49. Tropp J.A., Wright S.J.: Computational methods for sparse solution of linear inverse problems. In: Proc. IEEE 98(6), 948–958 (2010)

  50. Ugrinovskii, V., Petersen, I.R., Shames, I.: Global convergence and asymptotic optimality of the heavy ball method for a class of nonconvex optimization problems. IEEE Control Syst. Lett. 6, 2449–2454 (2022)

    MathSciNet  Google Scholar 

  51. Wipf, D.P., Rao, B.D.: Sparse Bayesian learning for basis selection. IEEE Trans. Signal Process. 52(8), 2153–2164 (2004)

    MathSciNet  MATH  Google Scholar 

  52. Xiang, J., Dong, Y., Yang, Y.: FISTA-net: Learning a fast iterative shrinkage thresholding network for inverse problems in imaging. IEEE Trans. Med. Imag. 40(5), 1329–1339 (2021)

    Google Scholar 

  53. Xin, R., Khan, U.A.: Distributed heavy-ball: A generalization and acceleration of first-order methods with gradient tracking. IEEE Trans. Auto. Control 65(6), 2627–2633 (2020)

    MathSciNet  MATH  Google Scholar 

  54. Yin, W., Osher, S., Goldfarb, D., Darbon, J.: Bregman iterative algorithms for \(\ell _1\)-minimization with applications to compressed sensing. SIAM J. Imag. Sci. 1(1), 143–168 (2008)

    MATH  Google Scholar 

  55. Yin, W.: Analysis and generalizations of the linearized Bregman method. SIAM J. Imag. Sci. 3(4), 856–877 (2010)

    MATH  Google Scholar 

  56. Zhao, Y.B.: Optimal \(k\)-thresholding algorithms for sparse optimization problems. SIAM J. Opt. 30(1), 31–55 (2020)

    MathSciNet  MATH  Google Scholar 

  57. Zhao, Y.B., Luo, Z.Q.: Constructing new reweighted \(\ell _1\)-algorithms for the sparsest points of polyhedral sets. Math. Oper. Res. 42(1), 57–76 (2017)

    MathSciNet  Google Scholar 

  58. Zhao, Y.B., Luo, Z.Q.: Analysis of optimal thresholding algorithms for compressed sensing. Signal Process. 187, 108148 (2021)

    Google Scholar 

  59. Zhao, Y.B., Luo, Z.Q.: Natural thresholding algorithms for signal recovery with sparsity. IEEE Open J. Signal Process. 3, 417–431 (2022)

    Google Scholar 

  60. Zhao, Y.B., Luo, Z.Q.: Improved RIP-based bounds for guaranteed performance of two compressed sensing algorithms. Sci. China Math. 66(5), 1123–1140 (2023)

    MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The authors thank two anonymous reviewers for their helpful comments and suggestions that help improve the paper.

Author information

Authors and Affiliations

  1. School of Mathematics and Statistics, Shandong University of Technology, Zibo, Shandong, China

    Zhong-Feng Sun & Jin-Chuan Zhou

  2. Shenzhen Research Institute of Big Data, Chinese University of Hong Kong, Shenzhen, Guangdong, China

    Yun-Bin Zhao

Authors
  1. Zhong-Feng Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jin-Chuan Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yun-Bin Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the conception and analysis in this work. Specifically, ZFS contributed in methodology, analysis, the first draft writing, coding and numerical experiments; YBZ contributed in conceptualization, methodology, analysis, funding acquisition, resources, supervision, and editing; JCZ contributed to analysis, methodology and reviewing. The first draft of the manuscript was written by ZFS supported by JCZ. All authors read, discussed and approved the final manuscript.

Corresponding author

Correspondence to Yun-Bin Zhao.

Ethics declarations

Conflict of interest

The authors have no competing interests to declare that are relevant to the content of this article.

Funding

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The work was founded by the National Natural Science Foundation of China (NSFC 12071307 and 11771255), Young Innovation Teams of Shandong Province (2019KJI013), and Shandong Province Natural Science Foundation (ZR2021MA066, ZR2023MA020).

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sun, ZF., Zhou, JC. & Zhao, YB. Heavy-Ball-Based Optimal Thresholding Algorithms for Sparse Linear Inverse Problems. J Sci Comput 96, 93 (2023). https://doi.org/10.1007/s10915-023-02315-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10915-023-02315-1

Keywords

Mathematics Subject Classification

Search

Navigation