Skip to main content
SpringerLink
Log in

Randomized Kaczmarz method with adaptive stepsizes for inconsistent linear systems

  • Original Paper
  • Published:
Numerical Algorithms Aims and scope Submit manuscript

Abstract

We investigate the randomized Kaczmarz method that adaptively updates the stepsize using readily available information for solving inconsistent linear systems. A novel geometric interpretation is provided which shows that the proposed method can be viewed as an orthogonal projection method in some sense. We prove that this method converges linearly in expectation to the unique minimum Euclidean norm least-squares solution of the linear system, and provide a tight upper bound for the convergence of the proposed method. Numerical experiments are also given to illustrate the theoretical results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Availability of supporting data

Data sharing not applicable to this article as no datasets were generated or analyzed during the current study. The authors confirm that all data generated or analyzed in the development of this work are adequately included or referenced in the article itself.

References

  1. Byrne, C.: A unified treatment of some iterative algorithms in signal processing and image reconstruction. Inverse Problems 20(1), 103–120 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  2. Patrascu, A., Necoara, I.: Nonasymptotic convergence of stochastic proximal point methods for constrained convex optimization. J. Mach. Learn. Res. 18(1), 7204–7245 (2017)

    MathSciNet  MATH  Google Scholar 

  3. Chang, K.-W., Hsieh, C.-J., Lin, C.-J.: Coordinate descent method for large-scale l2-loss linear support vector machines. J. Mach. Learn. Res. 9(7), 1369–1398 (2008)

    MathSciNet  MATH  Google Scholar 

  4. Olshanskii, E.E., Tyrtyshnikov, M.A.: Iterative Methods for Linear Systems: Theory and Applications. SIAM, Philadelphia, PA (2014)

    Book  MATH  Google Scholar 

  5. Karczmarz, S.: Angenäherte auflösung von systemen linearer glei-chungen. Bull. Int. Acad. Pol. Sic. Let., Cl. Sci. Math. Nat. 355–357 (1937)

  6. Herman, G.T., Meyer, L.B.: Algebraic reconstruction techniques can be made computationally efficient (positron emission tomography application). IEEE Trans. Medical Imaging 12(3), 600–609 (1993)

    Article  Google Scholar 

  7. Gordon, R., Bender, R., Herman, G.T.: Algebraic reconstruction techniques (ART) for three-dimensional electron microscopy and X-ray photography. J. Theor. Biol. 29(3), 471–481 (1970)

    Article  Google Scholar 

  8. Natterer, F.: The Mathematics of Computerized Tomography. SIAM (2001)

  9. Feichtinger, H.G., Cenker, C., Mayer, M., Steier, H., Strohmer, T.: New variants of the POCS method using affine subspaces of finite codimension with applications to irregular sampling. In: Visual Communications and Image Processing’92, vol. 1818, pp. 299–310 (1992). SPIE

  10. Strohmer, T., Vershynin, R.: A randomized Kaczmarz algorithm with exponential convergence. J. Fourier Anal. Appl. 15(2), 262–278 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  11. Liu, J., Wright, S.: An accelerated randomized Kaczmarz algorithm. Math. Comp. 85(297), 153–178 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  12. Han, D., Xie, J.: On pseudoinverse-free randomized methods for linear systems: Unified framework and acceleration. Preprint at http://arxiv.org/abs/2208.05437 (2022)

  13. Loizou, N., Richtárik, P.: Momentum and stochastic momentum for stochastic gradient, newton, proximal point and subspace descent methods. Comput. Optim. Appl. 77(3), 653–710 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  14. Necoara, I.: Faster randomized block Kaczmarz algorithms. SIAM J. Matrix Anal. Appl. 40(4), 1425–1452 (2019)

  15. Needell, D., Tropp, J.A.: Paved with good intentions: analysis of a randomized block kaczmarz method. Linear Algebra Appl. 441, 199–221 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  16. Moorman, J.D., Tu, T.K., Molitor, D., Needell, D.: Randomized Kaczmarz with averaging. BIT. 61(1), 337–359 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  17. Gower, R.M., Richtárik, P.: Randomized iterative methods for linear systems. SIAM J. Matrix Anal. Appl. 36(4), 1660–1690 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  18. Bai, Z.-Z., Wu, W.-T.: On greedy randomized Kaczmarz method for solving large sparse linear systems. SIAM J. Sci. Comput. 40(1), 592–606 (2018)

    Article  MathSciNet  Google Scholar 

  19. Gower, R.M., Molitor, D., Moorman, J., Needell, D.: On adaptive sketch-and-project for solving linear systems. SIAM J. Matrix Anal. Appl. 42(2), 954–989 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  20. Schöpfer, F., Lorenz, D.A.: Linear convergence of the randomized sparse Kaczmarz method. Math. Program. 173(1), 509–536 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  21. Chen, X., Qin, J.: Regularized Kaczmarz algorithms for tensor recovery. SIAM J. Imaging Sci. 14(4), 1439–1471 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  22. Needell, D.: Randomized Kaczmarz solver for noisy linear systems. BIT. 50(2), 395–403 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  23. Bai, Z.-Z., Wu, W.-T.: On greedy randomized augmented Kaczmarz method for solving large sparse inconsistent linear systems. SIAM J. Sci. Comput. 43(6), 3892–3911 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  24. Censor, Y., Eggermont, P.P., Gordon, D.: Strong underrelaxation in Kaczmarz’s method for inconsistent systems. Numer. Math. 41(1), 83–92 (1983)

    Article  MathSciNet  MATH  Google Scholar 

  25. Hanke, M., Niethammer, W.: On the acceleration of Kaczmarz’s method for inconsistent linear systems. Linear Algebra Appl. 130, 83–98 (1990)

    Article  MathSciNet  MATH  Google Scholar 

  26. Zouzias, A., Freris, N.M.: Randomized extended Kaczmarz for solving least squares. SIAM J. Matrix Anal. Appl. 34(2), 773–793 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  27. Du, K.: Tight upper bounds for the convergence of the randomized extended Kaczmarz and Gauss-Seidel algorithms. Numer. Linear Algebra Appl. 26(3), 2233 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  28. Wu, N.-C., Xiang, H.: Semiconvergence analysis of the randomized row iterative method and its extended variants. Numer. Linear Algebra Appl. 28(1), 2334 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  29. Needell, D., Ward, R.: Two-subspace projection method for coherent overdetermined systems. J. Fourier Anal. Appl. 19(2), 256–269 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  30. Du, K., Sun, X.-H.: Pseudoinverse-free randomized block iterative algorithms for consistent and inconsistent linear systems. Preprint at http://arxiv.org/abs/2011.10353 (2020)

  31. Du, K., Si, W.-T., Sun, X.-H.: Randomized extended average block Kaczmarz for solving least squares. SIAM J. Sci. Comput. 42(6), 3541–3559 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  32. Wu, W.-T.: On two-subspace randomized extended Kaczmarz method for solving large linear least-squares problems. Numer. Algorithms 89(1), 1–31 (2022)

    Article  MathSciNet  MATH  Google Scholar 

  33. Wu, N.-C., Liu, C., Wang, Y., Zuo, Q.: On the extended randomized multiple row method for solving linear least-squares problems. Preprint at http://arxiv.org/abs/2210.03478 (2022)

  34. Popa, C.: Extensions of block-projections methods with relaxation parameters to inconsistent and rank-deficient least-squares problems. BIT. 38(1), 151–176 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  35. Popa, C.: Characterization of the solutions set of inconsistent least-squares problems by an extended Kaczmarz algorithm. Korean J. Comput. Appl. Math. 6(1), 51–64 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  36. Bai, Z.-Z., Wu, W.-T.: On partially randomized extended Kaczmarz method for solving large sparse overdetermined inconsistent linear systems. Linear Algebra Appl. 578, 225–250 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  37. Ma, A., Needell, D., Ramdas, A.: Convergence properties of the randomized extended Gauss-Seidel and Kaczmarz methods. SIAM J. Matrix Anal. Appl. 36(4), 1590–1604 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  38. Leventhal, D., Lewis, A.S.: Randomized methods for linear constraints: convergence rates and conditioning. Math. Oper. Res. 35(3), 641–654 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  39. Bai, Z.-Z., Wang, L., Wu, W.-T.: On convergence rate of the randomized Gauss-Seidel method. Linear Algebra Appl. 611, 237–252 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  40. Golub, G.H., Van Loan, C.F.: Matrix Computations. JHU Press (2013)

  41. Ben-Israel, A., Greville, T.N.: Generalized Inverses: Theory and Applications vol. 15. Springer (2003)

  42. Needell, D., Srebro, N., Ward, R.: Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm. Math. Program. 155, 549–573 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  43. Steinerberger, S.: A weighted randomized Kaczmarz method for solving linear systems. Math. Comp. 90, 2815–2826 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  44. Deren, H., Su, Y., Xie, J.: Randomized Douglas-Rachford method for linear systems: Improved accuracy and efficiency. Preprint at http://arxiv.org/abs/2207.04291 (2022)

  45. Kolodziej, S.P., Aznaveh, M., Bullock, M., David, J., Davis, T.A., Henderson, M., Hu, Y., Sandstrom, R.: The suitesparse matrix collection website interface. J. Open Source Softw. 4(35), 1244 (2019)

    Article  Google Scholar 

  46. Hardt, M., Recht, B., Singer, Y.: Train faster, generalize better: Stability of stochastic gradient descent. In: Proceedings of the 33rd International Conference on Machine Learning, pp. 1225–1234. PMLR (2016)

  47. Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Statistics. 400–407 (1951)

  48. Ma, A., Needell, D.: Stochastic gradient descent for linear systems with missing data. Numer. Math. Theory Methods Appl. 12(1), 1–20 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  49. Nguyen, L.M., Liu, J., Scheinberg, K., Takáč, M.: SARAH: A novel method for machine learning problems using stochastic recursive gradient. In: Proceedings of the 34th International Conference on Machine Learning, pp. 2613–2621. PMLR (2017)

  50. Johnson, R., Zhang, T.: Accelerating stochastic gradient descent using predictive variance reduction. In: Proceedings of the Advances in Neural Information Processing Systems, pp. 315–323. (2013)

  51. Nemirovski, A., Juditsky, A., Lan, G., Shapiro, A.: Robust stochastic approximation approach to stochastic programming. SIAM J. Optim. 19(4), 1574–1609 (2009)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

We would like to thank the anonymous reviewers for their valuable comments, which have tremendously improved the content and quality of this paper.

Funding

Deren Han is supported by the NSFC grant (12131004, 11625105). Jiaxin Xie is supported by NSFC grant (12001026, 12071019,12126608) and the Fundamental Research Funds for the Central Universities.

Author information

Authors and Affiliations

  1. School of Mathematical Sciences, Beihang University, Beijing, 10191, China

    Yun Zeng & Yansheng Su

  2. LMIB of the Ministry of Education, School of Mathematical Sciences, Beihang University, Beijing, 10191, China

    Deren Han & Jiaxin Xie

Authors
  1. Yun Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Deren Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yansheng Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jiaxin Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed equally to the manuscript and read and approved the final manuscript.

Corresponding author

Correspondence to Jiaxin Xie.

Ethics declarations

Ethical approval and consent to participate

Not applicable.

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zeng, Y., Han, D., Su, Y. et al. Randomized Kaczmarz method with adaptive stepsizes for inconsistent linear systems. Numer Algor 94, 1403–1420 (2023). https://doi.org/10.1007/s11075-023-01540-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11075-023-01540-x

Keywords

Mathematics Subject Classification (2010)

Search

Navigation