Abstract
The Levenberg–Marquardt algorithm is one of the most popular algorithms for finding the solution of nonlinear least squares problems. Across different modified variations of the basic procedure, the algorithm enjoys global convergence, a competitive worst-case iteration complexity rate, and a guaranteed rate of local convergence for both zero and nonzero small residual problems, under suitable assumptions. We introduce a novel Levenberg-Marquardt method that matches, simultaneously, the state of the art in all of these convergence properties with a single seamless algorithm. Numerical experiments confirm the theoretical behavior of our proposed algorithm.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Tarantola, A.: Inverse Problem Theory and Methods for Model Parameter Estimation. SIAM, Philadelphia (2005)
Trémolet, Y.: Model error estimation in 4D-Var. Q. J. R. Meteorol. Soc. 133, 1267–1280 (2007)
Levenberg, K.: A method for the solution of certain problems in least squares. Quart. Appl. Math. 2, 164–168 (1944)
Marquardt, D.: An algorithm for least-squares estimation of nonlinear parameters. SIAM J. Appl. Math. 11, 431–441 (1963)
Osborne, M.R.: Nonlinear least squares—the Levenberg algorithm revisited. J. Austral. Math. Soc. Ser. B 19, 343–357 (1976)
Yamashita, N., Fukushima, M.: On the rate of convergence of the Levenberg–Marquardt method. In: Topics in Numerical Analysis, pp. 239–249. Springer (2001)
Fan, J., Yuan, Y.: On the quadratic convergence of the Levenberg–Marquardt method without nonsingularity assumption. Computing 74, 23–39 (2005)
Dan, H., Yamashita, N., Fukushima, M.: Convergence properties of the inexact Levenberg–Marquardt method under local error bound conditions. Optim. Methods Softw. 17, 605–626 (2002)
Facchinei, F., Fischer, A., Herrich, M.: A family of newton methods for nonsmooth constrained systems with nonisolated solutions. Math. Methods Oper. Res. 77, 433–443 (2013)
Ipsen, I.C.F., Kelley, C.T., Pope, S.R.: Rank-deficient nonlinear least squares problems and subset selection. SIAM J. Numer. Anal. 49, 1244–1266 (2011)
Fan, J.: Convergence rate of the trust region method for nonlinear equations under local error bound condition. Comput. Optim. Appl. 34, 215–227 (2006)
Conn, A.R., Gould, N.I.M., Toint, PhL: Trust-Region Methods. SIAM, Philadelphia, PA, USA (2000)
Ueda, K., Yamashita, N.: On a global complexity bound of the Levenberg–Marquardt method. J. Optim. Theory Appl. 147, 443–453 (2010)
Ueda, K., Yamashita, N.: Global complexity bound analysis of the Levenberg–Marquardt method for nonsmooth equations and its application to the nonlinear complementarity problem. J. Optim. Theory Appl. 152, 450–467 (2012)
Zhao, R., Fan, J.: Global complexity bound of the Levenberg–Marquardt method. Optim. Methods Softw. 31, 805–814 (2016)
Nocedal, J., Wright, S.J.: Numerical Optimization, 2nd edn. Springer, Berlin (2006)
Bergou, E., Gratton, S., Vicente, L.N.: Levenberg–Marquardt methods based on probabilistic gradient models and inexact subproblem solution, with application to data assimilation. SIAM/ASA J. Uncertain. Quantif. 4, 924–951 (2016)
Bonnans, J.F., Shapiro, A.: Perturbation Analysis of Optimization Problems. Springer, Berlin (2013)
Fischer, A., Shukla, P., Wang, M.: On the inexactness level of robust Levenberg–Marquardt methods. Optimization 59, 273–287 (2010)
Moré, J.J., Garbow, B.S., Hillstrom, K.E.: Testing unconstrained optimization software. ACM Trans. Math. Softw. 7, 17–41 (1981)
Dolan, E.D., Moré, J.J.: Benchmarking optimization software with performance profiles. Math. Program. 91(2), 201–213 (2002)
Acknowledgements
We would like to thank Clément Royer and the referees for their careful readings and corrections that helped us to improve our manuscript significantly. Support for Vyacheslav Kungurtsev was provided by the OP VVV Project CZ.02.1.01/0.0/0.0/16_019/0000765 “Research Center for Informatics”.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by Nobuo Yamashita.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Bergou, E.H., Diouane, Y. & Kungurtsev, V. Convergence and Complexity Analysis of a Levenberg–Marquardt Algorithm for Inverse Problems. J Optim Theory Appl 185, 927–944 (2020). https://doi.org/10.1007/s10957-020-01666-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10957-020-01666-1