Abstract
In this paper we consider large scale nonlinear least-squares problems for which function and gradient are evaluated with dynamic accuracy and propose a Levenberg–Marquardt method for solving such problems. More precisely, we consider the case in which the exact function to optimize is not available or its evaluation is computationally demanding, but approximations of it are available at any prescribed accuracy level. The proposed method relies on a control of the accuracy level, and imposes an improvement of function approximations when the accuracy is detected to be too low to proceed with the optimization process. We prove global and local convergence and complexity of our procedure and show encouraging numerical results on test problems arising in data assimilation and machine learning.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Anderson, T.W., Darling, D.A.: A test of goodness of fit. J. Am. Stat. Assoc. 49(268), 765–769 (1954)
Bandeira, A.S., Scheinberg, K., Vicente, L.N.: Convergence of trust-region methods based on probabilistic models. SIAM J. Opt. 24(3), 1238–1264 (2014)
Bellavia, S., Morini, B., Riccietti, E.: On an adaptive regularization for ill-posed nonlinear systems and its trust-region implementation. Comput. Opt. Appl. 64(1), 1–30 (2016)
Bellavia, S., Riccietti E.: On an elliptical trust-region procedure for Ill-posed nonlinear least-squares problems. J. Optim. Theor. Appl. (2018). https://doi.org/10.1007/s10957-018-1318-1
Bergou, E., Gratton, S., Vicente, L.: Levenberg-marquardt methods based on probabilistic gradient models and inexact subproblem solution, with application to data assimilation. SIAM/ASA J. Uncertain. Quantif. 4(1), 924–951 (2016)
Blanchet, J., Cartis, C., Menickelly, M., Scheinberg, K.: Convergence rate analysis of a stochastic trust region method for nonconvex optimization (2016) arXiv preprint arXiv:1609.07428
Bollapragada, R., Byrd, R., Nocedal, J.: Exact and inexact subsampled Newton methods for optimization (2016) arXiv preprint arXiv:1609.08502
Bottou, L., Curtis, F.E., Nocedal, J.: Optimization methods for large-scale machine learning (2016) arXiv preprint arXiv:1606.04838
Causality Workbench Team. A Marketing Dataset. http://www.causality.inf.ethz.ch/data/CINA.html (2008). Accessed 23 Jan 2017
Chen, R., Menickelly, M., Scheinberg, K.: Stochastic optimization using a trust-region method and random models. Math. Program. 144, 1–41 (2015)
Conn, A.R., Gould, N.I., Toint, P.L.: Trust Region Methods, vol. 1. SIAM, Philadelphia (2000)
Conn, A.R., Scheinberg, K., Vicente, L.N.: Introduction to Derivative-Free Optimization. SIAM, Philadelphia (2009)
Courtier, P., Thépaut, J.N., Hollingsworth, A.: A strategy for operational implementation of 4d-var, using an incremental approach. Quart. J. R. Meteorol. Soc. 120(519), 1367–1387 (1994)
Friedlander, M.P., Schmidt, M.: Hybrid deterministic-stochastic methods for data fitting. SIAM J. Sci. Comput. 34(3), A1380–A1405 (2012)
Gratton, S., Gürol, S., Toint, P.: Preconditioning and globalizing conjugate gradients in dual space for quadratically penalized nonlinear-least squares problems. Comput. Opt. Appl. 54(1), 1–25 (2013)
Gratton, S., Rincon-Camacho, M., Simon, E., Toint, P.L.: Observation thinning in data assimilation computations. EURO J. Comput. Opt. 3(1), 31–51 (2015)
Hanke, M.: A regularizing Levenberg–Marquardt scheme, with applications to inverse groundwater filtration problems. Inverse Probl. 13(1), 79 (1997)
Kaltenbacher, B., Neubauer, A., Scherzer, O.: Iterative Regularization Methods for Nonlinear Ill-Posed Problems, vol. 6. Walter de Gruyter, Berlin (2008)
Kelley, C.T.: Iterative Methods for Optimization: Matlab Codes. http://www4.ncsu.edu/~ctk/matlab_darts.html. Accessed 12 Mar 2017
Krejić, N., Jerinkić, N.K.: Nonmonotone line search methods with variable sample size. Numer. Algorithms 68(4), 711–739 (2015)
Krejić, N., Martínez, J.: Inexact restoration approach for minimization with inexact evaluation of the objective function. Math. Comput. 85(300), 1775–1791 (2016)
Nesterov, Y.: Introductory Lectures on Convex Optimization: A Basic Course, vol. 87. Springer, Berlin (2013)
Roosta-Khorasani, F., Mahoney, M.W.: Sub-sampled Newton methods 1: globally convergent algorithms (2016) arXiv preprint arXiv:1601.04737
Saunders, M.: Systems Optimization Laboratory. http://web.stanford.edu/group/SOL/software/cgls/. Accessed 15 Nov 2016
Stephens, M.A.: Edf statistics for goodness of fit and some comparisons. J. Am. Stat. Assoc. 69(347), 730–737 (1974)
Trémolet, Y.: Model-error estimation in 4d-var. Quart. J. R. Meteorol. Soc. 133(626), 1267–1280 (2007)
Weaver, A., Vialard, J., Anderson, D.: Three-and four-dimensional variational assimilation with a general circulation model of the tropical pacific ocean. Part I: formulation, internal diagnostics, and consistency checks. Mon. Weather Rev. 131(7), 1360–1378 (2003)
Wright, S., Nocedal, J.: Numerical optimization. Science 35, 67–68 (1999)
Zhao, R., Fan, J.: Global complexity bound of the Levenberg–Marquardt method. Opt. Methods Softw. 31(4), 805–814 (2016)
Acknowledgements
We thank the authors of [16] for providing us the Matlab code for the data assimilation test problem.
Author information
Authors and Affiliations
Corresponding author
Additional information
Work partially supported by INdAM-GNCS.
Rights and permissions
About this article
Cite this article
Bellavia, S., Gratton, S. & Riccietti, E. A Levenberg–Marquardt method for large nonlinear least-squares problems with dynamic accuracy in functions and gradients. Numer. Math. 140, 791–825 (2018). https://doi.org/10.1007/s00211-018-0977-z
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00211-018-0977-z