Skip to main content
SpringerLink
Log in

Worst-case evaluation complexity of a derivative-free quadratic regularization method

  • Original Paper
  • Published:
Optimization Letters Aims and scope Submit manuscript

Abstract

This manuscript presents a derivative-free quadratic regularization method for unconstrained minimization of a smooth function with Lipschitz continuous gradient. At each iteration, trial points are computed by minimizing a quadratic regularization of a local model of the objective function. The models are based on forward finite-difference gradient approximations. By using a suitable acceptance condition for the trial points, the accuracy of the gradient approximations is dynamically adjusted as a function of the regularization parameter used to control the stepsizes. Worst-case evaluation complexity bounds are established for the new method. Specifically, for nonconvex problems, it is shown that the proposed method needs at most \({\mathcal {O}}\left( n\epsilon ^{-2}\right)\) function evaluations to generate an \(\epsilon\)-approximate stationary point, where n is the problem dimension. For convex problems, an evaluation complexity bound of \({\mathcal {O}}\left( n\epsilon ^{-1}\right)\) is obtained, which is reduced to \({\mathcal {O}}\left( n\log (\epsilon ^{-1})\right)\) under strong convexity. Numerical results illustrating the performance of the proposed method are also reported.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Data availability

Data sharing not applicable to this article as no datasets were generated or analysed during the current study.

Notes

  1. In the context of nonconvex problems, evaluation complexity bounds of \({\mathcal {O}}\left( n^{2}\epsilon ^{-2}\right)\) were obtained by Vicente [23] and by Konecny and Richtárik [10] for direct search methods, and also by Garmanjani, Júdice and Vicente [7] for a derivative-free trust-region method.

  2. Evaluation complexity bounds of \({\mathcal {O}}\left( n\epsilon ^{-1}\right)\) and \({\mathcal {O}}\left( n\log (\epsilon ^{-1})\right)\) (in the convex and strongly convex cases, respectively) were established in [4, 18] for randomized DFO methods. They constitute upper bounds for the number of function evaluations that the corresponding methods need to find \(\bar{x}\) such that \(E[f(\bar{x})]-f^{*}\le \epsilon\), where \(f^{*}\) is the optimal value of \(f(\,\cdot \,)\) and E[X] denotes the expected value of a random variable X. For deterministic direct search methods, bounds of \({\mathcal {O}}\left( n^{2}\epsilon ^{-1}\right)\) and \({\mathcal {O}}\left( n^{2}\log (\epsilon ^{-1})\right)\) were established in [6, 10].

  3. Assumptions A1 and A2 are the usual assumptions for the analysis of first-order and derivative-free methods (see, e.g., Section 1.2.3 of [16]). In particular, any twice continuously differentiable function with uniformly bounded Hessian satisfies A1.

  4. In fact, for the theoretical guarantees, any other factor bigger than one can be used.

  5. If assumptions A1 and A5 hold, then \(\mu \le L\). Since \(L<C_{f}\), under these two assumptions it follows that \(\frac{\mu }{C_{f}}\in (0,1)\).

  6. The data profiles were generated using the code data_profile.m freely available in the website https://www.mcs.anl.gov/~more/dfo/.

  7. Namely, the datasets Iris, Breast Cancer Wisconsin, Wine, Sonar, Phishing, Ionosphere, Diabetes, Musk, Seeds, Bank Note Authentication, freely available in the website http://archive.ics.uci.edu/ml/index.php.

References

  1. Andrei, N.: An unconstrained optimization test functions collection. Adv. Model. Optim. 10(1), 147–161 (2008)

    MathSciNet  Google Scholar 

  2. Audet, C., Hare, W.: Derivative-free and blackbox optimization. Springer, Berlin (2017)

    Book  Google Scholar 

  3. Audet, C., Orban, D.: Finding optimal algorithmic parameters using derivative-free optimization. SIAM J. Optim. 17(3), 642–664 (2006)

    Article  MathSciNet  Google Scholar 

  4. Bergou, E.H., Gorbunov, E., Richtárik, P.: Stochastic three points method for unconstrained smooth minimization. SIAM J. Optim. 30(4), 2726–2749 (2020)

    Article  MathSciNet  Google Scholar 

  5. Conn, A.R., Scheinberg, K., Vicente, L.N.: Introduction to Derivative-Free Optimization. SIAM (2009)

  6. Dodangeh, M., Vicente, L.N.: Worst-case complexity of direct search under convexity. Math. Program. 155, 307–332 (2016)

    Article  MathSciNet  Google Scholar 

  7. Garmanjani, R., Júdice, D., Vicente, L.N.: Trust-region methods without using derivatives: worst case complexity and the nonsmooth case. SIAM J. Optim. 26(4), 1987–2011 (2016)

    Article  MathSciNet  Google Scholar 

  8. Grapiglia, G.N.: Quadratic regularization methods with finite-difference gradient approximations. Comput. Optim. Appl. (2022). https://doi.org/10.1007/s10589-022-00373-z

    Article  MathSciNet  Google Scholar 

  9. Karbasian, H.R., Vermeire, B.C.: Gradient-free aerodynamic shape optimization using large Eddy simulation. Comput. Fluids 232, 105185 (2022)

    Article  MathSciNet  Google Scholar 

  10. Konecny, J., Richtárik, P.: Simple complexity analysis of simplified direct search. arXiv:1410.0390 [math.OC] (2014)

  11. Larson, J., Menickelly, M., Wild, S.M.: Derivative-free optimization. Acta Numer. 28, 287–404 (2019)

    Article  MathSciNet  Google Scholar 

  12. Lojasiewicz, S.: A topological property of real analytic subsets (in French), Coll. du CNRS, Les équations aux dérivées partielles, pp. 87–89 (1963)

  13. Marsden, A.L., Feinstein, J.A., Taylor, C.A.: A computational framework for derivative-free optimization of cardiovascular geometries. Comput. Methods Appl. Mech. Eng. 197, 1890–1905 (2008)

    Article  MathSciNet  Google Scholar 

  14. Moré, J.J., Wild, S.M.: Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20(1), 172–191 (2009)

    Article  MathSciNet  Google Scholar 

  15. Nesterov, Yu.: How to make gradients small. Optima 88, 10–11 (2012)

    Google Scholar 

  16. Nesterov, Yu.: Lectures on Convex Optimization, 2nd edn. Springer, Berlin (2018)

    Book  Google Scholar 

  17. Nesterov, Yu., Polyak, B.T.: Cubic regularization of Newton method and its global performance. Math. Program. 108, 177–205 (2006)

    Article  MathSciNet  Google Scholar 

  18. Nesterov, Yu., Spokoiny, V.: Random gradient-free minimization of convex functions. Found. Comput. Math. 17, 527–566 (2017)

    Article  MathSciNet  Google Scholar 

  19. Polyak, B.T.: Gradient methods for minimizing functionals (in Russian). Zh. Vychisl. Mat. Mat. Fiz., 643–653 (1963)

  20. Russ, J.B., Li, R.L., Herschman, A.R., Haim, W., Vedula, V., Kysar, J.W., Kalfa, D.: Design optimization of a cardiovascular stent with application to a balloon expandable prosthetic heart valve. Mater. Des. 209 (2021)

  21. Sophy, O., Cartis, C., Kriest, I., Tett, S.F.B., Khatiwala, S.: A derivative-free optimisation method for global ocean biogeochemical models. Geosci. Model Dev. 15(9), 3537–3554 (2022)

    Article  Google Scholar 

  22. Sun, Y., Sahinidis, N.V., Sundaram, A., Cheon, M.-S.: Derivative-free optimization for chemical product design. Curr. Option Chem. Eng. 27, 98–106 (2020)

    Article  Google Scholar 

  23. Vicente, L.N.: Worst case complexity of direct search. EURO J. Comput. Optim. 1, 143–153 (2013)

    Article  Google Scholar 

Download references

Acknowledgements

The author is very grateful to the two anonymous referees, whose comments helped to improve the manuscript.

Author information

Authors and Affiliations

  1. ICTEAM/INMA, Université catholique de Louvain, Avenue Georges Lemaître, 4-6/ L4.05.01, B-1348, Louvain-la-Neuve, Belgium

    Geovani Nunes Grapiglia

Authors
  1. Geovani Nunes Grapiglia
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Geovani Nunes Grapiglia.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

G. N. Grapiglia was partially supported by CNPq (Grant 312777/2020-5).

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Grapiglia, G.N. Worst-case evaluation complexity of a derivative-free quadratic regularization method. Optim Lett 18, 195–213 (2024). https://doi.org/10.1007/s11590-023-01984-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11590-023-01984-z

Keywords

Search

Navigation