Abstract
Nonconvex minimization algorithms often benefit from the use of second-order information as represented by the Hessian matrix. When the Hessian at a critical point possesses negative eigenvalues, the corresponding eigenvectors can be used to search for further improvement in the objective function value. Computing such eigenpairs can be computationally challenging, particularly if the Hessian matrix itself cannot be built directly but must rather be sampled or approximated. In blackbox optimization, such derivative approximations are built at a significant cost in terms of function values. In this paper, we investigate practical approaches to detect negative eigenvalues in Hessian matrices without accessing the full matrix. We propose a general framework that begins with the diagonal and gradually builds submatrices to detect negative curvature. Crucially, our approach works both when exact Hessian coordinate values are available and when Hessian coordinate values are approximated. We compare several instances of our framework on a test set of Hessian matrices from a popular optimization library, and finite-differences approximations thereof. Our experiments highlight the importance of the variable order in the problem description, and show that forming submatrices is often an efficient approach to detect negative curvature.
Similar content being viewed by others
Notes
As A is a principle submatrix of A, if all principle submatrices are positive definite, then obviously A is positive definite. Conversely, if A is positive definite, then \(\lambda _1 > 0\), so the smallest eigenvalue of any principle submatrix is strictly positive and thus all principle submatrices are positive definite.
This number differs from the previous ones because applying an orthogonal transformation does not preserve the sign of the diagonal elements.
References
Goldfarb, D.: Curvilinear path steplength algorithms for minimization which use directions of negative curvature. Math. Program. 18, 31–40 (1980)
McCormick, G.P.: A modification of Armijo step-size rule for negative curvature. Math. Program. 13, 111–115 (1977)
Moré, J.J., Sorensen, D.C.: On the use of directions of negative curvature in a modified Newton method. Math. Program. 16, 1–20 (1979)
Cartis, C., Gould, N.I.M., Toint, P.L.: Complexity bounds for second-order optimality in unconstrained optimization. J. Complex. 28, 93–108 (2012)
Curtis, F.E., Lubberts, Z., Robinson, D.P.: Concise complexity analyses for trust region methods. Optim. Lett. 12, 1713–1724 (2018)
Royer, C.W., Wright, S.J.: Complexity analysis of second-order line-search algorithms for smooth nonconvex optimization. SIAM J. Optim. 28, 1448–1477 (2018)
Conn, A.R., Gould, N.I.M., Toint, P.L.: Trust-Region Methods. MPS-SIAM Series on Optimization. SIAM, Philadelphia (2000)
Fasano, G., Lucidi, S.: A nonmonotone truncated Newton–Krylov method exploiting negative curvature directions, for large-scale unconstrained optimization. Optim. Lett. 3, 521–535 (2009)
Audet, C., Hare, W.: Derivative-Free and Blackbox Optimization. Springer Series in Operations Research and Financial Engineering, Springer, Berlin (2017)
Conn, A.R., Scheinberg, K., Vicente, L.N.: Introduction to Derivative-Free Optimization. MPS-SIAM Series on Optimization. SIAM, Philadelphia (2009)
Dennis, J.E., Jr., Schnabel, R.B.: Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Classics in Applied Mathematics, SIAM, Philadelphia (1996)
Hare, W., Jarry-Bolduc, G., Planiden, C.: Hessian approximations. arXiv:2011.02584 (2020)
Hare, W., Srivastava, K.: Applying complex-step derivative approximations in model-based derivative-free optimization. Technical Report (2020)
Abramson, M.A., Frimannslund, L., Steihaug, T.: A subclass of generating set search with convergence to second-order stationary points. Optim. Methods Softw. 29, 900–918 (2014)
Conn, A.R., Scheinberg, K., Vicente, L.N.: Global convergence of general derivative-free trust-region algorithms to first- and second-order critical points. SIAM J. Optim. 20, 387–415 (2009)
Gratton, S., Royer, C.W., Vicente, L.N.: A second-order globally convergent direct-search method and its worst-case complexity. Optimization 65, 1105–1128 (2016)
Gratton, S., Royer, C.W., Vicente, L.N.: A decoupled first/second-order steps technique for nonconvex nonlinear unconstrained optimization with improved complexity bounds. Math. Program. 179, 195–222 (2020)
Júdice, D.: Trust-region methods without using derivatives: worst case complexity and the non-smooth case. PhD thesis, Dept. Mathematics, Univ. Coimbra (2015)
Partlett, B.N.: The Symmetric Eigenvalue Problem. Society for Industrial and Applied Mathematics, Philadelphia (1998)
Bron, C., Kerbosch, J.: Algorithm 457: finding all cliques of an undirected graph. Commun. ACM 16, 575–577 (1973)
Gould, N.I.M., Orban, D., Toint, P.L.: CUTEst: a constrained and unconstrained testing environment with safe threads. Comput. Optim. Appl. 60, 545–557 (2015)
Acknowledgements
The authors are grateful to two anonymous referees, whose insightful comments lead to improvements in the Nesa and Nesa \(_{\tilde{H}}\) algorithms.
Funding
Hare’s research is partially supported by NSERC Discovery Grant #2018-03865 and by France-Canada Research Funds 2022. Royer’s research is partially supported by Agence Nationale de la Recherche through program ANR-19-P3IA-0001 (PRAIRIE 3IA Institute) and by France-Canada Research Funds 2022.
Author information
Authors and Affiliations
Corresponding authors
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Hare, W., Royer, C.W. Detecting negative eigenvalues of exact and approximate Hessian matrices in optimization. Optim Lett 17, 1739–1756 (2023). https://doi.org/10.1007/s11590-023-02033-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11590-023-02033-5