Abstract
We consider a family of dense initializations for limited-memory quasi-Newton methods. The proposed initialization exploits an eigendecomposition-based separation of the full space into two complementary subspaces, assigning a different initialization parameter to each subspace. This family of dense initializations is proposed in the context of a limited-memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) trust-region method that makes use of a shape-changing norm to define each subproblem. As with L-BFGS methods that traditionally use diagonal initialization, the dense initialization and the sequence of generated quasi-Newton matrices are never explicitly formed. Numerical experiments on the CUTEst test set suggest that this initialization together with the shape-changing trust-region method outperforms other L-BFGS methods for solving general nonconvex unconstrained optimization problems. While this dense initialization is proposed in the context of a special trust-region method, it has broad applications for more general quasi-Newton trust-region and line search methods. In fact, this initialization is suitable for use with any quasi-Newton update that admits a compact representation and, in particular, any member of the Broyden class of updates.
Similar content being viewed by others
References
Becker, S.: LBFGSB (L-BFGS-B) mex wrapper (2012–2015). https://www.mathworks.com/matlab central/fileexchange/35104-lbfgsb-l-bfgs-b-mex-wrapper. Accessed Jan 2017
Brust, J., Burdakov, O., Erway, J.B., Marcia, R.F., Yuan, Y.X.: Shape-changing L-SR1 trust-region methods. Technical Report 2016-2, Wake Forest University (2016)
Burdakov, O., Gong, L., Yuan, Y.X., Zikrin, S.: On efficiently combining limited memory and trust-region techniques. Math. Program. Comput. 9, 101–134 (2016)
Burke, J.V., Wiegmann, A., Xu, L.: Limited memory BFGS updating in a trust-region framework. Technical Report, University of Washington (1996)
Byrd, R.H., Nocedal, J., Schnabel, R.B.: Representations of quasi-Newton matrices and their use in limited-memory methods. Math. Program. 63, 129–156 (1994)
DeGuchy, O., Erway, J.B., Marcia, R.F.: Compact representation of the full Broyden class of quasi-Newton updates. Numer. Linear Algebra Appl. 25(5), e2186 (2018)
Dolan, E., Moré, J.: Benchmarking optimization software with performance profiles. Math. Program. 91, 201–213 (2002)
Erway, J.B., Marcia, R.F.: On efficiently computing the eigenvalues of limited-memory quasi-Newton matrices. SIAM J. Matrix Anal. Appl. 36(3), 1338–1359 (2015)
Erway, J.B., Marcia, R.F.: On solving large-scale limited-memory quasi-Newton equations. Linear Algebra Appl. 515, 196–225 (2017)
Gould, N.I.M., Orban, D., Toint, P.L.: CUTEr and SifDec: a constrained and unconstrained testing environment, revisited. ACM Trans. Math. Softw. 29(4), 373–394 (2003)
Lu, X.: A study of the limited memory SR1 method in practice. Ph.D. thesis, University of Colorado (1992)
Lukšan, L., Vlček, J.: Recursive form of general limited memory variable metric methods. Kybernetika 49, 224–235 (2013)
Nocedal, J., Wright, S.J.: Numerical Optimization. Springer, New York (1999)
Shanno, D.F., Phua, K.H.: Matrix conditioning and nonlinear optimization. Math. Program. 14(1), 149–160 (1978)
Zhu, C., Byrd, R., Nocedal, J.: Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization. ACM Trans. Math. Softw. 23, 550–560 (1997)
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This research is supported by NSF Grants CMMI-1334042, CMMI-1333326, IIS-1741490, and IIS-1741264.
Rights and permissions
About this article
Cite this article
Brust, J., Burdakov, O., Erway, J.B. et al. A dense initialization for limited-memory quasi-Newton methods. Comput Optim Appl 74, 121–142 (2019). https://doi.org/10.1007/s10589-019-00112-x
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10589-019-00112-x
Keywords
- Large-scale nonlinear optimization
- Limited-memory quasi-Newton methods
- Trust-region methods
- Quasi-Newton matrices
- Shape-changing norm