Practical inexact proximal quasi-Newton method with global complexity analysis

Abstract

Recently several methods were proposed for sparse optimization which make careful use of second-order information (Hsieh et al. in Sparse inverse covariance matrix estimation using quadratic approximation. In: NIPS, 2011; Yuan et al. in An improved GLMNET for l1-regularized logistic regression and support vector machines. National Taiwan University, Taipei City, 2011; Olsen et al. in Newton-like methods for sparse inverse covariance estimation. In: NIPS, 2012; Byrd et al. in A family of second-order methods for convex l1-regularized optimization. Technical report, 2012) to improve local convergence rates. These methods construct a composite quadratic approximation using Hessian information, optimize this approximation using a first-order method, such as coordinate descent and employ a line search to ensure sufficient descent. Here we propose a general framework, which includes slightly modified versions of existing algorithms and also a new algorithm, which uses limited memory BFGS Hessian approximations, and provide a novel global convergence rate analysis, which covers methods that solve subproblems via coordinate descent.

Appendix

Proof of Lemma 2.

Proof

Let $p_{\phi }(v)$ denote $p_{H,\phi }(v)$ for brevity. From (3.6) and (2.1), we have

$$\begin{aligned} F(u) - F(p_{\phi }(v))\ge & {} F(u) - Q(H, p_{\phi }(v),v) -\epsilon \nonumber \\= & {} F(u) - (f(v)+ g(p_{\phi }(v)) + \langle \nabla f(v),p_{\phi }(v)-v\rangle \nonumber \\&+ \frac{1}{2}\Vert p_{\phi }(v)-v\Vert _H^2)-\epsilon . \end{aligned}$$

(9.1)

Also

$$\begin{aligned} g(u) \ge g(p_{\phi }(v)) + \langle u-p_{\phi }(v), \gamma _g(p_{\phi }(v)) \rangle - \phi \end{aligned}$$

(9.2)

by the definition of $\phi$-subgradient, and

$$\begin{aligned} f(u) \ge f(v) + \langle u- v, \nabla f(v) \rangle , \end{aligned}$$

(9.3)

due to the convexity of f. Here $\gamma _g(\cdot )$ is any subgradient of $g(\cdot )$ and $\gamma _g(p_{\phi }(v))$ is an $\phi$-subgradient, which satisfies the first-order optimality conditions for $\phi$-approximate minimizer from Lemma 1 with $z=v-H^{-1}\nabla f(v)$, i.e.,

$$\begin{aligned} \gamma _g(p_{\phi }(v)) = H(v-p_{\phi }(v)) - \nabla f(v) - \eta , \text{ with } \frac{1}{2} \Vert \eta \Vert ^2_{H^{-1}} \le \phi . \end{aligned}$$

(9.4)

Summing (9.2) and (9.3) yields

$$\begin{aligned} F(u) \ge g(p_{\phi }(v)) + \langle u-p_{\phi }(v), \gamma _g(p_{\phi }(v)) \rangle - \phi + f(v) + \langle u- v, \nabla f(v)\rangle .\qquad \end{aligned}$$

(9.5)

Therefore, from (9.1), (9.4) and (9.5) it follows that

$$\begin{aligned} F(u) - F(p_{\phi }(v))&\ge \langle \nabla f(v)+\gamma _g(p_{\phi }(v)), u-p_{\phi }(v) \rangle - \frac{1}{2}\Vert p_{\phi }(v)-v\Vert _H^2 - \epsilon - \phi \\&= \langle -H(p_{\phi }(v)-v) - \eta , u-p_{\phi }(v)\rangle - \frac{1}{2}\Vert p_{\phi }(v)-v\Vert _H^2-\epsilon - \phi \\&= \frac{1}{2}\Vert p_{\phi }( v)-u\Vert _H^2 - \frac{1}{2} \Vert v-u\Vert _H^2-\epsilon -\phi - \langle \eta , u - p_{\phi }(v) \rangle . \end{aligned}$$

$\square$