A variation of Broyden class methods using Householder adaptive transforms

Abstract

In this work we introduce and study novel Quasi Newton minimization methods based on a Hessian approximation Broyden Class-type updating scheme, where a suitable matrix $\tilde{B}_k$ is updated instead of the current Hessian approximation $B_k$. We identify conditions which imply the convergence of the algorithm and, if exact line search is chosen, its quadratic termination. By a remarkable connection between the projection operation and Krylov spaces, such conditions can be ensured using low complexity matrices $\tilde{B}_k$ obtained projecting $B_k$ onto algebras of matrices diagonalized by products of two or three Householder matrices adaptively chosen step by step. Experimental tests show that the introduction of the adaptive criterion, which theoretically guarantees the convergence, considerably improves the robustness of the minimization schemes when compared with a non-adaptive choice; moreover, they show that the proposed methods could be particularly suitable to solve large scale problems where L-BFGS is not able to deliver satisfactory performance.

Appendices

Appendix 1: Householder matrices

The results contained in this section are borrowed from [13] and we refer the interested reader there for more details.

Definition 1

(Householder Orthogonal Matrix) Given a vector ${\mathbf {p}}\in \mathbb {R}^n$ define

$$\begin{aligned} {\mathcal {H}}({\mathbf {p}}):=I_n-\frac{2}{\Vert {\mathbf {p}}\Vert ^2}{\mathbf {p}}{\mathbf {p}}^T. \end{aligned}$$

Consider two vectors ${\mathbf {v}},\, {\mathbf {z}}\in \mathbb {R}^n$. From direct computation one can check that defining ${\mathbf {p}}= {\mathbf {v}}- \frac{\Vert {\mathbf {v}}\Vert }{\Vert {\mathbf {z}}\Vert }{\mathbf {z}}$ with ${\mathbf {z}}\ne 0,$ we have

$$\begin{aligned} {\mathcal {H}}({\mathbf {p}}){\mathbf {v}}=\frac{\Vert {\mathbf {v}}\Vert }{\Vert {\mathbf {z}}\Vert }{\mathbf {z}}. \end{aligned}$$

Lemma 5

([13]) Consider $W=[{\mathbf {w}}_1|\dots |{\mathbf {w}}_s] \in \mathbb {R}^{n\times s}, V=[{\mathbf {v}}_1|\dots |{\mathbf {v}}_s] \in \mathbb {R}^{n\times s}$of full rank and such that $s \le n$, $W^TW=V^TV$. Then there exist $\,{\mathbf {h}}_1, \dots ,{\mathbf {h}}_{s} \in \mathbb {R}^n$, $\Vert {\mathbf {h}}_i\Vert =\sqrt{2}$, such that the orthogonal matrix $U=\mathcal {H}({\mathbf {h}}_s)\cdots \mathcal {H}({\mathbf {h}}_1)$, product of s Householder matrices, satisfies the following identities

$$\begin{aligned} U {\mathbf {w}}_{i}= {\mathbf {v}}_i \hbox { for all } i \in \{1, \dots , s\}. \end{aligned}$$

The vectors ${\mathbf {h}}_i$for $i\in \{1,\dots ,s\}$can be obtained by setting:

$$\begin{aligned} \begin{aligned}&{ \tilde{{\mathbf {h}}}_i := \mathcal {H}({\mathbf {h}}_{i-1}) \cdots \mathcal {H}({\mathbf {h}}_1) ({\mathbf {w}}_{i}-{\mathbf {w}}_{i-1}) - ({\mathbf {v}}_{i}-{\mathbf {v}}_{i-1}),}\\&{\mathbf {h}}_i:= ( \sqrt{2}/\Vert \tilde{{\mathbf {h}}}_i\Vert )\tilde{{\mathbf {h}}}_i \end{aligned} \end{aligned}$$

(65)

(where we set ${\mathbf {h}}_0={\mathbf {w}}_0={\mathbf {v}}_0=\varvec{0}$). If $s=n$we have ${\mathbf {h}}_n=\mathbf {0}$or ${\mathbf {h}}_n=\frac{\sqrt{2}}{\Vert {\mathbf {v}}_n\Vert }{\mathbf {v}}_n.$The cost of the computation of the ${\mathbf {h}}_i$for $i=1,\dots ,s$is:

$$\begin{aligned} \begin{aligned}&[s(s-1)n+ s(2n+1)] \hbox { multiplications } \\&\quad +[(s(s+2) - 2)n + s(n-1)] \hbox { additions } \\&\quad + s \hbox { square roots.} \end{aligned} \end{aligned}$$

Observe that when ${\mathbf {w}}_i={\mathbf {e}}_{k_i}$for $i=1,\dots ,s$, that is when ${\mathbf {v}}_1, \dots , {\mathbf {v}}_s$are orthonormal and we are interested to construct an orthogonal U with s columns fixed as ${\mathbf {v}}_1, \dots , {\mathbf {v}}_s$, it is possible to save $(s-1)n \hbox { mult.}$and $(3s-2)n \hbox { add..}$

Proof

The explicit expression of the ${\mathbf {h}}_i$ in (65) is obtained by applying the techniques for their construction introduced in [13]. $\square$

Appendix 2: Details on Theorem 1

In order to prove inequality (34) it is enough to prove the following:

Lemma 6

There exists $c_3$, constant with respect to j and depending only on s and M, such that

$$\begin{aligned} \gamma ((j+1-s)+1)^{n} \le c_3^{j+1-s} \hbox { for all } j \ge s, \hbox { where } \gamma :=\left( \frac{c_1}{n}\right) ^n\frac{1}{\det B_s} \end{aligned}$$

(of course, such $c_3$turns out to be greater than 1).

In fact, once Lemma 6 is proved, the constant $c_2$ (constant with respect to j) for which (34) is verified, will be $c_2=2c_1c_3/(1-\beta )$ (note that $c_2$ depends only s, M, $\beta$ but not on j).

Proof

Fix $\tilde{c}_3>1$. Note that the sequence of positive numbers

$$\begin{aligned} \frac{\gamma ((j+1-s)+1)^n}{\tilde{c}_3^{j+1-s}} \hbox { for } j= s,s+1,\dots \end{aligned}$$

converges to zero as $j \rightarrow +\infty$; thus there exists $j^{*}\ge s$ (depending on s, M and $\tilde{c}_3$) s.t.

$$\begin{aligned} \gamma ((j+1-s)+1)^n \le {\tilde{c}_3^{j+1-s}} \hbox { for all } j \ge j^{*}. \end{aligned}$$

Note also that for all $j \in \{s+1, \dots , j^{*}-1\}$ we have

$$\begin{aligned} \gamma ((j+1-s)+1)^n \le \gamma (j^{*}-s+1)^n \end{aligned}$$

(66)

and consider $\hat{j} \ge j^{*}$ s.t. $\gamma (\hat{j}-s+1)^n>1$ ($\hat{j}$ depends on s, M, $\gamma$ and $\tilde{c}_3$). From (66) we have

$$\begin{aligned} \gamma ((j+1-s)+1)^n \le \gamma (\hat{j}-s+1)^n \le (\gamma (\hat{j}-s+1)^n)^{j+1-s} \end{aligned}$$

for all $j \in \{s,s+1, \dots j^{*}-1\}$.

Collecting the above results, we can conclude that

$$\begin{aligned} \gamma ((j+1-s)+1)^n \le c_3^{j+1-s} \hbox { for all } j \ge s \end{aligned}$$

(67)

where $c_3:=\max \{ \tilde{c}_3, \gamma (\hat{j}-s+1)^n\}$ ($c_3>1$ and depends on s, M and $\tilde{c}_3$).

Finally note that, once $\tilde{c}_3$ is fixed, it is clear that $c_3$ depends only on s, M. $\square$

In order to prove inequality ($34_{1}$), define $a_k:=(1-\phi -\psi _k\phi )\Vert {\mathbf {g}}_k\Vert ^2/{\mathbf {s}}_k^{T}(-{\mathbf {g}}_k)>0$. We know that $\lim _{k \rightarrow +\infty }a_k=+\infty$ and we have to show that there exists $j^{*}\ge s$ such that

$$\begin{aligned} \prod _{k=s}^{j} a_k > c_2^{j+1-s} \hbox { for all } j \ge j^{*}. \qquad \qquad \qquad \qquad \qquad (34_{1}\hbox {bis}) \end{aligned}$$

If $a_k\ge c_2$ for all $k \ge s$, since it must be $a_k>c_2$ for infinite indexes k, then the thesis is obvious. So assume that there exists some index k such that $a_k < c_2$. Let $r \ge s$ be such that $a_k>c_2$ for all $k>r$. Note that $c_2>\min _{k=s, \dots , r}a_k$. Set

$$\begin{aligned} t:=\left( \frac{c_2}{\min _{k=s, \dots , r}a_k} \right) ^{r+1-s} >1. \end{aligned}$$

Let $j^{*}>r+1$ be such that $a_k \ge t c_2$ for all $k\ge j^{*}$. Then we have

$$\begin{aligned} \begin{aligned} \prod _{k=s}^{j^{*}}a_k&=\left( \prod _{k=s}^{r}a_k\right) \left( \prod _{k=r+1}^{j^{*}-1}a_k\right) a_{j^{*}} \\&>\left( \min _{k=s,\dots ,r}a_k\right) ^{r-s+1}c_2^{j^{*}-r-1}tc_2\\&=\left( \min _{k=s,\dots ,r}a_k\right) ^{r-s+1}\left( \frac{c_2}{\min _{k=s, \dots , r}a_k} \right) ^{r-s+1}c_2^{j^{*}-r}=c_2^{j^*-s+1}, \end{aligned} \end{aligned}$$

i.e., $\prod _{k=s}^{j^{*}}a_k>c_2^{j^*-s+1}$. Thus we obtain ($34_{1}\hbox {bis}$) since $a_k\ge tc_2>c_2$ for $k> j^{*}$.