An adaptive trust-region method without function evaluations

Abstract

In this paper we propose an adaptive trust-region method for smooth unconstrained optimization. The update rule for the trust-region radius relies only on gradient evaluations. Assuming that the gradient of the objective function is Lipschitz continuous, we establish worst-case complexity bounds for the number of gradient evaluations required by the proposed method to generate approximate stationary points. As a corollary, we establish a global convergence result. We also present numerical results on benchmark problems. In terms of the number of calls of the oracle, the proposed method compares favorably with trust-region methods that use evaluations of the objective function.

Appendix

1.1 Proof of Lemma 4

By definition, $k_{i}\le q$. If $k_{i}\ge q-1$, then $|I(k_{i},q)|\le 2$ and so (2.22) holds. Now, suppose that $k_{i}<q-1$. By (2.4) we have

$$\begin{aligned} b_{j+1}-b_{j}=\dfrac{\Vert \nabla f(x_{j+1})\Vert ^{2}}{b_{j}}\quad \text {for}\quad j=k_{i},\ldots ,q-1. \end{aligned}$$

Summing up these equalities, it follows from (2.11) and (2.21) that

$$\begin{aligned} b_{q}-b_{k_{i}}=\sum _{j\in I(k_{i},q-1)}\dfrac{\Vert \nabla f(x_{j+1})\Vert ^{2}}{b_{j}}> |I(k_{i},q-1)|{\tilde{L}}^{-1}\epsilon ^{2}, \end{aligned}$$

and so

$$\begin{aligned} |I(k_{i},q-1)|<{\tilde{L}}\left( b_{q}-b_{k_{i}}\right) \epsilon ^{-2}<{\tilde{L}}b_{q}\epsilon ^{-2}. \end{aligned}$$

Since $b_q<{\tilde{L}}$, we obtain

$$\begin{aligned} |I(k_{i},q-1)|< {\tilde{L}}^{2}\epsilon ^{-2}, \end{aligned}$$

which gives

$$\begin{aligned} |I(k_{i},q)|<{\tilde{L}}^{2}\epsilon ^{-2}+1, \end{aligned}$$

Therefore, (2.22) also holds in this case. $\square$

1.2 Proof of Lemma 5

Let $k\in I(p+1,k_{i+1}-1)$. Then, by (2.4), $b_{j}\ge {\tilde{L}}$ for $j=p,\ldots ,k-1$. Consequently, by Lemma 3, we have

$$\begin{aligned} f(x_{j})-f(x_{j+1})\ge \dfrac{\Vert \nabla f(x_{j})\Vert ^{2}}{4b_{j}}\quad \text {for all}\quad j=p,\ldots ,k-1. \end{aligned}$$

Summing up these inequalities we get

$$\begin{aligned} f(x_{p})-f(x_{k})\ge \dfrac{1}{4}\sum _{j=p}^{k-1}\dfrac{\Vert \nabla f(x_{j})\Vert ^{2}}{b_{j}}, \end{aligned}$$

(4.1)

and so, by A2,

$$\begin{aligned} \sum _{j=p}^{k-1}\dfrac{\Vert \nabla f(x_{j})\Vert ^{2}}{b_{j}}\le 4\left( f(x_{p})-f(x_{k})\right) \le 4\left( f(x_{p})-f_{low}\right) . \end{aligned}$$

(4.2)

On the other hand, by (2.4) and A1, we also have

$$\begin{aligned} b_{k}-b_{p}= & {} \sum _{j=p}^{k-1}\left( b_{j+1}-b_{j}\right) =\sum _{j=p}^{k-1}\dfrac{\Vert \nabla f(x_{j+1})\Vert ^{2}}{b_{j}}\nonumber \\\le & {} \sum _{j=p}^{k-1}\dfrac{\left( \Vert \nabla f(x_{j})-\nabla f(x_{j+1})\Vert +\Vert \nabla f(x_{j})\Vert \right) ^{2}}{b_{j}}\nonumber \\\le & {} 2\sum _{j=p}^{k-1}\dfrac{\Vert \nabla f(x_{j})-\nabla f(x_{j+1})\Vert ^{2}+\Vert \nabla f(x_{j})\Vert ^{2}}{b_{j}}\nonumber \\\le & {} 2\sum _{j=p}^{k-1}\dfrac{L^{2}\Vert x_{j}-x_{j+1}\Vert ^{2}+\Vert \nabla f(x_{j})\Vert ^{2}}{b_{j}}. \end{aligned}$$

(4.3)

By (2.2),

$$\begin{aligned} \Vert x_{j}-x_{j+1}\Vert =\Vert d_{j}\Vert \le \delta _{j}\Vert \nabla f(x_{j})\Vert =\dfrac{\Vert \nabla f(x_{j})\Vert }{b_{j}}. \end{aligned}$$

(4.4)

Then, combining (4.3) and (4.4) and using $b_{j}\ge {\tilde{L}}$, it follows that

$$\begin{aligned} b_{k}-b_{p}\le & {} 2\sum _{j=p}^{k-1}\dfrac{L^{2}\Vert \nabla f(x_{j})\Vert ^{2}}{b_{j}^{3}}+2\sum _{j=p}^{k-1}\dfrac{\Vert \nabla f(x_{j})\Vert ^{2}}{b_{j}}\nonumber \\\le & {} 2\sum _{j=p}^{k-1}\dfrac{{\tilde{L}}^{2}\Vert \nabla f(x_{j})\Vert ^{2}}{b_{j}^{2}b_{j}}+2\sum _{j=p}^{k-1}\dfrac{\Vert \nabla f(x_{j})\Vert ^{2}}{b_{j}}\nonumber \\\le & {} 4\sum _{j=p}^{k-1}\dfrac{\Vert \nabla f(x_{j})\Vert ^{2}}{b_{j}}. \end{aligned}$$

(4.5)

Now, combining (4.5) and (4.2), we obtain

$$\begin{aligned} b_{k}-b_{p}\le 16\left( f(x_{p})-f_{low}\right) , \end{aligned}$$

and so

$$\begin{aligned} b_{k}\le b_{p}+16\left( f(x_{p})-f_{low}\right) ,\quad \forall k\in I(p,k_{i+1}-1). \end{aligned}$$

(4.6)

Our first goal is to refine the upper bound for $b_{k}$ in (4.6). For that, we will break the analysis into a few cases and subcases related with the position of p in the set $I(k_{i},k_{i+1}-2)$.

Case I $p=k_{i}$.

In this case, it follows from (4.6) that

$$\begin{aligned} b_{k}\le b_{k_{i}}+16(f(x_{k_{i}})-f_{low})\quad \forall k\in I(p,k_{i+1}-1). \end{aligned}$$

(4.7)

Case II $p\in I(k_{i}+1,k_{i+1}-2)$.

By A1 and the trust-region constraint, we have

$$\begin{aligned} b_{p}= & {} b_{p-1}+\dfrac{\Vert \nabla f(x_{p})\Vert ^{2}}{b_{p-1}}\nonumber \\\le & {} b_{p-1}+\dfrac{\left( \Vert \nabla f(x_{p-1})-\nabla f(x_{p})\Vert +\Vert \nabla f(x_{p-1})\Vert \right) ^{2}}{b_{p-1}}\nonumber \\\le & {} b_{p-1}+2\dfrac{\Vert \nabla f(x_{p-1})-\nabla f(x_{p})\Vert ^{2}+\Vert \nabla f(x_{p-1})\Vert ^{2}}{b_{p-1}}\nonumber \\\le & {} b_{p-1}+2\dfrac{L^{2}\Vert x_{p-1}-x_{p}\Vert ^{2}+\Vert \nabla f(x_{p-1})\Vert ^{2}}{b_{p-1}}\nonumber \\\le & {} b_{p-1}+\dfrac{2L^{2}\Vert \nabla f(x_{p-1})\Vert ^{2}}{b_{p-1}^{3}}+\dfrac{2\Vert \nabla f(x_{p-1})\Vert ^{2}}{b_{p-1}}. \end{aligned}$$

(4.8)

Moreover, given $j\in I(k_{i},p-1)$, we also have

$$\begin{aligned} f(x_{j+1})\le & {} f(x_{j})+\nabla f(x_{j})^{T}(x_{j+1}-x_{j})+\dfrac{L}{2}\Vert x_{j+1}-x_{j}\Vert ^{2}\nonumber \\\le & {} f(x_{j})+\Vert \nabla f(x_{j})\Vert \Vert d_{j}\Vert +\dfrac{L}{2}\Vert d_{j}\Vert ^{2}\nonumber \\\le & {} f(x_{j})+\dfrac{\Vert \nabla f(x_{j})\Vert ^{2}}{b_{j}}+\dfrac{L}{2b_{j}}\dfrac{\Vert \nabla f(x_{j})\Vert ^{2}}{b_{j}}\nonumber \\= & {} f(x_{j})+\left( 1 + \dfrac{L}{2b_j} \right) \dfrac{\Vert \nabla f(x_{j})\Vert ^{2}}{b_{j}}\nonumber \\\le & {} f(x_{j})+\left( 1 + \dfrac{L}{2b_{\min }} \right) \dfrac{\Vert \nabla f(x_{j})\Vert ^{2}}{b_{j}}. \end{aligned}$$

(4.9)

Case II(a) $p=k_{i}+1$.

In this case, by (4.8) we have

$$\begin{aligned} b_{p}\le b_{k_{i}}+\dfrac{2L^{2}\Vert \nabla f(x_{k_{i}})\Vert ^{2}}{b_{k_{i}}^{3}}+\dfrac{2\Vert \nabla f(x_{k_{i}})\Vert ^{2}}{b_{k_{i}}}. \end{aligned}$$

(4.10)

On the other hand, by (4.9) with $j=k_{i}$, we get

$$\begin{aligned} f(x_{p})\le f(x_{k_{i}})+\left( 1+\dfrac{L}{2b_{\min }}\right) \dfrac{\Vert \nabla f(x_{k_{i}})\Vert ^{2}}{b_{k_{i}}}. \end{aligned}$$

(4.11)

Thus, combining (4.6), (4.10) and (4.11), it follows that

$$\begin{aligned} b_{k}\le & {} b_{k_{i}}+\dfrac{2L^{2}\Vert \nabla f(x_{k_{i}})\Vert ^{2}}{b_{k_{i}}^{3}}+\dfrac{2\Vert \nabla f(x_{k_{i}})\Vert ^{2}}{b_{k_{i}}}\nonumber \\&+16\left[ \left( 1+\dfrac{L}{2b_{\min }}\right) \dfrac{\Vert \nabla f(x_{k_{i}})\Vert ^{2}}{b_{k_{i}}}+f(x_{k_{i}})-f_{low}\right] \nonumber \\= & {} b_{k_{i}}+\left( \dfrac{4L^{2}+36b_{\min }^{2}+16b_{min}L}{2b_{\min }^{3}}\right) \Vert \nabla f(x_{k_{i}})\Vert ^{2}+16\left( f(x_{k_{i}})-f_{low}\right) \end{aligned}$$

(4.12)

for all $k\in I(p,k_{i+1}-1)$.

Case II(b) $p\in I(k_{i}+2,k_{i+1}-2)$.

In this case, $b_{p-1}\ge b_{p-2}$ and so, by (4.8), we get

$$\begin{aligned} b_{p}\le & {} b_{p-1}+\dfrac{2L^{2}\Vert \nabla f(x_{p-1})\Vert ^{2}}{b_{p-1}^{2}b_{p-2}}+\dfrac{2\Vert \nabla f(x_{p-1})\Vert ^{2}}{b_{p-2}}\\= & {} b_{p-1}+\dfrac{2L^{2}(b_{p-1}-b_{p-2})}{b_{p-1}^{2}}+\dfrac{2\Vert \nabla f(x_{p-1})\Vert ^{2}}{b_{p-2}}\\< & {} b_{p-1}+\dfrac{2L^{2}}{b_{p-1}}+2(b_{p-1}-b_{p-2})\\< & {} 3b_{p-1}+\dfrac{2L^{2}}{b_{p-1}}. \end{aligned}$$

Since $b_{\min }\le b_{p-1}<{\tilde{L}}$, it follows that

$$\begin{aligned} b_{p}\le 3{\tilde{L}}+\dfrac{2L^{2}}{b_{\min }}. \end{aligned}$$

(4.13)

On the other hand, by (4.9) we have

$$\begin{aligned} f(x_{p})-f(x_{k_{i}})= & {} \sum _{j=k_{i}}^{p-1}f(x_{j+1})-f(x_{j})\\\le & {} \left( 1+\dfrac{L}{2b_{\min }}\right) \sum _{j=k_{i}}^{p-1}\dfrac{\Vert \nabla f(x_{j})\Vert ^{2}}{b_{j}}\\= & {} \left( 1+\dfrac{L}{2b_{\min }}\right) \dfrac{\Vert \nabla f(x_{k_{i}})\Vert ^{2}}{b_{k_{i}}}+\left( 1+\dfrac{L}{2b_{\min }}\right) \sum _{j=k_{i}+1}^{p-1}\dfrac{\Vert \nabla f(x_{j})\Vert ^{2}}{b_{j}}\\\le & {} \left( 1+\dfrac{L}{2b_{\min }}\right) \dfrac{\Vert \nabla f(x_{k_{i}})\Vert ^{2}}{b_{k_{i}}}+\left( 1+\dfrac{L}{2b_{\min }}\right) \sum _{j=k_{i}+1}^{p-1}\dfrac{\Vert \nabla f(x_{j})\Vert ^{2}}{b_{j-1}}\\= & {} \left( 1+\dfrac{L}{2b_{\min }}\right) \dfrac{\Vert \nabla f(x_{k_{i}})\Vert ^{2}}{b_{k_{i}}}+\left( 1+\dfrac{L}{2b_{\min }}\right) \sum _{j=k_{i}+1}^{p-1}(b_{j}-b_{j-1})\\= & {} \left( 1+\dfrac{L}{2b_{\min }}\right) \dfrac{\Vert \nabla f(x_{k_{i}})\Vert ^{2}}{b_{k_{i}}}+\left( 1+\dfrac{L}{2b_{\min }}\right) (b_{p-1}-b_{k_{i}})\\< & {} \left( 1+\dfrac{L}{2b_{\min }}\right) \dfrac{\Vert \nabla f(x_{k_{i}})\Vert ^{2}}{b_{k_{i}}}+\left( 1+\dfrac{L}{2b_{\min }}\right) b_{p-1}\\< & {} \left( 1+\dfrac{L}{2b_{\min }}\right) \dfrac{\Vert \nabla f(x_{k_{i}})\Vert ^{2}}{b_{k_{i}}}+\left( 1+\dfrac{L}{2b_{\min }}\right) {\tilde{L}}. \end{aligned}$$

and so

$$\begin{aligned} f(x_{p})\le f(x_{k_{i}})+\left( 1+\dfrac{L}{2b_{\min }}\right) \dfrac{\Vert \nabla f(x_{k_{i}})\Vert ^{2}}{b_{\min }}+\left( 1+\dfrac{L}{2b_{\min }}\right) {\tilde{L}}. \end{aligned}$$

(4.14)

Thus, combining (4.6), (4.13) and (4.14), it follows that

$$\begin{aligned} b_{k}\le & {} 3{\tilde{L}}+\dfrac{2L^{2}}{b_{\min }}+16\left[ \left( 1+\dfrac{L}{2b_{\min }}\right) \dfrac{\Vert \nabla f(x_{k_{i}})\Vert ^{2}}{b_{\min }}+\left( 1+\dfrac{L}{2b_{\min }}\right) {\tilde{L}} + f(x_{k_{i}})-f_{low}\right] \nonumber \\\le & {} 19{\tilde{L}}+\dfrac{10{\tilde{L}}^{2}}{b_{\min }}+16\left( 1+\dfrac{L}{2b_{\min }}\right) \dfrac{\Vert \nabla f(x_{k_{i}})\Vert ^{2}}{b_{\min }}+16(f(x_{k_{i}})-f_{low}), \end{aligned}$$

(4.15)

for all $k\in I(p,k_{i+1}-1)$.

Summarizing all cases and subcases above, it follows from (4.7), (4.12), and (4.15) that

$$\begin{aligned} b_{k}\le b_{k_{i}}+19{\tilde{L}}+\dfrac{10{\tilde{L}}^{2}}{b_{\min }}+\left( \dfrac{4L^{2}+36b_{\min }^{2}+16b_{\min }L}{2b_{\min }^{3}}\right) \Vert \nabla f(x_{k_{i}})\Vert ^{2}+16(f(x_{k_{i}})-f_{low}) \end{aligned}$$

(4.16)

for all $k\in I(p,k_{i+1}-1)$, regardless of the position of p in the set $I(k_{i},k_{i+1}-2)$. Finally, by (2.4) and Lemma 3,

$$\begin{aligned} f(x_{j})-f(x_{j+1})\ge \dfrac{\Vert \nabla f(x_{j})\Vert ^{2}}{4b_{j}}\quad \text {for}\,\,j=p,\ldots ,k_{i+1}-1. \end{aligned}$$

Summing up these inequalities, it follows from A2, (2.11) and (4.16) that

$$\begin{aligned} f(x_{p})-f_{low}\ge & {} f(x_{p})-f(x_{k_{i+1}})\nonumber \\\ge & {} \sum _{j=p}^{k_{i+1}-1}\dfrac{\Vert \nabla f(x_{j})\Vert ^{2}}{4b_{j}}\nonumber \\\ge & {} \dfrac{|I(p,k_{i+1}-1)|\epsilon ^{2}}{4\left[ b_{k_{i}}+19{\tilde{L}}+\dfrac{10{\tilde{L}}^{2}}{b_{\min }}+\left( \dfrac{4L^{2}+36b_{\min }^{2}+16b_{\min }L}{2b_{\min }^{3}}\right) \Vert \nabla f(x_{k_{i}})\Vert ^{2}+16(f(x_{k_{i}})-f_{low})\right] }\nonumber \\&\end{aligned}$$

(4.17)

By (4.11) and (4.14), we also have

$$\begin{aligned} f(x_{p})-f_{low}\le & {} \left( 1+\dfrac{L}{2b_{\min }}\right) {\tilde{L}}+\left( 1+\dfrac{L}{2b_{\min }}\right) \dfrac{\Vert \nabla f(x_{k_{i}})\Vert ^{2}}{b_{\min }}+(f(x_{k_{i}})-f_{low})\nonumber \\\le & {} {\tilde{L}}+\dfrac{{\tilde{L}}^{2}}{2b_{\min }}+\left( 1+\dfrac{L}{2b_{\min }}\right) \dfrac{\Vert \nabla f(x_{k_{i}})\Vert ^{2}}{b_{\min }}+(f(x_{k_{i}})-f_{low})\nonumber \\\le & {} b_{k_{i}}+19{\tilde{L}}+\dfrac{10{\tilde{L}}^{2}}{b_{\min }}+\left( \dfrac{4L^{2}+36b_{\min }^{2}+16b_{\min }L}{2b_{\min }^{3}}\right) \Vert \nabla f(x_{k_{i}})\Vert ^{2}+16(f(x_{k_{i}})-f_{low}). \end{aligned}$$

(4.18)

Thus, combining (4.17), (4.18) and using $\Vert \nabla f(x_{k_{i}})\Vert \le \Vert \nabla f(x_{0})\Vert$ (by Lemma 1), we get

$$\begin{aligned} |I(p,k_{i+1}-1)|\le 4\left[ b_{k_{i}}+19{\tilde{L}}+\dfrac{10{\tilde{L}}^{2}}{b_{\min }}+\left( \dfrac{4L^{2}+36b_{\min }^{2}+16b_{\min }L}{2b_{\min }^{3}}\right) \Vert \nabla f(x_{k_{i}})\Vert ^{2}+16(f(x_{k_{i}})-f_{low})\right] ^{2}\epsilon ^{-2}. \end{aligned}$$

$\square$