Optimized first-order methods for smooth convex minimization

Abstract

We introduce new optimized first-order methods for smooth unconstrained convex minimization. Drori and Teboulle (Math Program 145(1–2):451–482, 2014. doi:10.1007/s10107-013-0653-0) recently described a numerical method for computing the N-iteration optimal step coefficients in a class of first-order algorithms that includes gradient methods, heavy-ball methods (Polyak in USSR Comput Math Math Phys 4(5):1–17, 1964. doi:10.1016/0041-5553(64)90137-5), and Nesterov’s fast gradient methods (Nesterov in Sov Math Dokl 27(2):372–376, 1983; Math Program 103(1):127–152, 2005. doi:10.1007/s10107-004-0552-5). However, the numerical method in Drori and Teboulle (2014) is computationally expensive for large N, and the corresponding numerically optimized first-order algorithm in Drori and Teboulle (2014) requires impractical memory and computation for large-scale optimization problems. In this paper, we propose optimized first-order algorithms that achieve a convergence bound that is two times smaller than for Nesterov’s fast gradient methods; our bound is found analytically and refines the numerical bound in Drori and Teboulle (2014). Furthermore, the proposed optimized first-order methods have efficient forms that are remarkably similar to Nesterov’s fast gradient methods.

Appendix

1.1 Proof of Lemma 2

We prove that the choice in (6.9), (6.10), (6.11) and (6.12) satisfies the feasible conditions (6.14) of (RD1).

Using the definition of $\varvec{\breve{Q}}(\varvec{r},\varvec{\lambda },\varvec{{\tau }})$ in (6.7), and considering the first two conditions of (6.14), we get

$$\begin{aligned} \lambda _{i+1}&= \breve{Q}_{i,i}(\varvec{r},\varvec{\lambda },\varvec{{\tau }}) = 2\breve{q}_i^2(\varvec{r},\varvec{\lambda },\varvec{{\tau }}) = \frac{1}{2\tau _N^2}\tau _i^2 \\&= {\left\{ \begin{array}{ll} \frac{1}{2(1-\lambda _N)^2}\lambda _1^2, &{} i = 0 \\ \frac{1}{2(1-\lambda _N)^2} (\lambda _{i+1} - \lambda _i)^2, &{} i = 1,\ldots ,N-1, \end{array}\right. } \end{aligned}$$

where the last equality comes from $(\varvec{\lambda },\varvec{{\tau }})\in {\varLambda }$, and this reduces to the following recursion:

$$\begin{aligned} {\left\{ \begin{array}{ll} \lambda _1 = 2(1 - \lambda _N)^2, &{} \\ (\lambda _i - \lambda _{i-1})^2 - \lambda _1\lambda _i = 0. \quad i=2,\ldots ,N. &{} \end{array}\right. } \end{aligned}$$

(10.1)

We use induction to prove that the solution of (10.1) is

$$\begin{aligned} \lambda _i = {\left\{ \begin{array}{ll} \frac{2}{\theta _N^2}, &{} i=1, \\ \theta _{i-1}^2\lambda _1, &{} i=2,\ldots ,N, \end{array}\right. } \end{aligned}$$

which is equivalent to $\varvec{\hat{\lambda }}$ (6.10). It is obvious that $\lambda _1 = \theta _0\lambda _1$, and for $i=2$ in (10.1), we get

$$\begin{aligned} \lambda _2 = \frac{3\lambda _1 + \sqrt{9\lambda _1^2 - 4\lambda _1^2}}{2} = \frac{3+\sqrt{5}}{2}\lambda _1 = \theta _1^2\lambda _1 . \end{aligned}$$

Then, assuming $\lambda _i = \theta _{i-1}^2\lambda _1$ for $i=1,\ldots ,n$ and $n\le N-1$, and using the second equality in (10.1) for $i=n+1$, we get

$$\begin{aligned} \lambda _{n+1}&= \frac{\lambda _1 + 2\lambda _n + \sqrt{(\lambda _1 + 2\lambda _n)^2 - 4\lambda _n^2}}{2} = \frac{1 + 2\theta _{n-1}^2 + \sqrt{1 + 4\theta _{n-1}^2}}{2}\lambda _1 \\&= \left( \theta _{n-1}^2 + \frac{1 + \sqrt{1 + 4\theta _{n-1}^2}}{2}\right) \lambda _1 = \theta _n^2\lambda _1, \end{aligned}$$

where the last equality uses (3.2). Then we use the first equality in (10.1) to find the value of $\lambda _1$ as

$$\begin{aligned}&\lambda _1 = 2(1 - \theta _{N-1}^2\lambda _1)^2 \\&\theta _{N-1}^4\lambda _1^2 - 2\left( \theta _{N-1}^2 + \frac{1}{4}\right) \lambda _1 + 1 = 0 \\&\lambda _1 = \frac{\theta _{N-1}^2 + \frac{1}{4} - \sqrt{\left( \theta _{N-1}^2 + \frac{1}{4}\right) ^2 - \theta _{N-1}^4}}{\theta _{N-1}^4}\\&\quad = \frac{1}{\theta _{N-1}^2 + \frac{1}{4} + \sqrt{\frac{\theta _{N-1}^2}{2} + \frac{1}{16}}} \\&\quad = \frac{8}{\left( 1 + \sqrt{1 + 8\theta _{N-1}^2}\right) ^2} = \frac{2}{\theta _N^2} \end{aligned}$$

with $\theta _N$ in (6.13).

Until now, we derived $\varvec{\hat{\lambda }}$ (6.10) using some conditions of (6.14). Consequently, using the last two conditions in (6.14) with (3.2) and (6.15), we can easily derive the following:

$$\begin{aligned} \tau _i&= {\left\{ \begin{array}{ll} \hat{\lambda } _1 = \frac{2}{\theta _N^2}, &{} i = 0, \\ \hat{\lambda } _{i+1} - \hat{\lambda } _i = \frac{2\theta _i^2}{\theta _N^2} - \frac{2\theta _{i-1}^2}{\theta _N^2} = \frac{2\theta _i}{\theta _N^2}, &{} i=1,\ldots ,N-1, \\ 1 - \hat{\lambda } _N = 1 - \frac{2\theta _{N-1}^2}{\theta _N^2} = \frac{1}{\theta _N}, &{} i = N, \end{array}\right. } \\ \gamma&= \tau _N^2 = \frac{1}{\theta _N^2}, \end{aligned}$$

which are equivalent to $\varvec{{\hat{\tau }}}$ (6.11) and $\hat{\gamma }$ (6.12).

Next, we derive for given $\varvec{\hat{\lambda }}$ (6.10) and $\varvec{{\hat{\tau }}}$ (6.11). Inserting $\varvec{{\hat{\tau }}}$ (6.11) to the first two conditions of (6.14), we get

(10.2)

for $i,k=0,\ldots ,N-1$, and considering (6.5) and (10.2), we get

(10.3)

Finally, using the two equivalent forms (6.2) and (10.3) of , we get

(10.4)

and this can be easily converted to the choice $\hat{r}_{i,k}$ in (6.9).

For these given , we can easily notice that

(10.5)

for $\varvec{\check{\theta }}= \left( \theta _0,\ldots ,\theta _{N-1},\frac{\theta _N}{2}\right) ^\top$, showing that the choice is feasible in both (RD) and (RD1). $\square$

1.2 Proof of (8.2)

We prove that (8.2) holds for the coefficients $\varvec{\hat{h}}$ (7.1) of OGM1 and OGM2.

We first show the following property using induction:

$$\begin{aligned} \sum _{k=0}^{j-1}\hat{h}_{j,k} = {\left\{ \begin{array}{ll} \theta _j, &{} j = 1,\ldots ,N-1, \\ \frac{1}{2}(\theta _N+1), &{} j = N. \end{array}\right. } \end{aligned}$$

Clearly, $\hat{h}_{1,0} = 1 + \frac{2\theta _0 - 1}{\theta _1} = \theta _1$ using (3.2). Assuming $\sum _{k=0}^{j-1}\hat{h}_{j,k} = \theta _j$ for $j=1,\ldots ,n$ and $n\le N-1$, we get

$$\begin{aligned} \sum _{k=0}^n\hat{h}_{n+1,k}&= 1 + \frac{2\theta _n - 1}{\theta _{n+1}} + \frac{\theta _n - 1}{\theta _{n+1}}(\hat{h}_{n,n-1}- 1) + \frac{\theta _n - 1}{\theta _{n+1}}\sum _{k=0}^{n-2}\hat{h}_{n,k} \\&= 1 + \frac{\theta _n}{\theta _{n+1}} + \frac{\theta _n - 1}{\theta _{n+1}}\sum _{k=0}^{n-1}\hat{h}_{n,k} = \frac{\theta _{n+1} + \theta _n^2}{\theta _{n+1}} \\&= {\left\{ \begin{array}{ll} \theta _n, &{} n = 1,\ldots ,N-2, \\ \frac{1}{2}(\theta _N + 1), &{} n = N-1, \end{array}\right. } \end{aligned}$$

where the last equality uses (3.2) and (6.15).

Then, (8.2) can be easily derived using (3.2) and (6.15) as

$$\begin{aligned} \sum _{j=1}^i\sum _{k=0}^{j-1}\hat{h}_{j,k}&= {\left\{ \begin{array}{ll} \sum \nolimits _{j=1}^i\theta _j, &{} i = 1,\ldots ,N-1, \\ \sum \nolimits _{j=1}^{N-1}\theta _j + \frac{1}{2}(\theta _N+1), &{} i = N, \end{array}\right. } \\&= {\left\{ \begin{array}{ll} \theta _i^2 - 1, &{} i = 1,\ldots ,N-1, \\ \frac{1}{2}(\theta _N^2 - 1), &{} i = N. \end{array}\right. } \end{aligned}$$

$\square$