Principled analyses and design of first-order methods with inexact proximal operators

Abstract

Proximal operations are among the most common primitives appearing in both practical and theoretical (or high-level) optimization methods. This basic operation typically consists in solving an intermediary (hopefully simpler) optimization problem. In this work, we survey notions of inaccuracies that can be used when solving those intermediary optimization problems. Then, we show that worst-case guarantees for algorithms relying on such inexact proximal operations can be systematically obtained through a generic procedure based on semidefinite programming. This methodology is primarily based on the approach introduced by Drori and Teboulle (Math Program 145(1–2):451–482, 2014) and on convex interpolation results, and allows producing non-improvable worst-case analyses. In other words, for a given algorithm, the methodology generates both worst-case certificates (i.e., proofs) and problem instances on which they are achieved. Relying on this methodology, we study numerical worst-case performances of a few basic methods relying on inexact proximal operations including accelerated variants, and design a variant with optimized worst-case behavior. We further illustrate how to extend the approach to support strongly convex objectives by studying a simple relatively inexact proximal minimization method.

Appendices

A More examples of fixed-step inexact proximal methods

This section extends the list of examples from Sect. 3.1.2.

• The hybrid approximate extragradient algorithm (see [90] or [67, Section 4]) can be described as

  $$\begin{aligned} x_{k+1}=x_k-\eta _{k+1}g_{k+1},\end{aligned}$$

  such that $\exists u_{k+1}, {{\,\mathrm{\textrm{PD}}\,}}_{\eta _{k+1} h}(u_{k+1},g_{k+1};\,x_{k}) \leqslant \tfrac{\sigma ^2}{2}{\Vert u_{k+1}-x_{k}\Vert ^2}$ (see Lemma 1 for a link between $\varepsilon$-subgradient formulation and primal-dual gap). One iteration of this form can be artificially cast into three iterations of (2) as

  $$\begin{aligned}\left\{ \begin{array}{rcl} w_{3k+1} &{}=&{} w_{3k}-e_{3k}\\ w_{3k+2} &{}=&{} w_{3k+1}-e_{3k+1}\\ w_{3k+3} &{}=&{} w_{3k+2} +e_{3k}+e_{3k+1} - \eta _{k+1}v_{3k+2} \\ \end{array}\right. \end{aligned}$$

  with $v_{3k+2} \in \partial h(w_{3k+2})$ . This corresponds to setting $\lambda _{3k+1} = \lambda _{3k+2} =\lambda _{3k+3} = 0$, $\alpha _{3k+3,3k+2}=\eta _{k+1}$, $\beta _{3k+1,3k} = \beta _{3k+2,3k+1}=1$, $\beta _{3k+3,3k+1}=\beta _{3k+3,3k+2} = -1$ and the other parameters to zero. Notice that $w_{3k+3} = w_{3k} - \eta _{k+1}v_{3k+2}$. By requiring ${{\,\mathrm{\textrm{PD}}\,}}_{\eta _{k+1} h}(w_{3k+1},v_{3k+2};\,w_{3k}) \leqslant \tfrac{\sigma ^2}{2}{\Vert w_{3k+1}-w_{3k}\Vert ^2}$ we can identify the primal-dual pair $(u_{k+1},g_{k+1})$ with $(w_{3k+1},v_{3k+2})$ and iterates $x_{k+1}$ with $w_{3k+3}$. In addition, we set

  $$\begin{aligned} \text {EQ}_{3k+1}&=0,\\ \text {EQ}_{3k+2}&=0,\\ \text {EQ}_{3k+3}&= {{\,\mathrm{\textrm{PD}}\,}}_{\eta _{k+1} h}(w_{3k+1},v_{3k+2};\,w_{3k}) - \tfrac{\sigma ^2}{2}{\Vert w_{3k+1}-w_{3k}\Vert ^2}. \end{aligned}$$

  Using $v_{3k+2}\in \partial h (w_{3k+2})$, we have $h^*(v_{3k+2}) = {\langle v_{3k+2}; w_{3k+2}\rangle } - h(w_{3k+2})$ and thus

  $$\begin{aligned} \text {EQ}_{3k+3} =&\,\tfrac{1}{2}{\Vert w_{3k+1}-w_{3k+3}\Vert ^2}+\eta _{k+1}(h(w_{3k+1})-h(w_{3k+2}) \\&- {\langle v_{3k+2}; w_{3k+1}-w_{3k+2}\rangle })-\tfrac{\sigma ^2}{2}{\Vert w_{3k+1}-w_{3k}\Vert ^2}, \end{aligned}$$

  which complies with (3) and is Gram-representable.

• The inexact accelerated proximal point algorithm IAPPA1 in its form from [87, Section 5] can be written as

  $$\begin{aligned}\left\{ \begin{array}{rcl} t_{k+1}&{}=&{} \tfrac{1+\sqrt{1+4t_k^2\tfrac{\eta _{k+1}}{\eta _{k+2}}}}{2} \\ x_{k+1} &{}=&{} y_{k} - \eta _{k+1}(g_{k+1}+r_{k+1}) \\ y_{k+1} &{}=&{} x_{k+1} + \tfrac{t_k-1}{t_{k+1}}(x_{k+1}-x_k) \end{array}\right. \end{aligned}$$

  with $t_0 = 1$, $\{\eta _k\}_k$ a sequence of step sizes, $y_0=x_0\in \mathbb {R}^d$ along with an inexactness criterion of the form ${{\,\mathrm{\textrm{PD}}\,}}_{\eta _{k+1} h}(x_{k+1},g_{k+1};y_k) \leqslant \varepsilon _{k+1}$ given a nonnegative sequence $\{\varepsilon _k\}_k$. Similarly to Güler’s method we get the recursive formulation

  $$\begin{aligned} x_{k+2} = \left( 1+\tfrac{t_k-1}{t_{k+1}}\right) x_{k+1} - \tfrac{t_k-1}{t_{k+1}}x_k - \eta _{k+2}(g_{k+2}+r_{k+2}). \end{aligned}$$

  We consider particular iterations from (2) of the form

  $$\begin{aligned}\left\{ \begin{array}{rcl} w_{2k+1} &{}=&{} w_{2k} - e_{2k} \\ w_{2k+2} &{}=&{} w_{2k+1} -\displaystyle \sum _{i=1}^{2k+1}\alpha _{2k+2,i}v_{i} -\sum _{i=0}^{2k+1}\beta _{2k+2,i}e_i, \end{array}\right. \end{aligned}$$

  with initial iterate $w_0=x_0$. We aim at finding parameters $\alpha _{i,j}$, $\beta _{i,j}$ such that we can identify $\{w_{2k}\}_k$ with $\{x_k\}_k$ (i.e., any sequence $\{x_k\}_k$ can be obtained as a sequence $\{w_{2k}\}_k$). We set $\alpha _{2k+2,2k+1}=\beta _{2k+2,2k+1} = \eta _{k+1}$, $\alpha _{2k+2,i} = \tfrac{t_{k-1}-1}{t_k}\alpha _{2k,i}$ for $i=1,\ldots ,2k-1$ and $\beta _{2k+2,i} = \tfrac{t_{k-1}-1}{t_k}\beta _{2k,i}$ for $i\in \{0,\ldots ,2k-1\}\backslash \{2(k-1)\}$ as well as $\beta _{2k+2,2k} = -1$ and $\beta _{2k+2,2(k-1)} = \tfrac{t_{k-1}-1}{t_k}(1+\beta _{2k,2(k-1)})$.

  This gives

  $$\begin{aligned} w_{2(k+1)} =\,&w_{2k+1} + e_{2k} - \tfrac{t_{k-1}-1}{t_k}(e_{2(k-1)}) -\tfrac{t_{k-1}-1}{t_k}\displaystyle \sum _{i=1}^{2k-1}\alpha _{2k,i}v_{i} \\&-\tfrac{t_{k-1}-1}{t_k}\sum _{i=0}^{2k-1}\beta _{2k,i}e_i - \eta _{k+1}(v_{2k+1}+e_{2k+1})\\ =&\, (1+\tfrac{t_{k-1}-1}{t_k})w_{2k} -\tfrac{t_{k-1}-1}{t_k}w_{2(k-1)} - \eta _{k+1}(v_{2k+1}+e_{2k+1}), \end{aligned}$$

  which shows that $\{w_{2k}\}_k$ follows the same recursive equation as $\{x_{k}\}_k$. In addition, we have $w_0 = x_0$ and $w_2 = x_0 - \eta _{1}(v_1+e_1)$ similar to $x_1 = x_0 -\eta _1(g_1+r_1)$. Requiring ${{\,\mathrm{\textrm{PD}}\,}}_{\eta _{k+1} h}(w_{2k+2},v_{2k+1};\,w_{2k+2}+\eta _{k+1}(v_{2k+1}+e_{2k+1})) \leqslant \varepsilon _{k+1}$ (with the convention $w_{-1}=w_0$) allows to identify the primal-dual pair $(x_{k+1},g_{k+1})$ with $(w_{2k+2},v_{2k+1})$.

  Finally, we can set $\text {EQ}_{2k+2} = {{\,\mathrm{\textrm{PD}}\,}}_{\eta _{k+1} h}(w_{2k+2},v_{2k+1};\,w_{2k+2}+\eta _{k+1}(v_{2k+1}+e_{2k+1})) - \varepsilon _{k+1}$ which is Gram-representable (similar to hybrid approximate extragradient algorithm).

  Note that we can proceed similarly for IAPPA2 from [87, Section 5] with sequence $\{a_k\}_k$ constant equal to 1, by removing the sequence $\{r_k\}_k$ “type 2” errors).

• The accelerated hybrid proximal extragradient algorithm (A-HPE) [68, Section 3] can be written as

  $$\begin{aligned}\left\{ \begin{array}{rcl} a_{k+1} &{}=&{} \tfrac{\eta _{k+1}+\sqrt{\eta _{k+1}^2+4\eta _{k+1}A_k}}{2}\\ A_{k+1} &{}=&{} A_k + a_{k+1}\\ \tilde{x}_k &{}=&{} y_k + \tfrac{a_{k+1}}{A_{k+1}}(x_k-y_k)\\ y_{k+1} &{}=&{} \tilde{x}_k - \eta _{k+1}(g_{k+1}+r_{k+1})\\ x_{k+1} &{}=&{} x_k - a_{k+1}g_{k+1}, \end{array}\right. \end{aligned}$$

  with $A_0=0$, $\{\eta _k\}_k$ a sequence of step sizes, $y_0=x_0\in \mathbb {R}^d$ along with an inexactmess criterion of the form ${{\,\mathrm{\textrm{PD}}\,}}_{\eta _{k+1} h}(y_{k+1},g_{k+1};\tilde{x}_k) \leqslant \tfrac{\sigma }{2}{\Vert y_{k+1}-\tilde{x}_k\Vert ^2}$ given a parameter $\sigma \in [0,1]$. As in the previous examples, we search for a recursive equation followed by the sequence $\{y_k\}_k$. By performing multiple substitutions, we obtain

  $$\begin{aligned} y_{k+2}=&\,\tilde{x}_{k+1} - \eta _{k+2}(g_{k+2}+r_{k+2})\\ =&\,\tfrac{A_{k+1}}{A_{k+2}}y_{k+1} + \tfrac{a_{k+2}}{A_{k+2}}x_{k+1}- \eta _{k+2}(g_{k+2}+r_{k+2})\\ =&\,\tfrac{A_{k+1}}{A_{k+2}}y_{k+1} + \tfrac{a_{k+2}}{A_{k+2}}\left( x_{k}- a_{k+1}g_{k+1}\right) - \eta _{k+2}(g_{k+2}+r_{k+2})\\ =&\,\tfrac{A_{k+1}}{A_{k+2}}y_{k+1} + \tfrac{a_{k+2}}{A_{k+2}}\left( \tfrac{A_{k+1}}{a_{k+1}}\tilde{x}_k - \tfrac{A_k}{a_{k+1}}y_{k} - a_{k+1}g_{k+1}\right) - \eta _{k+2}(g_{k+2}+r_{k+2})\\ =&\,\tfrac{A_{k+1}}{A_{k+2}}y_{k+1} + \tfrac{a_{k+2}}{A_{k+2}}\left( \tfrac{A_{k+1}}{a_{k+1}}\left( y_{k+1}+\eta _{k+1}(g_{k+1}+r_{k+1}) \right) \right. \\&\left. - \tfrac{A_k}{a_{k+1}}y_{k} - a_{k+1}g_{k+1}\right) - \eta _{k+2}(g_{k+2}+r_{k+2})\\ =&\,\left( \tfrac{A_{k+1}}{A_{k+2}}+\tfrac{a_{k+2}A_{k+1}}{A_{k+2}a_{k+1}}\right) y_{k+1}-\tfrac{a_{k+2}A_k}{A_{k+2}a_{k+1}}y_k + \tfrac{a_{k+2}}{A_{k+2}}\left( \tfrac{A_{k+1}}{a_{k+1}}\eta _{k+1} - a_{k+1}\right) g_{k+1}\\&+\tfrac{a_{k+2}A_{k+1}}{A_{k+2}a_{k+1}}\eta _{k+1}r_{k+1}- \eta _{k+2}(g_{k+2}+r_{k+2})\\ =&\,\left( 1+\tfrac{a_{k+2}A_{k}}{A_{k+2}a_{k+1}}\right) y_{k+1}-\tfrac{a_{k+2}A_k}{A_{k+2}a_{k+1}}y_k + \tfrac{a_{k+2}}{A_{k+2}}\left( \tfrac{A_{k+1}}{a_{k+1}}\eta _{k+1} - a_{k+1}\right) g_{k+1}\\&+\tfrac{a_{k+2}A_{k+1}}{A_{k+2}a_{k+1}}\eta _{k+1}r_{k+1}- \eta _{k+2}(g_{k+2}+r_{k+2}). \end{aligned}$$

  Similar to IAPPA1, we consider particular iterations from (2) of the form

  $$\begin{aligned}\left\{ \begin{array}{rcl} w_{2k+1} &{}=&{} w_{2k} - e_{2k} \\ w_{2k+2} &{}=&{} w_{2k+1} -\displaystyle \sum _{i=1}^{2k+1}\alpha _{2k+2,i}v_{i} -\sum _{i=0}^{2k+1}\beta _{2k+2,i}e_i, \end{array}\right. \end{aligned}$$

  with initial iterate $w_0=x_0$. We aim at finding parameters $\alpha _{i,j}$, $\beta _{i,j}$ such that we can identify $\{w_{2k}\}_k$ with $\{y_k\}_k$ (i.e., any sequence $\{y_k\}_k$ can be obtained as a sequence $\{w_{2k}\}_k$). We set $\alpha _{2(k+1),2k+1}=\beta _{2(k+1),2k+1} = \eta _{k+1}$, $\alpha _{2(k+1),i} = \tfrac{a_{k+1}A_{k-1}}{A_{k+1}a_{k}}\alpha _{2k,i}$ for $i\in \{1,\ldots ,2(k-1)\}$ and $\beta _{2(k+1),i} = \tfrac{a_{k+1}A_{k-1}}{A_{k+1}a_{k}}\beta _{2k,i}$ for $i\in \{0,\ldots ,2k-3\}$ as well as $\beta _{2(k+1),2k} = -1$, $\beta _{2(k+1),2k-1} = \tfrac{a_{k+1}}{A_{k+1}a_{k}}(A_{k-1}\beta _{2k,2k-1} -A_k\eta _{k} )$, $\beta _{2(k+1),2(k-1)} = \tfrac{a_{k+1}A_{k-1}}{A_{k+1}a_{k}}(1+\beta _{2k,2(k-1)})$ and $\alpha _{2(k+1),2k-1} = \tfrac{a_{k+1}}{A_{k+1}}\left( \tfrac{A_{k-1}}{a_k}\alpha _{2k,2k-1}-\tfrac{A_{k}}{a_{k}}\eta _{k} + a_{k}\right)$.

  This gives

  $$\begin{aligned} w_{2(k+1)} =&\,w_{2k+1} +e_{2k} +\tfrac{a_{k+1}A_k}{A_{k+1}a_{k}}\eta _{k}e_{2k-1} - \tfrac{a_{k+1}A_{k-1}}{A_{k+1}a_k}e_{2(k-1)}\\ {}&+ \tfrac{a_{k+1}}{A_{k+1}}\left( \tfrac{A_{k}}{a_{k}}\eta _{k} - a_{k}\right) v_{2k-1}-\tfrac{a_{k+1}A_{k-1}}{A_{k+1}a_{k}}\sum _{i=1}^{2k-1}\alpha _{2k,i}v_i \\ {}&- \tfrac{a_{k+1}A_{k-1}}{A_{k+1}a_{k}}\sum _{i=0}^{2k-1}\beta _{2k,i}e_i - \eta _{n+1}(v_{2k+1}+e_{2k+1})\\ =&\,w_{2k} +\tfrac{a_{k+1}A_k}{A_{k+1}a_{k}}\eta _{k}e_{2k-1} - \tfrac{a_{k+1}A_{k-1}}{A_{k+1}a_k}e_{2(k-1)}+ \tfrac{a_{k+1}}{A_{k+1}}\left( \tfrac{A_{k}}{a_{k}}\eta _{k} - a_{k}\right) v_{2k-1}\\ {}&+\tfrac{a_{k+1}A_{k-1}}{A_{k+1}a_{k}}(w_{2k}-w_{2(k-1)}+e_{2(k-1)}) - \eta _{n+1}(v_{2k+1}+e_{2k+1})\\ =&\,\left( 1+\tfrac{a_{k+1}A_{k-1}}{A_{k+1}a_{k}}\right) w_{2k}-\tfrac{a_{k+1}A_{k-1}}{A_{k+1}a_{k}}w_{2(k-1)}+\tfrac{a_{k+1}}{A_{k+1}}\left( \tfrac{A_{k}}{a_{k}}\eta _{k} - a_{k}\right) v_{2k-1}\\&+\tfrac{a_{k+1}A_k}{A_{k+1}a_{k}}\eta _{k}e_{2k-1} - \eta _{n+1}(v_{2k+1}+e_{2k+1}), \end{aligned}$$

  which shows that $\{w_{2k}\}_k$ follows the same recursive equation as $\{y_{k}\}_k$. In addition, we have $w_0 = x_0 = y_0$ and $w_2 = y_0 - \eta _{1}(v_1+e_1)$ similar to $x_1 = x_0 -\eta _1(g_1+r_1)$. Requiring ${{\,\mathrm{\textrm{PD}}\,}}_{\eta _{k+1} h}(w_{2(k+1)},v_{2k+1};\,w_{2k+2}+\eta _{k+1}(v_{2k+1}+e_{2k+1})) \leqslant \tfrac{\sigma ^2}{2}{\Vert w_{2(k+1)}-w_{2k}\Vert ^2}$ allows to identify the primal-dual pair $(y_{k+1},g_{k+1})$ with $(w_{2(k+1)},v_{2k+1})$.

  Finally, we set $\text {EQ}_{2k+2} = {{\,\mathrm{\textrm{PD}}\,}}_{\eta _{k+1} h}(w_{2k+2},v_{2k+1};\,w_{2k+2}+\eta _{k+1}(v_{2k+1}+e_{2k+1})) - \tfrac{\sigma ^2}{2}{\Vert w_{2(k+1)}-w_{2k}\Vert ^2}$ which is Gram-representable (similar to hybrid approximate extragradient algorithm).

B Interpolation with $\varvec{\varepsilon }$-subdifferentials

In this section, we provide the necessary interpolation result for working with $\varepsilon$-subdifferentials inside performance estimation problems.

Theorem B.1

Let I be a finite set of indices and $S=\{(w_i,v_i,h_i,\varepsilon _i)\}_{i\in I}$ with $w_i,v_i\in \mathbb {R}^d$, $h_i,\varepsilon _i\in \mathbb {R}$ for all $i\in I$. There exists $h\in {{\,\mathrm{\mathcal {F}_{0,\infty }}\,}}(\mathbb {R}^d)$ satisfying

$$\begin{aligned} h_i = h(w_i),\text { and } v_i\in \partial _{\varepsilon _i}h(w_i) \text { for all } i\in I \end{aligned}$$

(24)

if and only if

$$\begin{aligned} \begin{aligned} h_i\geqslant h_j +{\langle v_j; w_i-w_j\rangle }-\varepsilon _j \end{aligned} \end{aligned}$$

(25)

holds for all $i,j\in I$.

Proof

$(\Rightarrow )$ Assuming $h\in {{\,\mathrm{\mathcal {F}_{0,\infty }}\,}}$ and (24), the inequalities (25) hold by definition.

$(\Leftarrow )$ Assuming (25) hold, one can perform the following construction:

$$\begin{aligned} \begin{aligned} \tilde{h}(x)=\max _i\{h_i+{\langle v_i; x-w_i\rangle }-\varepsilon _i\}, \end{aligned} \end{aligned}$$

and one can easily check that $h=\tilde{h}\in {{\,\mathrm{\mathcal {F}_{0,\infty }}\,}}$ satisfies (24). $\square$

C Equivalence with Güler’s method

In this section, we show that optimized algorithm (ORI-PPA) and Güler’s second method [49, Section 6] are equivalent (i.e., produce the same iterates), in the case of exact proximal computations (i.e., $\sigma =0$).

We consider a constant sequence of step sizes $\{\lambda _k\}_k$ with $\lambda _k = \lambda >0$. In Güler’s second method, the sequence $\{\beta _k\}_k$ is defined as $\beta _1 = 1$ and

$$\begin{aligned} \beta _{k+1} = \tfrac{1+\sqrt{4\beta _k^2+1}}{2}.\end{aligned}$$

The sequence $\{A_k\}_k$ generated by (ORI-PPA) satisfies $A_0=0$ and

$$\begin{aligned}A_{k+1} = A_k+\tfrac{\lambda +\sqrt{4\lambda A_k + \lambda ^2}}{2},\quad k\geqslant 0. \end{aligned}$$

We can link together these two sequences through the following equality

$$\begin{aligned} \beta _{k} = \tfrac{A_{k}-A_{k-1}}{\lambda }, \quad k\geqslant 1. \end{aligned}$$

(26)

Let us prove it recursively. First, observe that $\beta _1 = 1$ and $\tfrac{A_1-A_0}{\lambda } = 1$. Then assuming that the property is true for some $k \geqslant 1$, we have

$$\begin{aligned} \beta _{k+1}&= \tfrac{1+\sqrt{4\beta _k^2 +1}}{2}\\&= \tfrac{1+\sqrt{4\tfrac{(A_{k+1}-A_k)^2}{\lambda ^2} +1}}{2}. \end{aligned}$$

One might notice that

$$\begin{aligned}(A_{k+1}-A_k)^2 = \tfrac{2\lambda ^2+4\lambda A_k+2\lambda \sqrt{4\lambda A_k + \lambda ^2}}{4}= \lambda A_{k+1},\end{aligned}$$

giving

$$\begin{aligned} \beta _{k+1}&= \tfrac{1+\sqrt{4\tfrac{A_{k+1}}{\lambda } +1}}{2}\\&= \tfrac{\lambda + \sqrt{4\lambda A_{k+1} + \lambda ^2}}{2\lambda }\\&= \tfrac{A_{k+2}-A_{k+1}}{\lambda }, \end{aligned}$$

and we finally arrive to (26). In the case $\sigma =0$, the iterations of (ORI-PPA) can be written as

$$\begin{aligned}\left\{ \begin{array}{ccl} y_k &{}=&{} x_k + \tfrac{\lambda }{A_{k+1}-A_k}(z_k-x_k) \\ x_{k+1} &{}=&{} \textrm{prox}_{\lambda h}(y_k)\\ z_{k+1} &{}=&{} z_k +\tfrac{2(A_{k+1}-A_k)}{\lambda }(x_{k+1}-y_{k}). \end{array}\right. \end{aligned}$$

Therefore, we can write

$$\begin{aligned} y_{k+1}&= x_{k+1} + \tfrac{\lambda }{A_{k+2}-A_{k+1}}\bigg (z_k +\tfrac{2(A_{k+1}-A_k)}{\lambda }(x_{k+1}-y_{k})-x_{k+1}\bigg )\\&=x_{k+1} + \tfrac{\lambda }{A_{k+2}-A_{k+1}}\bigg (x_k - \tfrac{A_{k+1}-A_k}{\lambda }(x_k-y_k) \\ {}&\quad +\tfrac{2(A_{k+1}-A_k)}{\lambda }(x_{k+1}-y_{k})-x_{k+1}\bigg )\\&=x_{k+1} + \tfrac{\lambda }{A_{k+2}-A_{k+1}}\left( \left( \tfrac{A_{k+1}-A_k}{\lambda }-1\right) (x_{k+1}-x_k) +\tfrac{A_{k+1}-A_k}{\lambda }(x_{k+1}-y_{k})\right) . \end{aligned}$$

Combining the last equality with (26) leads to

$$\begin{aligned} y_{k+1} = x_{k+1} + \tfrac{\beta _{k+1}-1}{\beta _{k+2}}(x_{k+1}-x_k) + \tfrac{\beta _{k+1}}{\beta _{k+2}}(x_{k+1}-y_k) \end{aligned}$$

which is exactly the update in Güler’s second method [49, Section 6] modulo a translation in the indices of the $\{y_k\}_k$ sequence (indeed in Güler’s method $y_1=x_0$ whereas in (ORI-PPA) $y_0=x_0$).

D Missing details in Theorem 1

The missing elements in the proof of Theorem 1 are presented bellow.

Proof

Let us rewrite the method in terms of a single sequence, by substitution of $y_k$ and $z_k$:

$$\begin{aligned} \begin{aligned} e_k&{:}{=}\tfrac{1}{\lambda _k}\left( y_{k-1} -\lambda _kg_k - x_k\right) \\ x_{k}&=\tfrac{\lambda _k}{A_{k}-A_{k-1}}\left( x_0-\tfrac{2}{1+\sigma }\sum _{i=1}^{k-1}(A_{i}-A_{i-1}) g_i\right) \\ {}&\quad +\left( 1-\tfrac{\lambda _k}{A_{k}-A_{k-1}}\right) x_{k-1} -\lambda _{k} (g_{k}+e_{k}),\\ \end{aligned} \end{aligned}$$

(27)

and let us state the following identity on the $A_k$ coefficients

$$\begin{aligned} \lambda _{k+1}A_{k+1}=(A_{k+1}-A_k)^2 \text { (for}\, k\geqslant 0\text {)}. \end{aligned}$$

(28)

We prove the desired convergence result by induction. First, for $N=1$

$$\begin{aligned} \begin{aligned} 0 \geqslant&\, \nu _{\star ,1}\Bigg [h(u_1)-h(x_\star ) + {\langle g_1; x_\star -u_1\rangle }\Bigg ] + \nu _{1,1}\Bigg [h(u_1)-h(x_1)+{\langle g_1; x_1-u_1\rangle }\Bigg ] \\&+ \nu _1\Bigg [\tfrac{\lambda _1}{2}{\Vert e_1\Vert ^2}-\tfrac{\lambda _1\sigma ^2}{2}{\Vert e_1+g_1\Vert ^2} + h(x_1) - h(u_1) -{\langle g_1; x_1-u_1\rangle }\Bigg ] \end{aligned} \end{aligned}$$

with $\nu _{\star ,1}= \tfrac{A_1-A_0}{1+\sigma }=\tfrac{A_1}{1+\sigma }$ as $A_0=0$, $\nu _{1,1} = \tfrac{(1-\sigma )A_1}{\sigma (1+\sigma )}$ and $\nu _1 = \tfrac{A_1}{\sigma (1+\sigma )}$. This gives

$$\begin{aligned} 0 \geqslant&\,\tfrac{A_1}{1+\sigma }(h(x_1)-h_\star ) + \tfrac{A_1}{1+\sigma }{\langle g_1; x_\star -x_1\rangle } + \tfrac{A_1}{\sigma (1+\sigma )}\Bigg [\tfrac{\lambda _1}{2}{\Vert e_1\Vert ^2}-\tfrac{\lambda _1\sigma ^2}{2}{\Vert e_1+g_1\Vert ^2}\Bigg ]\\ =&\,\tfrac{A_1}{1+\sigma }(h(x_1)-h_\star ) + \tfrac{A_1}{1+\sigma }{\langle g_1; x_\star -x_0 + \lambda _1(g_1+e_1)\rangle }\\ {}&+ \tfrac{A_1}{\sigma (1+\sigma )}\Bigg [\tfrac{\lambda _1}{2}{\Vert e_1\Vert ^2}-\tfrac{\lambda _1\sigma ^2}{2}{\Vert e_1+g_1\Vert ^2}\Bigg ]\\ =&\,\tfrac{A_1}{1+\sigma }(h(x_1)-h_\star ) + \tfrac{1}{2}{\langle 2\tfrac{A_1}{1+\sigma }g_1; x_\star -x_0\rangle } + {\langle \tfrac{A_1}{1+\sigma }g_1; \lambda _1(g_1+e_1)\rangle }\\ {}&+ \tfrac{A_1}{\sigma (1+\sigma )}\left[ \tfrac{\lambda _1}{2}{\Vert e_1\Vert ^2}-\tfrac{\lambda _1\sigma ^2}{2}{\Vert e_1+g_1\Vert ^2}\right] \\ =&\,\tfrac{A_1}{1+\sigma }(h(x_1)-h_\star ) + \tfrac{1}{4}{\Vert x_\star -x_0+2\tfrac{A_1}{1+\sigma }g_1\Vert ^2} -\tfrac{1}{4}{\Vert x_\star -x_0\Vert ^2} -{\Vert \tfrac{A_1}{1+\sigma }g_1\Vert ^2} \\ {}&+{\langle \tfrac{A_1}{1+\sigma }g_1; \lambda _1(g_1+e_1)\rangle }+ \tfrac{A_1}{\sigma (1+\sigma )}\left[ \tfrac{\lambda _1}{2}{\Vert e_1\Vert ^2}-\tfrac{\lambda _1\sigma ^2}{2}{\Vert e_1+g_1\Vert ^2}\right] \\ =&\,\tfrac{A_1}{1+\sigma }(h(x_1)-h_\star ) + \tfrac{1}{4}{\Vert x_\star -x_0+2\tfrac{A_1}{1+\sigma }g_1\Vert ^2} -\tfrac{1}{4}{\Vert x_\star -x_0\Vert ^2} + \tfrac{A_1\lambda _1(1-\sigma )}{2\sigma }{\Vert e_1\Vert ^2} \\&+ \tfrac{A_1\lambda _1(1-\sigma )}{1+\sigma }{\langle g_1; e_1\rangle } + \tfrac{A_1}{1+\sigma }\left( -\tfrac{A_1}{1+\sigma } + \lambda _1 - \tfrac{\lambda _1\sigma }{2} \right) {\Vert g_1\Vert ^2} \\ =&\,\tfrac{A_1}{1+\sigma }(h(x_1)-h_\star ) + \tfrac{1}{4}{\Vert x_\star -x_0+2\tfrac{A_1}{1+\sigma }g_1\Vert ^2} -\tfrac{1}{4}{\Vert x_\star -x_0\Vert ^2} \\&+ \tfrac{A_1\lambda _1(1-\sigma )}{2\sigma }{\Vert e_1+\tfrac{\sigma }{1+\sigma }g_1\Vert ^2} + \tfrac{A_1}{1+\sigma }\left( -\tfrac{A_1}{1+\sigma } + \lambda _1 - \tfrac{\lambda _1\sigma }{2} -\tfrac{\lambda _1(1-\sigma )\sigma }{2(1+\sigma )}\right) {\Vert g_1\Vert ^2} \\ =&\,\tfrac{A_1}{1+\sigma }(h(x_1)-h_\star ) + \tfrac{1}{4}{\Vert x_\star -x_0+2\tfrac{A_1}{1+\sigma }g_1\Vert ^2} -\tfrac{1}{4}{\Vert x_\star -x_0\Vert ^2} \\&+ \tfrac{A_1\lambda _1(1-\sigma )}{2\sigma }{\Vert e_1+\tfrac{\sigma }{1+\sigma }g_1\Vert ^2} + \tfrac{A_1}{1+\sigma }\left( \tfrac{\lambda _1-A_1}{1+\sigma }\right) {\Vert g_1\Vert ^2} \\ =&\,\tfrac{A_1}{1+\sigma }(h(x_1)-h_\star ) + \tfrac{1}{4}{\Vert x_\star -x_0+2\tfrac{A_1}{1+\sigma }g_1\Vert ^2} -\tfrac{1}{4}{\Vert x_\star -x_0\Vert ^2}\\ {}&+ \tfrac{A_1\lambda _1(1-\sigma )}{2\sigma }{\Vert e_1+\tfrac{\sigma }{1+\sigma }g_1\Vert ^2}, \end{aligned}$$

where we used in the last line that $A_1 = \lambda _1$.

Now, assuming the weighted sum can be reformulated as the desired inequality for $N=k$, that is:

$$\begin{aligned} 0\geqslant&\, \tfrac{A_k}{1+\sigma }(h(x_k)-h_\star ) -\tfrac{1}{4}{\Vert x_\star -x_0\Vert ^2} + \tfrac{1}{4}{\Vert x_\star -x_0+\tfrac{2}{1+\sigma }\sum _{i=1}^{k}(A_{i}-A_{i-1})g_i\Vert ^2} \\&+ \tfrac{(1-\sigma )}{2\sigma }\sum _{i=1}^k A_i\lambda _i{\Vert e_i+\tfrac{\sigma }{1+\sigma }g_i\Vert ^2}, \end{aligned}$$

let us prove it also holds true for $N=k+1$. Noticing that the weighted sum for $k+1$ is exactly the weighted sum for k (which can be reformulated as desired, through our induction hypothesis) with 4 additional inequalities, we get the following valid inequality

$$\begin{aligned} 0\geqslant&\,\tfrac{A_k}{1+\sigma }(h(x_k)-h_\star ) -\tfrac{1}{4}{\Vert x_\star -x_0\Vert ^2} + \tfrac{1}{4}{\Vert x_\star -x_0+\tfrac{2}{1+\sigma }\sum _{i=1}^{k}(A_{i}-A_{i-1})g_i\Vert ^2} \\&+ \tfrac{(1-\sigma )}{2\sigma }\sum _{i=1}^k A_i\lambda _i{\Vert e_i+\tfrac{\sigma }{1+\sigma }g_i\Vert ^2} \\&+ \tfrac{A_{k+1}-A_k}{1+\sigma }\Bigg [h(u_{k+1})-h_\star + {\langle g_{k+1}; x_\star -u_{k+1}\rangle }\Bigg ]\\&+ \tfrac{(1-\sigma )A_{k+1}}{(1+\sigma )\sigma }\Bigg [h(u_{k+1})-h(x_{k+1}) + {\langle g_{k+1}; x_{k+1}-u_{k+1}\rangle }\Bigg ]\\&+ \tfrac{A_k}{1+\sigma }\Bigg [h(u_{k+1})-h(x_{k}) + {\langle g_{k+1}; x_k-u_{k+1}\rangle }\Bigg ]\\&+ \tfrac{A_{k+1}}{(1+\sigma )\sigma }\left[ \tfrac{\lambda _{k+1}}{2}{\Vert e_{k+1}\Vert ^2} - \tfrac{\lambda _{k+1}\sigma ^2}{2}{\Vert e_{k+1}+g_{k+1}\Vert ^2} + h(x_{k+1}) - h(u_{k+1})\right. \\&\left. -{\langle g_{k+1}; x_{k+1}-u_{k+1}\rangle }\right] . \end{aligned}$$

By grouping all function values we get the following simplification:

$$\begin{aligned}&\left[ \tfrac{A_k}{1+\sigma }-\tfrac{A_k}{1+\sigma }\right] h(x_k)+\tfrac{A_{k+1}}{1+\sigma } \left[ \tfrac{1}{\sigma }-\tfrac{1-\sigma }{\sigma }\right] (h(x_{k+1})-h_\star )\\&\quad +\tfrac{1}{1+\sigma }\left[ A_{k+1}-A_k + \tfrac{1-\sigma }{\sigma }A_{k+1} + A_k - \tfrac{1}{\sigma }A_{k+1}\right] h(u_{k+1}) =\tfrac{A_{k+1}}{1+\sigma }(h(x_{k+1})-h_\star ), \end{aligned}$$

where $h(x_k)$ and $h(u_{k+1})$ cancelled out. The remaining inequality is therefore

$$\begin{aligned} 0\geqslant{} & {} \,\tfrac{A_{k+1}}{1+\sigma }(h(x_{k+1})-h_\star ) -\tfrac{1}{4}{\Vert x_0-x_\star \Vert ^2} + \tfrac{1}{4}{\Vert x_\star -x_0+\tfrac{2}{1+\sigma }\sum _{i=1}^k (A_i-A_{i-1}) g_i\Vert ^2}\nonumber \\{} & {} + \tfrac{(1-\sigma )}{2\sigma }\sum _{i=1}^k A_i\lambda _i{\Vert e_i+\tfrac{\sigma }{1+\sigma }g_i\Vert ^2}+ \tfrac{A_{k+1}\lambda _{k+1}}{2(1+\sigma )\sigma }\Bigg [{\Vert e_{k+1}\Vert ^2} - \sigma ^2{\Vert e_{k+1}+g_{k+1}\Vert ^2}\Bigg ] \nonumber \\ {}{} & {} + \tfrac{1}{1+\sigma }{\langle g_{k+1}; (A_{k+1}-A_k)(x_\star -u_{k+1}) - A_{k+1}(x_{k+1}-u_{k+1})+A_k(x_k-u_{k+1})\rangle }\nonumber \\ ={} & {} \, \tfrac{A_{k+1}}{1+\sigma }(h(x_{k+1})-h_\star ) -\tfrac{1}{4}{\Vert x_0-x_\star \Vert ^2} + \tfrac{1}{4}{\Vert x_\star -x_0+\tfrac{2}{1+\sigma }\sum _{i=1}^k (A_i-A_{i-1}) g_i\Vert ^2}\nonumber \\{} & {} + \tfrac{(1-\sigma )}{2\sigma }\sum _{i=1}^k A_i\lambda _i{\Vert e_i+\tfrac{\sigma }{1+\sigma }g_i\Vert ^2}+ \tfrac{A_{k+1}\lambda _{k+1}}{2(1+\sigma )\sigma }\Bigg [{\Vert e_{k+1}\Vert ^2} - \sigma ^2{\Vert e_{k+1}+g_{k+1}\Vert ^2}\Bigg ] \nonumber \\ {}{} & {} + \tfrac{1}{1+\sigma }{\langle g_{k+1}; (A_{k+1}-A_k)x_\star - A_{k+1}x_{k+1} + A_kx_k\rangle }. \end{aligned}$$

(29)

Then, by using (28), one can observe that

$$\begin{aligned} A_{k+1}x_{k+1} =&\, \tfrac{A_{k+1}\lambda _{k+1}}{A_{k+1}-A_{k}}\left( x_0-\tfrac{2}{1+\sigma }\sum _{i=1}^{k}(A_{i}-A_{i-1}) g_i\right) +\left( A_{k+1}-\tfrac{A_{k+1}\lambda _{k+1}}{A_{k+1}-A_{k}}\right) x_{k} \\&-A_{k+1}\lambda _{k+1} (g_{k+1}+e_{k+1})\\ =&\, (A_{k+1}-A_k)\left( x_0-\tfrac{2}{1+\sigma }\sum _{i=1}^{k}(A_{i}-A_{i-1}) g_i\right) + A_kx_k\\&\quad -A_{k+1}\lambda _{k+1} (g_{k+1}+e_{k+1}), \end{aligned}$$

and by re-injecting this inside the last line of (29), we get

$$\begin{aligned} 0\geqslant&\, \tfrac{A_{k+1}}{1+\sigma }(h(x_{k+1})-h_\star ) -\tfrac{1}{4}{\Vert x_0-x_\star \Vert ^2} + \tfrac{1}{4}{\Vert x_\star -x_0+\tfrac{2}{1+\sigma }\sum _{i=1}^k (A_i-A_{i-1}) g_i\Vert ^2}\\&+ \tfrac{(1-\sigma )}{2\sigma }\sum _{i=1}^k A_i\lambda _i{\Vert e_i+\tfrac{\sigma }{1+\sigma }g_i\Vert ^2}+ \tfrac{A_{k+1}\lambda _{k+1}}{2(1+\sigma )\sigma }\Bigg [{\Vert e_{k+1}\Vert ^2} - \sigma ^2{\Vert e_{k+1}+g_{k+1}\Vert ^2}\Bigg ] \\ {}&+ \tfrac{1}{1+\sigma }{\langle (A_{k+1}-A_k)g_{k+1}; x_\star - x_0 +\tfrac{2}{1+\sigma }\sum _{i=1}^{k}(A_{i}-A_{i-1}) g_i \rangle }\\&+ \tfrac{A_{k+1}\lambda _{k+1}}{1+\sigma }{\langle g_{k+1}; (g_{k+1}+e_{k+1})\rangle }. \end{aligned}$$

We can then proceed in a similar manner as in the case $k=1$ for factorizing the quadratic terms,

$$\begin{aligned} 0\geqslant&\, \tfrac{A_{k+1}}{1+\sigma }(h(x_{k+1})-h_\star ) -\tfrac{1}{4}{\Vert x_0-x_\star \Vert ^2} + \tfrac{1}{4}{\Vert x_\star -x_0+\tfrac{2}{1+\sigma }\sum _{i=1}^{k+1} (A_i-A_{i-1}) g_i\Vert ^2}\\&+ \tfrac{(1-\sigma )}{2\sigma }\sum _{i=1}^k A_i\lambda _i{\Vert e_i+\tfrac{\sigma }{1+\sigma }g_i\Vert ^2}+ \tfrac{A_{k+1}\lambda _{k+1}}{2(1+\sigma )\sigma }\Bigg [{\Vert e_{k+1}\Vert ^2} - \sigma ^2{\Vert e_{k+1}+g_{k+1}\Vert ^2}\Bigg ] \\ {}&-\tfrac{(A_{k+1}-A_k)^2}{(1+\sigma )^2}{\Vert g_{k+1}\Vert ^2} + \tfrac{A_{k+1}\lambda _{k+1}}{1+\sigma }{\langle g_{k+1}; (g_{k+1}+e_{k+1})\rangle }\\ =&\, \tfrac{A_{k+1}}{1+\sigma }(h(x_{k+1})-h_\star ) -\tfrac{1}{4}{\Vert x_0-x_\star \Vert ^2} + \tfrac{1}{4}{\Vert x_\star -x_0+\tfrac{2}{1+\sigma }\sum _{i=1}^{k+1} (A_i-A_{i-1}) g_i\Vert ^2}\\&+ \tfrac{(1-\sigma )}{2\sigma }\sum _{i=1}^k A_i\lambda _i{\Vert e_i+\tfrac{\sigma }{1+\sigma }g_i\Vert ^2}+ \tfrac{A_{k+1}\lambda _{k+1}(1-\sigma )}{2\sigma }{\Vert e_{k+1}+\tfrac{\sigma }{1+\sigma }g_{k+1}\Vert ^2} \\ {}&+\left[ \tfrac{A_{k+1}\lambda _{k+1}}{(1+\sigma )}-\tfrac{A_{k+1}\lambda _{k+1}\sigma }{2(1+\sigma )}-\tfrac{(A_{k+1}-A_k)^2}{(1+\sigma )^2} - \tfrac{A_{k+1}\lambda _{k+1}\sigma (1-\sigma )}{2(1+\sigma )^2}\right] {\Vert g_{k+1}\Vert ^2} \\ =&\, \tfrac{A_{k+1}}{1+\sigma }(h(x_{k+1})-h_\star ) -\tfrac{1}{4}{\Vert x_0-x_\star \Vert ^2} + \tfrac{1}{4}{\Vert x_\star -x_0+\tfrac{2}{1+\sigma }\sum _{i=1}^{k+1} (A_i-A_{i-1}) g_i\Vert ^2}\\&+ \tfrac{(1-\sigma )}{2\sigma }\sum _{i=1}^{k+1} A_i\lambda _i{\Vert e_i+\tfrac{\sigma }{1+\sigma }g_i\Vert ^2}+\tfrac{A_{k+1}\lambda _{k+1}}{(1+\sigma )}\Bigg [1-\tfrac{\sigma }{2}-\tfrac{1}{(1+\sigma )} - \tfrac{\sigma (1-\sigma )}{2(1+\sigma )}\Bigg ]{\Vert g_{k+1}\Vert ^2}\\ =&\tfrac{A_{k+1}}{1+\sigma }(h(x_{k+1})-h_\star ) -\tfrac{1}{4}{\Vert x_0-x_\star \Vert ^2} + \tfrac{1}{4}{\Vert x_\star -x_0+\tfrac{2}{1+\sigma }\sum _{i=1}^{k+1} (A_i-A_{i-1}) g_i\Vert ^2}\\&+ \tfrac{(1-\sigma )}{2\sigma }\sum _{i=1}^{k+1} A_i\lambda _i{\Vert e_i+\tfrac{\sigma }{1+\sigma }g_i\Vert ^2}, \end{aligned}$$

since $1-\tfrac{\sigma }{2}-\tfrac{1}{1+\sigma } - \tfrac{\sigma (1-\sigma )}{2(1+\sigma )} = 0$ and this concludes the proof. $\square$

E Tightness of Theorem 1

Proof

The guarantee for (ORI-PPA) provided by Theorem 1 is non-improvable. That is, for all $\{\lambda _k\}_k$ with $\lambda _k>0$, $\sigma \in [0,1]$, $d\in \mathbb {N}$, $x_0\in \mathbb {R}^d$, and $N\in \mathbb {N}$, there exists $f\in \mathcal {F}_{\mu , \infty }(\mathbb {R}^d)$ such that this bound is achieved with equality. For proving this statement, it is sufficient to exhibit a one-dimensional function for which the bound is attained, which is what we do below. The bound is attained on the one-dimensional (constrained on the nonnegative orthant) linear minimization problem

$$\begin{aligned} \min _x\, \{f(x)\triangleq c\, x + i_{\mathbb {R}_+}(x)\}, \end{aligned}$$

(30)

with an appropriate choice of $c> 0$, where $i_{\mathbb {R}_+}$ denotes the convex indicator function of $\mathbb {R}_+$. Indeed, one can check that the relative error criterion

$$\begin{aligned} \exists u_{k}\in \mathbb {R}_+,\;\tfrac{\lambda _k}{2}{\Vert e_k\Vert ^2} + f(x_k)-f(u_k) -{\langle g_k; x_k-u_k\rangle }\leqslant \tfrac{\lambda _k\sigma ^2}{2}{\Vert e_k+g_{k}\Vert ^2} \end{aligned}$$

is satisfied with equality when picking $g_{k}=c$ ($g_k$ is thus a subgradient at $x_k\geqslant 0$), $u_k=x_k$, and $e_{k}=-\tfrac{c\sigma }{1+\sigma }$; and hence $x_{k}=y_{k-1}-\tfrac{c\lambda _k}{1+\sigma }$. The argument is then as follows: if for some $x_0>0$ and $0\leqslant h\leqslant x_0/c$ we manage to show that $x_N=x_0-c h$, then $f(x_N)-f(x_\star )=c(x_0-c h)$ and hence the value of c producing the worst possible (maximal) value of $f(x_N)$ is $c=\tfrac{x_0}{2h}$. In that case, the resulting value is $f(x_N)-f(x_\star )=\tfrac{x_0^2}{4h}$. Therefore, in order to prove that the guarantee from Theorem 1 cannot be improved, we show that $x_N=x_0-\tfrac{A_N}{1+\sigma }c$ on the linear problem (30). It is easy to show that $x_1=x_0-\tfrac{A_1}{1+\sigma }c$ using $A_1=\lambda _1$. The argument follows by induction: assuming $x_{k}=x_0-\tfrac{A_k}{1+\sigma }c$, one can compute

$$\begin{aligned} x_{k+1}= & {} \, \tfrac{\lambda _{k+1}}{A_{k+1}-A_{k}}\left( x_0-\tfrac{2}{1+\sigma }\sum _{i=1}^{k}(A_{i}-A_{i-1}) g_i\right) +\left( 1-\tfrac{\lambda _{k+1}}{A_{k+1}-A_{k}}\right) x_{k} \\ {}{} & {} -\lambda _{k+1} (g_{k+1}+e_{k+1})\\= & {} \, \tfrac{\lambda _{k+1}}{A_{k+1}-A_{k}}\left( x_0-\tfrac{2c}{1+\sigma }A_{k}\right) +\left( 1-\tfrac{\lambda _{k+1}}{A_{k+1}-A_{k}}\right) \left( x_0-\tfrac{A_k}{1+\sigma }c\right) -\lambda _{k+1} \tfrac{c}{1+\sigma }\\= & {} \,x_0-\tfrac{c}{1+\sigma }\tfrac{2\lambda _{k+1}A_k + (A_{k+1}-A_k)A_k - \lambda _{k+1}A_k + \lambda _{k+1}(A_{k+1}-A_k)}{A_{k+1}-A_k}\\= & {} \,x_0-\tfrac{c}{1+\sigma }\tfrac{ (A_{k+1}-A_k)A_k + \lambda _{k+1}A_{k+1}}{A_{k+1}-A_k}\\= & {} \,x_0-\tfrac{c}{1+\sigma }A_{k+1}, \end{aligned}$$

where the second equality follows from simple substitutions, and the last equalities follow from basic algebra and $\lambda _{k+1}A_{k+1} = (A_{k+1}-A_k)^2$. The desired statement is proved by picking $c=\tfrac{(1+\sigma )x_0}{2A_N}$, reaching $f(x_N)-f(x_\star ) = \tfrac{(1+\sigma )x_0^2}{A_N}$. $\square$

F Tightness of Theorem 2

Proof

We show that the guarantee provided in Theorem 2 is non-improvable. That is, for all $\mu \geqslant 0$, $\{\lambda _k\}_k$ with $\lambda _k\geqslant 0$, $\sigma \in [0,1]$, $d\in \mathbb {N}$, $w_0\in \mathbb {R}^d$, and $N\in \mathbb {N}$, there exists $h\in \mathcal {F}_{\mu , \infty }(\mathbb {R}^d)$ such that this bound is achieved with equality. Indeed, the bound is attained on the simple quadratic minimization problem

$$\begin{aligned} \min _x \{h(x) \triangleq \tfrac{\mu }{2}{\Vert x\Vert ^2}\}. \end{aligned}$$

(31)

We can check that the relative error criterion

$$\begin{aligned} \tfrac{\lambda _k}{2}{\Vert e_k\Vert ^2} \leqslant \tfrac{\sigma ^2\lambda _k}{2}{\Vert e_k+v_k\Vert ^2}, \end{aligned}$$

is satisfied with equality when picking $v_k = \nabla h(w_{k+1}) = \mu w_{k+1}$ and $e_k = -\tfrac{\sigma }{1+\sigma }v_k$. Under these choices, one can write

$$\begin{aligned}w_{k+1} = w_k -\tfrac{\lambda _{k+1}\mu }{1+\sigma }w_{k+1}, \end{aligned}$$

which leads to $w_{k+1} = \tfrac{1+\sigma }{1+\sigma + \lambda _{k+1}}w_k$, hence

$$\begin{aligned} w_{N} = \prod _{i=1}^{N}\tfrac{1+\sigma }{1+\sigma +\lambda _{i}\mu }w_0,\end{aligned}$$

and the desired result follows. $\square$