A piecewise conservative method for unconstrained convex optimization

Abstract

We consider a continuous-time optimization method based on a dynamical system, where a massive particle starting at rest moves in the conservative force field generated by the objective function, without any kind of friction. We formulate a restart criterion based on the mean dissipation of the kinetic energy, and we prove a global convergence result for strongly-convex functions. Using the Symplectic Euler discretization scheme, we obtain an iterative optimization algorithm. We have considered a discrete mean dissipation restart scheme, but we have also introduced a new restart procedure based on ensuring at each iteration a decrease of the objective function greater than the one achieved by a step of the classical gradient method. For the discrete conservative algorithm, this last restart criterion is capable of guaranteeing a qualitative convergence result. We apply the same restart scheme to the Nesterov Accelerated Gradient (NAG-C), and we use this restarted NAG-C as benchmark in the numerical experiments. In the smooth convex problems considered, our method shows a faster convergence rate than the restarted NAG-C. We propose an extension of our discrete conservative algorithm to composite optimization: in the numerical tests involving non-strongly convex functions with $\ell ^1$-regularization, it has better performances than the well known efficient Fast Iterative Shrinkage-Thresholding Algorithm, accelerated with an adaptive restart scheme.

Appendices

Appendix 1: Proof of Proposition 1

Proof

Let us prove that the function $t\mapsto E_K(t)$ has a local maximum in $[0,+\infty )$. By contradiction, if $t\mapsto E_K(t)$ has no local maxima, then $t\mapsto E_K(t)$ is injective (otherwise we can apply twice Weierstrass Theorem and we can find a local maximum). Since $t\mapsto E_K(t)$ is continuous, it has to be strictly increasing. This implies that $t \mapsto \dot{x}(t)$ can not change sign and hence that it is monotone as well. Moreover, it follows that $t \mapsto x(t)$ is monotone as well. Since both x(t) and $\dot{x}(t)$ remain bounded for every $t \in [0, +\infty )$, there exist $x_\infty , v_\infty \in \mathbb {R}$ such that

$$\begin{aligned} \lim _{t \rightarrow +\infty }x(t) = x_{\infty } \,\,\,\,\, \text{ and } \,\, \lim _{t \rightarrow +\infty }\dot{x}(t) = v_{\infty }. \end{aligned}$$

On the other hand, $v_{\infty }$ should be zero, and this is a contradiction.

Let $\bar{t}$ be a point of local maximum for the kinetic energy function $t \mapsto E_K(t)$. This implies that $|\dot{x}(\bar{t})| > 0$. The conservation of the total mechanical energy ensures that the function $t\mapsto f(x(t))$ attains a local minimum at $\bar{t}$. Using the Implicit Function Theorem, we obtain that $t \mapsto x(t)$ is a local homeomorphism around $\bar{t}$. This implies that $x(\bar{t})$ is a point of local minimum for f. $\square$

Appendix 2: Proof of Proposition 2

Proof

Without loss of generality, we can assume that $x^*=0$ and that $x_0>0$. We define a strongly convex function $g: \mathbb {R}\mapsto \mathbb {R}$ as follows:

$$\begin{aligned} g(x) := {\frac{1}{2} \mu |x-x^*|^2=} \frac{1}{2} \mu |x|^2. \end{aligned}$$

We claim that, for every $y \in [0,x_0]$, the following inequality is satisfied:

$$\begin{aligned} f(x_0) - f(y) \ge g(x_0) - g(y). \end{aligned}$$

(59)

Indeed, we have that

$$\begin{aligned} f(x_0) - g(x_0)&= f(y) - g(y) + \int _y ^{x_0}(f'(u) - g'(u))\,du \ge f(y) - g(y), \end{aligned}$$

since $f'(u) - g'(u)\ge 0$ for every $u\ge 0$. Combining (5) and (59) we obtain:

$$\begin{aligned} t_1 = \int _0^{x_0}\frac{1}{\sqrt{2(f(x_0)-f(y))}}dy \le&\int _0^{x_0}\frac{1}{\sqrt{2(g(x_0)-g(y))}}dy\\&= \int _0^{x_0}\frac{1}{\sqrt{\mu (x_0^2- y^2)}}dy = \frac{\pi }{2\sqrt{\mu }}. \end{aligned}$$

This completes the proof. $\square$

Appendix 3: Proof of Lemma 1

Proof

Up to a linear orthonormal change of coordinates, we can assume that the function f is of the form

$$\begin{aligned} f(x) = \sum _{i=1}^n {\lambda }_i \frac{x_i^2}{2}. \end{aligned}$$

Hence, the differential system (7) becomes

$$\begin{aligned} {\left\{ \begin{array}{ll} \ddot{x}_1 + {\lambda }_1 x_1 = 0, \\ \vdots \\ \ddot{x}_n + {\lambda }_n x_n = 0, \end{array}\right. } \end{aligned}$$

i.e., the components evolve independently one of each other. If the Cauchy datum is

$$\begin{aligned} x(0) = (x_{1,0}, \, \ldots \, x_{n,0} ) \,\,\, \text{ and } \,\, \dot{x}(0)=0, \end{aligned}$$

then we can compute the expression of the kinetic energy function $E_K: t\mapsto \frac{1}{2} |\dot{x}(t)|^2$:

$$\begin{aligned} E_K(t) = \sum _{i=1}^n {\lambda }_i \frac{x_{i,0}^2}{2} {\sin ^2( \sqrt{{\lambda }_i} t)}. \end{aligned}$$

For every $0 \le t\le \frac{\pi }{2\sqrt{{\lambda }_n}}$, we have that

$$\begin{aligned} 0 \le \sin ( \sqrt{{\lambda }_1} t )\le \cdots \le \sin ( \sqrt{{\lambda }_n} t ), \end{aligned}$$

and then we deduce that

$$\begin{aligned} E_K(t) \ge \left( \sum _{i=1}^n {\lambda }_i\frac{x_{i,0}^2}{2} \right) {\sin ^2(\sqrt{{\lambda }_1} t)}, \end{aligned}$$

for every $t \in [ 0, {\pi }/{(2\sqrt{{\lambda }_n})} ]$. Evaluating the last inequality for $t = \frac{\pi }{2\sqrt{{\lambda }_n}}$ and using the conservation of the energy, we obtain the thesis. $\square$