Hierarchical Convex Optimization by the Hybrid Steepest Descent Method with Proximal Splitting Operators—Enhancements of SVM and Lasso

Abstract

The breakthrough ideas in the modern proximal splitting methodologies allow us to express the set of all minimizers of a superposition of multiple nonsmooth convex functions as the fixed point set of computable nonexpansive operators. In this paper, we present practical algorithmic strategies for the hierarchical convex optimization problems which require further strategic selection of a most desirable vector from the solution set of the standard convex optimization. The proposed algorithms are established by applying the hybrid steepest descent method to special nonexpansive operators designed through the art of proximal splitting. We also present applications of the proposed strategies to certain unexplored hierarchical enhancements of the support vector machine and the Lasso estimator.

Appendices

Appendices

16.1.1 A: Proof of Proposition 16.9(a)

Fact 16.5(i)⇔(ii) in Section 16.2.1 yields

The remaining follows from the proof in [40, Proposition 18]. □

16.1.2 B: Proof of Proposition 16.10(a)(d)

(a) From (16.58) and (16.59), there exists $(x_{\star },\nu _{\star }) \in \mathcal {S}_{\text{pLAL}} \times \mathcal {S}_{\text{dLAL}}$. Fact 16.5(i)⇔(ii) in Section 16.2.1 yields the equivalence

(16.175)

(16.176)

(16.177)

□

(d) Choose arbitrarily $(\bar {x},\bar {\nu }) \in \operatorname {Fix}(T_{\text{LAL}})$, i.e.,

Let $(x_n,\nu _n)_{n \in \mathbb {N}} \subset \mathcal {X} \times \mathcal {K}$ be generated, with any $(x_0,\nu _0) \in \mathcal {X} \times \mathcal {K}$, by

(16.178)

Then [150, (B.3)] yields

(16.179)

Equation (16.179) and ∥A∥_op < 1 imply that $(\|x_n-\bar {x} \|{ }_{\mathcal {X}}^2 +\|\nu _n-\bar {\nu } \|{ }_{\mathcal {K}}^2)_{n \in \mathbb {N}}$ decreases monotonically, i.e., $(x_{n},\nu _n)_{n \in \mathbb {N}}$ is Fejér monotone with respect to $\operatorname {Fix}(T_{\text{LAL}})$, and $(\|x_n-\bar {x} \|{ }_{\mathcal {X}}^2 +\|\nu _n-\bar {\nu } \|{ }_{\mathcal {K}}^2)_{n \in \mathbb {N}}$ converges to some c ≥ 0. From this observation, we have

and thus

(16.180)

By [51, Theorem 9.12], the bounded sequence of $(x_n,\nu _n)_{n \in \mathbb {N}}$ has some subsequence $(x_{n_j},\nu _{n_j})_{j \in \mathbb {N}}$ which converges weakly to some point, say (x _⋆, ν _⋆), in the Hilbert space $\mathcal {X} \times \mathcal {K}$. Therefore, by applying [9, Theorem 9.1(iii)⇔(i)] to $f \in \varGamma _0(\mathcal {X})$, we have

(16.181)

and, by the Cauchy-Schwarz inequality and (16.180),

which implies Ax _⋆ = 0.

Meanwhile, by (16.178), we have

(16.182)

where the inner product therein satisfies

(16.183)

which is verified by Ax _⋆ = 0, the triangle inequality, the Cauchy-Schwarz inequality, and (16.180), as follows:

Now, by (16.182), (16.181), and (16.183), we have for any $x \in \mathcal {X}$

which implies

$$\displaystyle \begin{aligned} A^{*} \nu_{\star} \in \partial f(x_{\star}). \end{aligned} $$

(16.184)

By recalling (16.176)⇔(16.177), (16.184) and Ax _⋆ = 0 prove $(x_{\star }, \nu _{\star }) \in \operatorname {Fix}(T_{\text{LAL}})$. The above discussion implies that every weak sequential cluster point (see Footnote 7 in Section 16.2.2) of $(x_n,\nu _n)_{n \in \mathbb {N}}$, which is Fejér monotone with respect to $\operatorname {Fix}(T_{\text{LAL}})$, belongs to $\operatorname {Fix}(T_{\text{LAL}})$. Therefore, [9, Theorem 5.5] guarantees that $(x_n,\nu _n)_{n \in \mathbb {N}}$ converges weakly to a point in $\operatorname {Fix}(T_{\text{LAL}})$. □

16.1.3 C: Proof of Theorem 16.15

Now by recalling Proposition 16.9 in Section 16.2.3 and Remark 16.16 in Section 16.3.1, it is sufficient to prove Claim 16.15. Let $x_{\star } \in \mathcal {S}_p \neq \varnothing$. Then the Fermat’s rule, Fact 16.4(b) (applicable due to the qualification condition (16.40)) in Section 16.2.1, $\check {A}^*\colon \mathcal {K} \to \mathcal {X} \times \mathcal {K}\colon \nu \mapsto (A^*\nu , -\nu )$ for $\check {A}$ in (16.74), the property of ι _{0} in (16.35), the straightforward calculations, and Fact 16.5(ii)⇔(i) (in Section 16.2.1) yield

which confirms Claim 16.15. □

16.1.4 D: Proof of Theorem 16.17

Now by recalling Proposition 16.9 in Section 16.2.3 and Remark 16.18 in Section 16.3.1, it is sufficient to prove (16.97) by verifying Claim 16.17. We will use

$$\displaystyle \begin{aligned} A^* \circ \partial g \circ A=\sum_{i=1}^mA_i^* \circ \partial g_i \circ A_i = \sum_{i=1}^m\partial (g_i\circ A_i) \end{aligned} $$

(16.185)

which is verified by $g=\bigoplus _{i=1}^mg_i$, Fact 16.4(c) (see Section 16.2.1), and $\operatorname {ri}(\operatorname {dom}(g_j) - \operatorname {ran}(A_j))=\operatorname {ri}(\operatorname {dom}(g_j) - \mathbb {R})=\mathbb {R} \ni 0$ (j = 1, 2, …, m). Let $x_{\star }^{(m+1)} \in \mathcal {S}_p \neq \varnothing$. Then by using the Fermat’s rule, Fact 16.4(b) (applicable due to (16.40)), (16.185), D in (16.93), and H in (16.92), we deduce the equivalence

(16.186)

Then by $-\begin {pmatrix} \nu ^{(1)}, \ldots , \nu ^{(m)}, -\sum _{i=1}^m \nu ^{(i)} \end {pmatrix} \in D^{\perp }=\partial \iota _{D}(x_{\star }^{(1)},\ldots ,x_{\star }^{(m+1)})$ (see (16.34) ) and by Fact 16.5(ii)⇔(i) in Section 16.2.1, we have

which confirms Claim 16.17. □

16.1.5 E: Proof of Theorem 16.19

Now by recalling Proposition 16.10 in Section 16.2.3 and Remark 16.20 in Section 16.3.2, it is sufficient to prove Claim 16.19. Let $x_{\star } \in \mathcal {S}_p \neq \varnothing$. Then the Fermat’s rule, Fact 16.4(b) (applicable due to (16.40)) in Section 16.2.1, $\check {A}^*\colon \mathcal {K} \to \mathcal {X} \times \mathcal {K}\colon \nu \mapsto (A^*\nu , -\nu )$ for $\check {A}$ in (16.74), the property of ι _{0} in (16.35), the straightforward calculations, and Fact 16.5(ii)⇔(i) (in Section 16.2.1) yield

which confirms Claim 16.19. □

16.1.6 F: Proof of Theorem 16.23

• (a) We have seen in (16.78) that, under the assumptions of Theorem 16.23(a), for any vector $x_{\star } \in \mathcal {X}$,

  

  (16.187)

  for some $y_{\star } \in \mathcal {X}$ and some ${\mathbf \zeta }_{\star } \in \operatorname {Fix}\left ({\mathbf T}_{\text{DRS}_{\text{I}}}\right )$, where $\check {A}\colon \mathcal {X} \times \mathcal {K} \to \mathcal {K}\colon (x,y) \mapsto Ax-y$ (see (16.74)), $\mathcal {N}(\check {A})=\{(x,Ax) \in \mathcal {X} \times \mathcal {K}\mid x \in \mathcal {X} \}$, and ${\mathbf T}_{\text{DRS}_{\text{I}}}=(2\operatorname {prox}_F -\text{I}) \circ (2{P}_{\mathcal {N}(\check {A})} -\text{I})$ for $F\colon \mathcal {X} \times \mathcal {K} \to (-\infty ,\infty ]\colon (x,y)\mapsto f(x)+g(y)$ (see (16.71) and (16.73)).

  Choose ${\mathbf \zeta }_{\star }:=(\zeta ^x_{\star }, \zeta ^y_{\star }) \in \operatorname {Fix}\left ({\mathbf T}_{\text{DRS}_{\text{I}}}\right )$ arbitrarily and let ${\mathbf z}_{\star }:=(x_{\star }, y_{\star }) := {P}_{\mathcal {N}(\check {A})}({\mathbf \zeta }_{\star })$. Then we have

  

  (16.188)
  

  (16.189)
  

  (16.190)

  Meanwhile, we have

  

  (16.191)

  Equations (16.191) and (16.190) imply

  

  (16.192)

  Moreover, by noting that (16.187) ensures $x_{\star } \in \mathcal {S}_p$ and y _⋆ = Ax _⋆, we have from (16.192)

  

  Since ζ _⋆ is chosen arbitrarily from $\operatorname {Fix}\left ({\mathbf T}_{\text{DRS}_{\text{I}}}\right )$, we have

  

  (16.193)

  from which Theorem 16.23(a) is confirmed.

• (b) We have seen in (16.113) that, under the assumptions of Theorem 16.23(b), for any vector $x_{\star } \in \mathcal {X}$,

  

  (16.194)

  for some $(y_{\star },\nu _{\star }) \in \mathcal {K} \times \mathcal {K}$, where

  

  and $(\mathfrak {u} \check {A})^*\colon \mathcal {K} \to \mathcal {X} \times \mathcal {K}\colon \nu \mapsto (\mathfrak {u}A^* \nu , -\mathfrak {u} \nu )$ (see (16.108) and (16.120)).

  Choose $({\mathbf z}_{\star }, \nu _{\star }) \in \operatorname {Fix}({\mathbf T}_{\text{LAL}})$ arbitrarily and denote ${\mathbf z}_{\star }=(x_{\star },y_{\star }) \in \mathcal {X} \times \mathcal {K}$. By passing similar steps in (16.177)⇔(16.176), we deduce

  

  (16.195)

  and then, from (16.195), straightforward calculations yield

  

  (16.196)

  Moreover, by noting that (16.194), we have from (16.196)

  

  Since (x _⋆, y _⋆, ν _⋆) is chosen arbitrarily from $\operatorname {Fix}({\mathbf T}_{\text{LAL}})$, we have

  

  from which Theorem 16.23(b) is confirmed.

• (c) We have seen in (16.98) that, under the assumptions of Theorem 16.23(c), for any vector $x_{\star } \in \mathcal {X}$,

  

  (16.197)

  for some $\mathfrak {X}_{\star } \in \operatorname {Fix}\left ({\mathbf T}_{\text{DRS}_{\text{II}}}\right )$, where $D= \{(x^{(1)},\ldots ,x^{(m+1)}) \in \mathcal {X}^{m+1} \mid x^{(i)}=x^{(j)} \ (i,j =1,2,\ldots , m+1) \}$ (see (16.93)), $H\colon \mathcal {X}^{m+1} \to (-\infty ,\infty ]\colon (x^{(1)},\ldots ,x^{(m+1)}) \mapsto \sum _{i=1}^m g_i(A_ix^{(i)})+f(x^{(m+1)})$ (see (16.92)), and ${\mathbf T}_{\text{DRS}_{\text{II}}}=(2\operatorname {prox}_H -\text{I}) \circ (2{P}_{D} -\text{I})$ (see (16.90)) [For the availability of $\operatorname {prox}_H$ and P _D as computational tools, see Remark 16.18(a)].

  Choose $\mathfrak {X}_{\star }:=(\zeta _{\star }^{(1)},\ldots , \zeta _{\star }^{(m+1)}) \in \operatorname {Fix}\left ({\mathbf T}_{\text{DRS}_{\text{II}}}\right )$ arbitrarily, and let ${\mathbf X}_{\star }:=(x_{\star }, \ldots , x_{\star }) = {P}_D(\mathfrak {X}_{\star })$. Then we have

  

  (16.198)

  Now, by passing similar steps for (16.188)⇒(16.189), we deduce that

  

  (16.199)

  where the last equivalence follows from Fact 16.4(c) (applicable due to $\operatorname {ri}(\operatorname {dom}(g_j) - \operatorname {ran}(A_j))=\operatorname {ri}(\operatorname {dom}(g_j) - \mathbb {R})=\mathbb {R} \ni 0$). Meanwhile, we have

  

  (16.200)

  Equations (16.200) and (16.199) imply

  

  (16.201)

  Moreover, by noting that (16.197) ensures $x_{\star } \in \mathcal {S}_p$, we have from (16.201)

  

  Since $\mathfrak {X}_{\star }$ is chosen arbitrarily from $\operatorname {Fix}({\mathbf T}_{\text{DRS}_{\text{II}}})$, we have

  

  (16.202)

  from which Theorem 16.23(c) is confirmed. □

16.1.7 G: Proof of Lemma 16.27

Obviously, we have from (16.158)

(16.203)

By recalling $0 \neq {\mathbf x}_{j} \in \mathbb {R}^{N}\text{ in (16.153)}$ and ${\mathbf M}_{j} \in \mathbb {R}^{(N+1) \times p}$ in (16.159), we have

and therefore

(16.204)

To prove $\operatorname {dom}(g_{(j,q)}) - {\mathbf M}_j\operatorname {dom}(\|\cdot \|{ }_1)=\mathbb {R} \times \mathbb {R}^N$, choose arbitrarily $(\eta , {\mathbf y}) \in \mathbb {R} \times \mathbb {R}^N$. Then (16.203) and (16.204) guarantee

implying thus

(16.205)

□

16.1.8 H: Proof of Theorem 16.28

By recalling Remark 16.29 in Section 16.5.2, it is sufficient to prove Claim 16.28, for which we use the following inequality: for each j = 1, 2, …, 2p,

(16.206)

where ${\mathbf x}_j \in \mathbb {R}^{N}$ in (16.153) and ${\mathbf M}_j \in \mathbb {R}^{(N+1) \times p}$ in (16.159). Equation (16.206) is confirmed by

(16.207)

and

Let $U_S:=\sup \{\|{\mathbf b}\| \mid {\mathbf b} \in S\}(<\infty )$. By supercoercivity of φ and Example 16.3, the subdifferential of its perspective $\widetilde {\varphi }$ at each $(\eta , {\mathbf y}) \in \mathbb {R} \times \mathbb {R}^N$ can be expressed as (16.32), and thus, to prove Claim 16.28, it is sufficient to show

Proof of (i)

Choose $(\eta ,{\mathbf y}) \in \mathbb {R}_{++} \times \mathbb {R}^{N}$ arbitrarily. Then, from (16.32), every ${\mathbf c}_{(\eta ,{\mathbf y})} \in ({\mathbf M}_j^{\top })^{-1}(S) \cap \partial \widetilde {\varphi }(\eta ,{\mathbf y}) \subset \mathbb {R} \times \mathbb {R}^{N}$ can be expressed with some u ∈ ∂φ(y∕η) as

$$\displaystyle \begin{aligned} {\mathbf c}_{(\eta,{\mathbf y})}=(\varphi({\mathbf y}/\eta) - \langle {\mathbf y}/\eta,{\mathbf u}\rangle, {\mathbf u})=(-\varphi^*({\mathbf u}), {\mathbf u}), \end{aligned} $$

(16.208)

where the last equality follows from φ(y∕η) + φ ^∗(u) = 〈y∕η, u〉 due to the Fenchel-Young identity (16.30). By ${\mathbf M}_j^{\top }{\mathbf c}_{(\eta ,{\mathbf y})} \in S$ and by applying the inequality (16.206) to (16.208), we have

(16.209)

where and are coercive convex functions (see Section 16.2.1) and independent from the choice of (η, y). The coercivity of ensures the existence of an open ball $B(0,\hat {U}_{\text{(i)}})$ of radius $\hat {U}_{\text{(i)}}>0$ such that , and thus (16.209) implies

(16.210)

Moreover, by x _j ≠ 0, the triangle inequality, the Cauchy-Schwarz inequality, (16.209), and (16.210), we have

(16.211)

which yields ${\mathbf c}_{(\eta ,{\mathbf y})}=(- \varphi ^*({\mathbf u}), {\mathbf u}) \in [-U_{\text{(i)}}, {U}_{\text{(i)}}] \times B(0,\hat {U}_{\text{(i)}} )$. Since $(\eta ,{\mathbf y})\in \mathbb {R}_{++} \times \mathbb {R}^{N}$ is chosen arbitrarily and ${\mathbf c}_{(\eta ,{\mathbf y})} \in ({\mathbf M}_j^{\top })^{-1}(S) \cap \partial \widetilde {\varphi }(\eta ,{\mathbf y})$ is also chosen arbitrarily, we have

which confirms the statement (i).

Proof of (ii)

By introducing

$$\displaystyle \begin{aligned} \mathfrak{B}:= \left\{{\mathbf v} \in \mathbb{R}^N \left| \ \left|\left\langle \frac{2}{\|{\mathbf x}_j\|{}^2}{\mathbf x}_j, {\mathbf v} \right\rangle\right| > |\varphi^*({\mathbf v})|\right. \right\}, \end{aligned} $$

(16.212)

we can decompose the set $({\mathbf M}_j^{\top })^{-1}(S) \cap \partial \widetilde {\varphi }(0,0)$ into

(16.213)

In the following, we show the boundedness of each set in (16.213).

First, we show the boundedness of $\mathfrak {B}$ by contradiction. Suppose that $\mathfrak {B} \not \subset B(0,r)$ for all r > 0. Then there exists a sequence $({\mathbf u}_k)_{k \in \mathbb {N}} \subset \mathbb {R}^N$ such that

(16.214)

which contradicts the supercoercivity of φ ^∗, implying thus the existence of r _∗ > 0 such that $\mathfrak {B} \subset B(0,r_*)$.

Next, we show the boundedness of the former set in (16.213). Choose arbitrarily

(16.215)

By x _j ≠ 0, ${\mathbf M}_j^{\top } (\mu ,{\mathbf u}^{\top })^{\top } \in S \subset B(0,U_S)$, the inequality (16.206), the triangle inequality, the Cauchy-Schwarz inequality, and ${\mathbf u} \in \mathfrak {B} \subset B(0,r_*)$, we have

which yields

Therefore, we have $(\mu ,{\mathbf u}) \in [-\hat {U}_{\text{(iia)}}, \hat {U}_{\text{(iia)}}] \times B(0,r_{\star })$. Since $(\mu ,{\mathbf u}) \in ({\mathbf M}_j^{\top })^{-1}(S) \cap \partial \widetilde {\varphi }(0,0) \cap (\mathbb {R} \times \mathfrak {B})$ is chosen arbitrarily, we have

(16.216)

Finally, we show the boundedness of the latter set in (16.213). Let

(16.217)

From (16.32), we have

(16.218)

Note that coercivity of φ ^∗ ($\Rightarrow \exists \min \varphi ^*(\mathbb {R}^N) \in \mathbb {R}$, see Fact 16.2) and (16.218) yield $\varphi ^*({\mathbf u}) \in [\min \varphi ^*(\mathbb {R}^N), -\mu ]$ and thus

$$\displaystyle \begin{aligned} |\varphi^*({\mathbf u})| \leq \max\{|\min \varphi^*(\mathbb{R}^N)|, |\mu| \} \leq |\min \varphi^*(\mathbb{R}^N)|+ |\mu|. \end{aligned} $$

(16.219)

By x _j ≠ 0, ${\mathbf M}_j^{\top } (\mu ,{\mathbf u}^{\top })^{\top } \in S \subset B(0,U_S)$ (see (16.217)), the inequality (16.206), the triangle inequality, ${\mathbf u} \in \mathfrak {B}^c$ (see (16.217) and (16.212)), and (16.219), we have

and thus, with (16.219),

(16.220)

Hence, we have

Since $(\mu ,{\mathbf u}) \in ({\mathbf M}_j^{\top })^{-1}(S) \cap \partial \widetilde {\varphi }(0,0)\cap (\mathbb {R} \times \mathfrak {B}^c)$ is chosen arbitrarily, we have

(16.221)

Consequently, by using (16.216) and (16.221) and by letting $U_{\text{(ii)}}:=\max \{\hat {U}_{\text{(iia)}},\hat {U}_{\text{(iib)}} \}$, we have

which guarantees the boundedness of $({\mathbf M}_j^{\top })^{-1}(S) \cap \partial \widetilde {\varphi }(0,0)$, due to the coercivity of φ ^∗, implying thus finally the statement (ii).

□