A distributed algorithm for high-dimension convex quadratically constrained quadratic programs

Abstract

We propose a Jacobi-style distributed algorithm to solve convex, quadratically constrained quadratic programs (QCQPs), which arise from a broad range of applications. While small to medium-sized convex QCQPs can be solved efficiently by interior-point algorithms, high-dimension problems pose significant challenges to traditional algorithms that are mainly designed to be implemented on a single computing unit. The exploding volume of data (and hence, the problem size), however, may overwhelm any such units. In this paper, we propose a distributed algorithm for general, non-separable, high-dimension convex QCQPs, using a novel idea of predictor–corrector primal–dual update with an adaptive step size. The algorithm enables distributed storage of data as well as parallel, distributed computing. We establish the conditions for the proposed algorithm to converge to a global optimum, and implement our algorithm on a computer cluster with multiple nodes using message passing interface. The numerical experiments are conducted on data sets of various scales from different applications, and the results show that our algorithm exhibits favorable scalability for solving high-dimension problems.

Appendices

Appendix 1: Step-size update rule for $\rho ^{k+1}$

With a given scalar $0 \le \epsilon _0 < 1$, and a series of positive scalars $\epsilon _1, \ldots , \epsilon _8 > 0$ that satisfy $\sum _{s=1}^8 \epsilon _s \le 1 - \epsilon _0$, we define the following function $\rho : {\mathbb {X}} \times {\mathbb {R}}^{n_2} \times {\mathbb {R}}_+^{m_1} \times {\mathbb {R}}^{m_2} \rightarrow (0, +\infty )$ to update the adaptive step size $\rho ^{k+1}$ in Algorithm 1 at each iteration k:

$$\begin{aligned} \rho ^{k+1}&= \rho ({\mathbf {x}}^k, {\mathbf {u}}^k, \varvec{\lambda }^k, \varvec{\gamma }^k) \nonumber \\&:= \min \left\{ \rho _1, \rho _2({\mathbf {x}}^k, {\mathbf {u}}^k, \varvec{\lambda }^k), \rho _3({\mathbf {x}}^k, \varvec{\lambda }^k, \varvec{\gamma }^k), \rho _4, \rho _5({\mathbf {x}}^k), \rho _6, \rho _7, \rho _8\right\} , \end{aligned}$$

(49)

where

• (i) $\rho _1 = \left\{ \begin{array}{ll} \displaystyle \frac{\epsilon _1}{\Vert P_0 \Vert _F}, &{}\text {if }\Vert P_0 \Vert _F \not = 0 \\ \epsilon _1, &{}\text {if }\Vert P_0 \Vert _F = 0, \end{array} \right.$ with $\Vert \cdot \Vert _F$ representing the Frobenius norm of a matrix;

• (ii) $\rho _2({\mathbf {x}}^k, {\mathbf {u}}^k, \varvec{\lambda }^k) = \min _i\{\rho _{2i}({\mathbf {x}}^k, {\mathbf {u}}^k, \varvec{\lambda }^k)\}$, where

  $$\begin{aligned} \rho _{2i}({\mathbf {x}}^k, {\mathbf {u}}^k, \varvec{\lambda }^k) {:}{=}\left\{ \begin{array}{ll} \displaystyle \frac{-b_i + \sqrt{b_i^2 + 4 a_i c_i}}{2 a_i}, &{}\text {if }a_i> 0 \\ \displaystyle \frac{c_i}{b_i}, &{}\text {if }a_i = 0, b_i > 0 \\ M, &{}\text {if }a_i = 0, b_i = 0, \end{array} \right. \end{aligned}$$

  for all $i = 1, \dots , m_1$, with $a_i = |\frac{1}{2} ({\mathbf {x}}^k)^T P_i {\mathbf {x}}^k + {\mathbf {q}}_i^T {\mathbf {x}}^k + {\mathbf {c}}_i^T {\mathbf {u}}^k + r_i |\ge 0$, and $b_i = \lambda _i^k \ge 0$. For $c_i$, if $\Vert P_i \Vert _F \not = 0$, $c_i = \frac{\epsilon _2}{m_1 \Vert P_i \Vert _F} > 0$; otherwise $c_i = \frac{\epsilon _2}{m_1} > 0$. The constant $M > 0$ can be any fixed, arbitrarily large scalar;

• (iii) $\rho _{3}({\mathbf {x}}^k, \varvec{\lambda }^k, \varvec{\gamma }^k) =$

  $$\begin{aligned} \left\{ \begin{array}{ll} \displaystyle \min \left\{2 \epsilon _3, \frac{-b + \sqrt{b^2 + 4 a c}}{2 a}\right\}, &{}\text {if }a> 0 \\ \displaystyle \min \left\{2 \epsilon _3, \frac{c}{b}\right\}, &{}\text {if }a = 0, b > 0 \\ 2 \epsilon _3, &{}\text {if }a = 0, b = 0, \end{array} \right. \end{aligned}$$

  where $a = \Vert P_0 {\mathbf {x}}^k + {\mathbf {q}}_0 + \sum _{i=1}^{m_1} \lambda _i^k (P_i {\mathbf {x}}^k + {\mathbf {q}}_i) + A^T \varvec{\gamma }^k\Vert _2 \ge 0$, $b = 2 \Vert {\mathbf {x}}^k \Vert _2 \ge 0$ and $c = \frac{2 \epsilon _3}{\Vert P \Vert _F} > 0$ with $P \in {\mathbb {R}}^{m_1 n_1 \times n_1}$ denoting the stacked matrix $\left( \begin{array}{c}P_1 \\ \vdots \\ P_{m_1}\end{array}\right)$;

• (iv) $\rho _4 = \left\{ \begin{array}{ll} \displaystyle \frac{\epsilon _4}{\Vert Q \Vert _F}, &{}\text {if }\Vert Q \Vert _F \not = 0 \\ \epsilon _4, &{}\text {if }\Vert Q \Vert _F = 0 \end{array} \right.$, where $Q \in {\mathbb {R}}^{m_1 \times n_1}$ denotes matrix $\left( \begin{array}{c}{\mathbf {q}}_1^T \\ \vdots \\ {\mathbf {q}}_{m_1}^T\end{array}\right)$, with the ${\mathbf {q}}_i$’s being the vectors in the linear terms of ${\mathbf {x}}$ in the QCQP (8);

• (v) $\rho _5({\mathbf {x}}^k) = \left\{ \begin{array}{ll} \displaystyle \frac{\epsilon _5}{\Vert {\mathbf {x}}^k \Vert _2 \Vert P \Vert _F}, &{}\text {if }\Vert {\mathbf {x}}^k \Vert _2 \not = 0 \\ \epsilon _5, &{}\text {if }\Vert {\mathbf {x}}^k \Vert _2 = 0 \end{array} \right.$;

• (vi) $\rho _6 = \left\{ \begin{array}{ll} \displaystyle \frac{\epsilon _6}{\Vert C \Vert _F}, &{}\text {if }\Vert C \Vert _F \not = 0 \\ \epsilon _6, &{}\text {if }\Vert C \Vert _F = 0 \end{array} \right.$, where $C \in {\mathbb {R}}^{m_2 \times n_2}$ denotes matrix $\left( \begin{array}{c}{\mathbf {c}}_1^T \\ \vdots \\ {\mathbf {c}}_{m_2}^T\end{array}\right)$, with the ${\mathbf {c}}_j$’s being the vectors in the linear terms of ${\mathbf {u}}$ in the QCQP (8);

• (vii) $\rho _7 = \left\{ \begin{array}{ll} \displaystyle \frac{\epsilon _7}{\Vert A \Vert _F}, &{}\text {if }\Vert A \Vert _F \not = 0 \\ \epsilon _7, &{}\text {if }\Vert A \Vert _F = 0 \end{array} \right.$, where A is the matrix in the linear constraint $A {\mathbf {x}} + B {\mathbf {u}} = {\mathbf {b}}$ in (8);

• (viii) $\rho _8 = \left\{ \begin{array}{ll} \displaystyle \frac{\epsilon _8}{\Vert B \Vert _F}, &{}\text {if }\Vert B \Vert _F \not = 0 \\ \epsilon _8, &{}\text {if }\Vert B \Vert _F = 0 \end{array} \right.$, where B is the matrix in the linear constraint $A {\mathbf {x}} + B {\mathbf {u}} = {\mathbf {b}}$ in (8).

$\Box$

While the rules to update the step-size $\rho ^{k+1}$ may appear to be very cumbersome, the calculations are actually quite straightforward. Since the Frobenius norm of all matrices can be obtained in advance, the values of $\rho _1$, $\rho _4$, $\rho _6$, $\rho _7$ and $\rho _8$ are pre-determined. Given a current solution $({\mathbf {x}}^k, {\mathbf {u}}^k, \varvec{\lambda }^k, \varvec{\gamma }^k)$, $\rho _2$, $\rho _{3}$ and $\rho _5$ can also be easily calculated. The minimum of all the $\rho _s$’s then determines the value of the adaptive step size $\rho ^{k+1}$.

Appendix 2: Proofs in Sect. 3

1.1 Proof of Proposition 1

We first prove the inequality (24). Consider the linear approximation of the Lagrangian function of a QCQP, as defined in (21), with a given point $\zeta ^k \equiv ({\mathbf {x}}^k, \lambda ^k, \gamma ^k)$. Let $\widehat{{\mathbf {z}}} = ({\mathbf {y}}^{k+1}, {\mathbf {v}}^{k+1})$, the $(k+1)$th iteration of the primal predictor of ${\mathbf {x}}^k$ and ${\mathbf {u}}^k$ in the PC$^2$PM algorithm, as given in (14a) and (16a), respectively. By Lemma 2, we know that $\widehat{{\mathbf {z}}}$ is the unique minimizer of the corresponding proximal minimization problem in (22b). By defining $\bar{{\mathbf {z}}} = ({\mathbf {x}}^k, {\mathbf {u}}^k)$ and ${\mathbf {z}} = ({\mathbf {x}}^{k+1}, {\mathbf {u}}^{k+1})$, and using Lemma 1, we have that

$$\begin{aligned} 2\rho ^{k+1}\Bigg [{\mathcal {R}}(\widehat{{\mathbf {z}}}; \zeta ^k) - {\mathcal {R}}({\mathbf {z}};\zeta ^k)\Bigg ] \le \Vert \bar{{\mathbf {z}}} - {\mathbf {z}} \Vert _2^2 - \Vert \widehat{{\mathbf {z}}} - {\mathbf {z}} \Vert _2^2 - \Vert \widehat{{\mathbf {z}}} - \bar{{\mathbf {z}}} \Vert _2^2, \end{aligned}$$

(50)

which leads to the following expanded inequality

$$\begin{aligned}&2 \rho ^{k+1}\Bigg \{(P_0 {\mathbf {x}}^k + {\mathbf {q}}_0)^T {\mathbf {y}}^{k+1} + {\mathbf {c}}_0^T {\mathbf {v}}^{k+1} + r_0 \nonumber \\&\qquad + \sum _{i=1}^{m_1} \lambda _i^k \big [(P_i {\mathbf {x}}^k + {\mathbf {q}}_i)^T {\mathbf {y}}^{k+1} + {\mathbf {c}}_i^T {\mathbf {v}}^{k+1} + r_i\big ] \nonumber \\&\qquad + (\varvec{\gamma }^k)^T (A {\mathbf {y}}^{k+1} + B {\mathbf {v}}^{k+1} - {\mathbf {b}})\Bigg \} \nonumber \\&\qquad - 2 \rho ^{k+1}\Bigg \{(P_0 {\mathbf {x}}^k + {\mathbf {q}}_0)^T {\mathbf {x}}^{k+1} + {\mathbf {c}}_0^T {\mathbf {u}}^{k+1} + r_0 \nonumber \\&\qquad + \sum _{i=1}^{m_1} \lambda _i^k \big [(P_i {\mathbf {x}}^k + {\mathbf {q}}_i)^T {\mathbf {x}}^{k+1} + {\mathbf {c}}_i^T {\mathbf {u}}^{k+1} + r_i\big ] \nonumber \\&\qquad + (\varvec{\gamma }^k)^T (A {\mathbf {x}}^{k+1} + B {\mathbf {u}}^{k+1} - {\mathbf {b}})\Bigg \} \nonumber \\&\quad \le \Big (\Vert {\mathbf {x}}^k - {\mathbf {x}}^{k+1} \Vert _2^2 + \Vert {\mathbf {u}}^k - {\mathbf {u}}^{k+1} \Vert _2^2 \Big ) \nonumber \\&\qquad - \Big (\Vert {\mathbf {y}}^{k+1} + {\mathbf {x}}^{k+1} \Vert _2^2 + \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^{k+1} \Vert _2^2 \Big ) \nonumber \\&\qquad - \Big (\Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2 + \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^k \Vert _2^2\Big ). \end{aligned}$$

(51)

Now consider the ${\mathcal {R}}$ function at a different given point $\zeta ^{k+1} \equiv ({\mathbf {y}}^{k+1}, \mu ^{k+1}, \varvec{\nu }^{k+1})$. With a slight abuse of notation, we now let $\widehat{{\mathbf {z}}} = ({\mathbf {x}}^{k+1}, {\mathbf {u}}^{k+1})$, the primal correctors at the $(k+1)$th iteration of the PC$^2$PM algorithm. Also letting ${\mathbf {z}} = ({\mathbf {x}}^*, {\mathbf {u}}^*)$, but keeping $\bar{{\mathbf {z}}} = ({\mathbf {x}}^k, {\mathbf {u}}^k)$, by (22c) in Lemmas 2 and 1, we have that:

$$\begin{aligned} 2\rho ^{k+1}\Bigg [{\mathcal {R}}(\widehat{{\mathbf {z}}}; \zeta ^{k+1}) - {\mathcal {R}}({\mathbf {z}};\zeta ^{k+1})\Bigg ] \le \Vert \bar{{\mathbf {z}}} - {\mathbf {z}} \Vert _2^2 - \Vert \widehat{{\mathbf {z}}} - {\mathbf {z}} \Vert _2^2 - \Vert \widehat{{\mathbf {z}}} - \bar{{\mathbf {z}}} \Vert _2^2, \end{aligned}$$

which leads to the following expanded inequality

$$\begin{aligned}&2 \rho ^{k+1}\Bigg \{(P_0 {\mathbf {y}}^{k+1} + {\mathbf {q}}_0)^T {\mathbf {x}}^{k+1} + {\mathbf {c}}_0^T {\mathbf {u}}^{k+1} + r_0 \nonumber \\&\qquad +\sum _{i=1}^{m_1} \mu _i^{k+1} \big [(P_i {\mathbf {y}}^{k+1} + {\mathbf {q}}_i)^T {\mathbf {x}}^{k+1} + {\mathbf {c}}_i^T {\mathbf {u}}^{k+1} + r_i\big ] \nonumber \\&\qquad + (\varvec{\nu }^{k+1})^T (A {\mathbf {x}}^{k+1} + B {\mathbf {u}}^{k+1} - {\mathbf {b}})\Bigg \} \nonumber \\&\qquad - 2 \rho ^{k+1}\Bigg \{(P_0 {\mathbf {y}}^{k+1} + {\mathbf {q}}_0)^T {\mathbf {x}}^* + {\mathbf {c}}_0^T {\mathbf {u}}^* + r_0 \nonumber \\&\qquad +\sum _{i=1}^{m_1} \mu _i^{k+1} \big [(P_i {\mathbf {y}}^{k+1} + {\mathbf {q}}_i)^T {\mathbf {x}}^* + {\mathbf {c}}_i^T {\mathbf {u}}^* + r_i\big ] \nonumber \\&\qquad + (\varvec{\nu }^{k+1})^T (A {\mathbf {x}}^* + B {\mathbf {u}}^* - {\mathbf {b}})\Bigg \} \nonumber \\&\quad \le \Big (\Vert {\mathbf {x}}^k - {\mathbf {x}}^* \Vert _2^2 + \Vert {\mathbf {u}}^k - {\mathbf {u}}^* \Vert _2^2 \Big ) \nonumber \\&\qquad - \Big (\Vert {\mathbf {x}}^{k+1} - {\mathbf {x}}^* \Vert _2^2 + \Vert {\mathbf {u}}^{k+1} - {\mathbf {u}}^* \Vert _2^2 \Big ) \nonumber \\&\qquad - \Big (\Vert {\mathbf {x}}^{k+1} - {\mathbf {x}}^k \Vert _2^2 + \Vert {\mathbf {u}}^{k+1} - {\mathbf {u}}^k \Vert _2^2\Big ). \end{aligned}$$

(52)

The final piece to derive inequality (24) is to utilize Lemma 3. Let $({\mathbf {x}}^*, {\mathbf {u}}^*, \varvec{\lambda }^*, \varvec{\gamma }^*)$ be a saddle point of QCQP (8), and again, $\zeta ^{k+1} = ({\mathbf {y}}^{k+1}, \mu ^{k+1}, \varvec{\nu }^{k+1})$. By Lemma 3, we have that

$$\begin{aligned} \begin{aligned}&{\mathcal {R}}({\mathbf {x}}^*, {\mathbf {u}}^*; \zeta ^{k+1}) - {\mathcal {R}}({\mathbf {y}}^{k+1}, {\mathbf {v}}^{k+1} ;\zeta ^{k+1}) \\&\quad \le \displaystyle \sum _{i=1}^m \bigg [ (\lambda _i^* - \mu _i^{k+1}) \bigg (\frac{1}{2} {\mathbf {y}}^{{k+1}^T} P_i {\mathbf {y}}^{k+1} + {\mathbf {q}}_i^T {\mathbf {y}}^{k+1} + {\mathbf {c}}_i^T {\mathbf {v}}^{k+1} + r_i\bigg )\bigg ] \\&\qquad + (\varvec{\gamma }^* - \varvec{\nu }^{k+1})^T (A {\mathbf {y}}^{k+1} + B {\mathbf {v}}^{k+1} - {\mathbf {b}}). \end{aligned} \end{aligned}$$

(53)

Multiplying both sides by $2 \rho ^{k+1}$ and expanding the ${\mathcal {R}}$ function, we have that

$$\begin{aligned}&2 \rho ^{k+1} \Bigg \{(P_0 {\mathbf {y}}^{k+1} + {\mathbf {q}}_0)^T {\mathbf {x}}^* + {\mathbf {c}}_0^T {\mathbf {u}}^* + r_0 \nonumber \\&\qquad +\sum _{i=1}^m \mu _i^{k+1} \big [(P_i {\mathbf {y}}^{k+1} + {\mathbf {q}}_i)^T {\mathbf {x}}^* + {\mathbf {c}}_i^T {\mathbf {u}}^* + r_i\big ] \nonumber \\&\qquad + (\varvec{\nu }^{k+1})^T (A {\mathbf {x}}^* + B {\mathbf {u}}^* - {\mathbf {b}})\Bigg \} \nonumber \\&\qquad - 2 \rho ^{k+1} \Bigg \{(P_0 {\mathbf {y}}^{k+1} + {\mathbf {q}}_0)^T {\mathbf {y}}^{k+1} + {\mathbf {c}}_0^T {\mathbf {v}}^{k+1} + r_0 \nonumber \\&\qquad + \sum _{i=1}^m \mu _i^{k+1} \big [(P_i {\mathbf {y}}^{k+1} + {\mathbf {q}}_i)^T {\mathbf {y}}^{k+1} + {\mathbf {c}}_i^T {\mathbf {v}}^{k+1} + r_i\big ] \nonumber \\&\qquad + (\varvec{\nu }^{k+1})^T (A {\mathbf {y}}^{k+1} + B {\mathbf {v}}^{k+1} - {\mathbf {b}})\Bigg \} \nonumber \\&\quad \le 2 \rho ^{k+1} \Bigg \{\sum _{i=1}^m (\lambda _i^* - \mu _i^{k+1}) \big [\frac{1}{2} ({\mathbf {y}}^{k+1})^T P_i {\mathbf {y}}^{k+1} + {\mathbf {q}}_i^T {\mathbf {y}}^{k+1} + {\mathbf {c}}_i^T {\mathbf {v}}^{k+1} + r_i\big ] \nonumber \\&\qquad + (\varvec{\gamma }^* - \varvec{\nu }^{k+1})^T (A {\mathbf {y}}^{k+1} + B {\mathbf {v}}^{k+1} - {\mathbf {b}})\Bigg \}. \end{aligned}$$

(54)

Adding the three inequalities (51), (52) and (54) yields the inequality (24) in Proposition 1.

To prove the second inequality, (25), in Proposition 1, we use a similar approach as above, just replacing the linear approximation function ${\mathcal {R}}$ with the original Lagrangian function ${\mathcal {L}}$. More specifically, let $\widehat{{\mathbf {z}}} = (\varvec{\mu }^{k+1}, \varvec{\nu }^{k+1})$. By (22a) in Lemma 2, we know that

$$\begin{aligned} \begin{aligned}&\widehat{{\mathbf {z}}} {:}{=}(\varvec{\mu }^{k+1},\ \varvec{\nu }^{k+1}) \\&\quad = \underset{\varvec{\lambda } \in {\mathbb {R}}_+^{m_1},\ \varvec{\gamma } \in {\mathbb {R}}^{m_2}}{{{\,{\arg \!\min }\,}}} -{\mathcal {L}}({\mathbf {x}}^k, {\mathbf {u}}^k, \varvec{\lambda }, \varvec{\gamma }) + \frac{1}{2 \rho ^{k+1}} \Vert \varvec{\lambda } - \varvec{\lambda }^k \Vert _2^2 + \frac{1}{2 \rho ^{k+1}} \Vert \varvec{\gamma } - \varvec{\gamma }^k \Vert _2^2. \end{aligned} \end{aligned}$$

(55)

Letting $\bar{{\mathbf {z}}} = (\varvec{\lambda }^k, \varvec{\gamma }^k)$ and choosing a specific ${\mathbf {z}} = (\varvec{\lambda }^{k+1}, \varvec{\gamma }^{k+1})$, we use Lemma 1 to obtain that

$$\begin{aligned} 2\rho ^{k+1}\Bigg [ \bigg (-{\mathcal {L}} ({\mathbf {x}}^k,{\mathbf {u}}^k;\widehat{{\mathbf {z}}}) \bigg ) - \Bigg (-{\mathcal {L}} ({\mathbf {x}}^k,{\mathbf {u}}^k; {\mathbf {z}}) \Bigg ) \Bigg ] \le \Vert \bar{{\mathbf {z}}} - {\mathbf {z}} \Vert _2^2 - \Vert \widehat{{\mathbf {z}}} - {\mathbf {z}} \Vert _2^2 - \Vert \widehat{{\mathbf {z}}} - \bar{{\mathbf {z}}} \Vert _2^2, \end{aligned}$$

(56)

which yields the following expanded inequality:

$$\begin{aligned}&2 \rho ^{k+1}\Bigg \{\sum _{i=1}^{m_1} (\lambda _i^{k+1} - \mu _i^{k+1}) \bigg [\frac{1}{2} ({\mathbf {x}}^k)^T P_i {\mathbf {x}}^k + {\mathbf {q}}_0^T {\mathbf {x}}^k + {\mathbf {c}}_0^T {\mathbf {u}}^k + r_i\bigg ] \nonumber \\&\qquad + (\varvec{\gamma }^{k+1} - \varvec{\nu }^{k+1})^T (A {\mathbf {x}}^k + B {\mathbf {u}}^k - {\mathbf {b}})\Bigg \} \nonumber \\&\quad \le \Big (\Vert \varvec{\lambda }^k - \varvec{\lambda }^{k+1} \Vert _2^2 + \Vert \varvec{\gamma }^k - \varvec{\gamma }^{k+1} \Vert _2^2\Big ) \nonumber \\&\qquad - \Big (\Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^{k+1} \Vert _2^2 + \Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^{k+1} \Vert _2^2\Big ) - \Big (\Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^k \Vert _2^2 + \Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^k \Vert _2^2\Big ). \end{aligned}$$

(57)

Similarly, again with some abuse of notation, letting $\widehat{{\mathbf {z}}} = (\varvec{\lambda }^{k+1}, \varvec{\gamma }^{k+1})$, by (22d) in Lemma 2, we have that

$$\begin{aligned} \begin{aligned}&\widehat{{\mathbf {z}}} {:}{=}(\varvec{\lambda }^{k+1}, \varvec{\gamma }^{k+1}) \\&\quad = \underset{\varvec{\lambda } \in {\mathbb {R}}_+^{m_1},\ \varvec{\gamma } \in {\mathbb {R}}^{m_2}}{{{\,{\arg \!\min }\,}}} -{\mathcal {L}}({\mathbf {y}}^{k+1}, {\mathbf {v}}^{k+1}, \varvec{\lambda }, \varvec{\gamma }) + \frac{1}{2 \rho ^{k+1}} \Vert \varvec{\lambda } - \varvec{\lambda }^k \Vert _2^2 + \frac{1}{2 \rho ^{k+1}} \Vert \varvec{\gamma } - \varvec{\gamma }^k \Vert _2^2. \end{aligned} \end{aligned}$$

(58)

By choosing ${\mathbf {z}}$ to be $(\varvec{\lambda }^*, \varvec{\gamma }^*)$, while keeping $\bar{{\mathbf {z}}}$ at $(\varvec{\lambda }^k, \varvec{\gamma }^k)$, we have from Lemma 1 that

$$\begin{aligned} \begin{aligned}&2\rho ^{k+1}\Bigg [ \bigg (-{\mathcal {L}} ({\mathbf {y}}^{k+1},{\mathbf {v}}^{k+1};\widehat{{\mathbf {z}}}) \bigg ) - \Bigg (-{\mathcal {L}} ({\mathbf {y}}^{k+1},{\mathbf {v}}^{k+1}; {\mathbf {z}}) \Bigg ) \Bigg ] \\&\quad \le \Vert \bar{{\mathbf {z}}} - {\mathbf {z}} \Vert _2^2 - \Vert \widehat{{\mathbf {z}}} - {\mathbf {z}} \Vert _2^2 - \Vert \widehat{{\mathbf {z}}} - \bar{{\mathbf {z}}} \Vert _2^2, \end{aligned} \end{aligned}$$

(59)

which yields the following expanded inequality:

$$\begin{aligned}&2 \rho ^{k+1}\Bigg \{\sum _{i=1}^{m_1} (\lambda _i^* - \lambda _i^{k+1}) \bigg [\frac{1}{2} ({\mathbf {y}}^{k+1})^T P_i {\mathbf {y}}^{k+1} + {\mathbf {q}}_0^T {\mathbf {y}}^{k+1} + {\mathbf {c}}_0^T {\mathbf {v}}^{k+1} + r_i\bigg ] \nonumber \\&\qquad + (\varvec{\gamma }^* - \varvec{\gamma }^{k+1})^T (A {\mathbf {y}}^{k+1} + B {\mathbf {v}}^{k+1} - {\mathbf {b}})\Bigg \} \nonumber \\&\quad \le \Big (\Vert \varvec{\lambda }^k - \varvec{\lambda }^* \Vert _2^2 + \Vert \varvec{\gamma }^k - \varvec{\gamma }^* \Vert _2^2\Big ) \nonumber \\&\qquad - \Big (\Vert \varvec{\lambda }^{k+1} - \varvec{\lambda }^* \Vert _2^2 + \Vert \varvec{\gamma }^{k+1} - \varvec{\gamma }^* \Vert _2^2\Big ) - \Big (\Vert \varvec{\lambda }^{k+1} - \varvec{\lambda }^k \Vert _2^2 + \Vert \varvec{\gamma }^{k+1} - \varvec{\gamma }^k \Vert _2^2\Big ). \end{aligned}$$

(60)

Adding the two inequalities (57) and (60) leads to the second inequality, (25), in Proposition 1. $\Box$

1.2 Proof of Theorem 1

By adding the two inequalities (24) and (25) in Proposition 1, we have that

$$\begin{aligned}&\Vert {\mathbf {x}}^{k+1} - {\mathbf {x}}^* \Vert _2^2 + \Vert {\mathbf {u}}^{k+1} - {\mathbf {u}}^* \Vert _2^2 + \Vert \varvec{\lambda }^{k+1} - \varvec{\lambda }^* \Vert _2^2 + \Vert \varvec{\gamma }^{k+1} - \varvec{\gamma }^* \Vert _2^2 \nonumber \\&\quad \le \Vert {\mathbf {x}}^k - {\mathbf {x}}^* \Vert _2^2 + \Vert {\mathbf {u}}^k - {\mathbf {u}}^* \Vert _2^2 + \Vert \varvec{\lambda }^k - \varvec{\lambda }^* \Vert _2^2 + \Vert \varvec{\gamma }^k - \varvec{\gamma }^* \Vert _2^2 \nonumber \\&\qquad - \Big (\Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2 + \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^{k+1} \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2 + \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^k \Vert _2^2\Big ) \nonumber \\&\qquad - \Big (\Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^{k+1} \Vert _2^2 + \Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^{k+1} \Vert _2^2 + \Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^k \Vert _2^2 + \Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^k \Vert _2^2\Big ) \nonumber \\&\qquad + \underbrace{2 \rho ^{k+1} ({\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1})^T P_0 ({\mathbf {y}}^{k+1} - {\mathbf {x}}^k)}_\text {(a)} \nonumber \\&\qquad + \sum _{i=1}^{m_1} \underbrace{2 \rho ^{k+1} \mu _i^{k+1} ({\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1})^T P_i ({\mathbf {y}}^{k+1} - {\mathbf {x}}^k)}_{\text {(b)}_i} \nonumber \\&\qquad +\underbrace{2\rho ^{k+1} \sum _{i=1}^{m_1} (\lambda _i^{k+1} - \mu _i^{k+1}) \big [\frac{1}{2} ({\mathbf {y}}^{k+1})^T P_i {\mathbf {y}}^{k+1} - \frac{1}{2} ({\mathbf {x}}^k)^T P_i {\mathbf {x}}^k\big ]}_\text {(c)} \nonumber \\&\qquad + \underbrace{2 \rho ^{k+1} \sum _{i=1}^{m_1} (\lambda _i^{k+1} - \mu _i^{k+1}) {\mathbf {q}}_i^T ({\mathbf {y}}^{k+1} - {\mathbf {x}}^k)}_\text {(d)} \nonumber \\&\qquad + \underbrace{2 \rho ^{k+1} \sum _{i=1}^{m_1} (\mu _i^{k+1} - \lambda _i^k) {\mathbf {q}}_i^T ({\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1})}_\text {(e)} \nonumber \\&\qquad + \underbrace{2 \rho ^{k+1} \sum _{i=1}^{m_1} (\mu _i^{k+1} - \lambda _i^k) (P_i {\mathbf {x}}^k)^T ({\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1})}_\text {(f)} \nonumber \\&\qquad + \underbrace{2 \rho ^{k+1} \sum _{i=1}^{m_1} (\lambda _i^{k+1} - \mu _i^{k+1}) {\mathbf {c}}_i^T ({\mathbf {v}}^{k+1} - {\mathbf {u}}^k)}_\text {(g)} \nonumber \\&\qquad + \underbrace{2 \rho ^{k+1} \sum _{i=1}^{m_1} (\mu _i^{k+1} - \lambda _i^k) {\mathbf {c}}_i^T ({\mathbf {v}}^{k+1} - {\mathbf {u}}^{k+1})}_\text {(h)} \nonumber \\&\qquad + \underbrace{2 \rho ^{k+1} (\varvec{\gamma }^{k+1} - \varvec{\nu }^{k+1})^T A ({\mathbf {y}}^{k+1} - {\mathbf {x}}^k)}_\text {(i)} + \underbrace{2 \rho ^{k+1} (\varvec{\nu }^{k+1} - \varvec{\gamma }^k)^T A ({\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1})}_\text {(j)} \nonumber \\&\qquad + \underbrace{2 \rho ^{k+1} (\varvec{\gamma }^{k+1} - \varvec{\nu }^{k+1})^T B ({\mathbf {v}}^{k+1} - {\mathbf {u}}^k)}_\text {(k)} + \underbrace{2 \rho ^{k+1} (\varvec{\nu }^{k+1} - \varvec{\gamma }^k)^T B ({\mathbf {v}}^{k+1} - {\mathbf {u}}^{k+1})}_{\text {(l)}}. \end{aligned}$$

(61)

Next, we establish an upper bound for each term of the term from (a) to (l) in (61) using the adaptive step size $\rho ^{k+1} = \rho ({\mathbf {x}}^k, {\mathbf {u}}^k, \varvec{\lambda }^k, \varvec{\gamma }^k)$, as defined in (49).

• (a) First, we want to show that

  $$\begin{aligned} \text {(a)} \le \epsilon _1 \Big (\Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ). \end{aligned}$$

  (62)

  To prove this (and several inequalities below), we first show an extension of the Young’s inequality³ on vector products that will play a key role in the following proof. Given any two vectors ${\mathbf {z}}_1, {\mathbf {z}}_2 \in {\mathbb {R}}^n$, we have that

  $$\begin{aligned} {\mathbf {z}}_1^T {\mathbf {z}}_2 = \sum _{j=1}^n z_{1j} z_{2j} = \sum _{j=1}^n \Big (\frac{1}{\delta } z_{1j}\Big ) \Big (\delta z_{2j}\Big ) \le \sum _{j=1}^n \Big |\frac{1}{\delta } z_{1j} \Big |\Big |\delta z_{2j} \Big |, \end{aligned}$$

  (63)

  where $\delta$ is a non-zero real number. Applying Young’s inequality on each summation term with $p = q = 2$, we obtain that

  $$\begin{aligned} {\mathbf {z}}_1^T {\mathbf {z}}_2 \le \sum _{j=1}^n \left[ \frac{1}{2} \Big (\frac{1}{\delta } z_{1j}\Big )^2 + \frac{1}{2} \Big (\delta z_{2j}\Big )^2\right] = \frac{1}{2 \delta ^2} \Vert {\mathbf {z}}_1 \Vert _2^2 + \frac{\delta ^2}{2} \Vert {\mathbf {z}}_2 \Vert _2^2. \end{aligned}$$

  (64)

  Applying (64) on (a) yields

  $$\begin{aligned} \text {(a)} \le&2 \rho ^{k+1} \Big (\frac{1}{2 \delta ^2} \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2 + \frac{\delta ^2}{2} \Vert P_0 ({\mathbf {y}}^{k+1} - {\mathbf {x}}^k) \Vert _2^2\Big ) \nonumber \\ \le&2 \rho ^{k+1} \Big (\frac{1}{2 \delta ^2} \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2 + \frac{\delta ^2}{2} {\left| \left| \left| P_0 \right| \right| \right| }_2^2 \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ) \nonumber \\ \le&2 \rho ^{k+1} \Big (\frac{1}{2 \delta ^2} \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2 + \frac{\delta ^2}{2} \Vert P_0 \Vert _F^2 \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ). \end{aligned}$$

  (65)

  The second inequality holds due to the property that given a matrix $A \in {\mathbb {R}}^{m \times n}$ and a vector ${\mathbf {z}} \in {\mathbb {R}}^n$, $\Vert A {\mathbf {z}} \Vert _2 \le {\left| \left| \left| A \right| \right| \right| }_2 \Vert {\mathbf {z}} \Vert _2$ (see Theorem 5.6.2 in [9]), where we use the notation ${\left| \left| \left| \cdot \right| \right| \right| }_2$ to denote the matrix norm ${\left| \left| \left| A \right| \right| \right| }_2 {:}{=}\underset{{\mathbf {z}} \not = {\mathbf {0}}}{\sup } \frac{\Vert A {\mathbf {z}} \Vert _2}{\Vert {\mathbf {z}} \Vert _2}$. The last inequality holds due to the property ${\left| \left| \left| A \right| \right| \right| }_2 \le \Vert A \Vert _F$ [7], where $\Vert A \Vert _F {:}{=}\big (\sum _{i=1}^m \sum _{j=1}^n |A_{ij} |^2\big )^{\frac{1}{2}}$ denotes the Frobenius norm. From (65), if $\Vert P_0 \Vert _F \not = 0$, then letting $\delta ^2 = \frac{1}{\Vert P_0 \Vert _F}$ yields

  $$\begin{aligned} \text {(a)} \le \rho ^{k+1} \Vert P_0 \Vert _F \Big (\Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ). \end{aligned}$$

  (66)

  Since $\rho ^{k+1} \le \rho _1 = \frac{\epsilon _1}{\Vert P_0 \Vert _F}$, we obtain (62). If, on the other hand, $\Vert P_0 \Vert _F = 0$, then letting $\delta ^2 = 1$ yields

  $$\begin{aligned} \text {(a)} \le \rho ^{k+1} \Big (\Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ). \end{aligned}$$

  (67)

  Since $\rho ^{k+1} \le \rho _1 = \epsilon _1$, (62) is also obtained.

• (b) Here we want to show that

  $$\begin{aligned} \sum _{i=1}^{m_1} \text {(b)}_i \le \epsilon _2 \Big (\Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ). \end{aligned}$$

  (68)

  Applying (64) on each term $\text {(b)}_i$ yields

  $$\begin{aligned} \begin{aligned} \text {(b)}_i &\le 2 \rho ^{k+1} \mu _i^{k+1} \Big (\frac{1}{2 \delta _i^2} \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2 + \frac{\delta _i^2}{2} {\left| \left| \left| P_i \right| \right| \right| }_2^2 \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ) \\ &\le 2 \rho ^{k+1} \mu _i^{k+1} \Big (\frac{1}{2 \delta _i^2} \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2 + \frac{\delta _i^2}{2} \Vert P_i \Vert _F^2 \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ). \end{aligned} \end{aligned}$$

  (69)

  • If $\Vert P_i \Vert _F \not = 0$, then letting $\delta _i^2 = \frac{1}{\Vert P_i \Vert _F}$ yields

    $$\begin{aligned} \begin{aligned} \text {(b)}_i &\le \rho ^{k+1} \mu _i^{k+1} \Vert P_i \Vert _F \Big (\Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ) \\ &\le \rho ^{k+1} {\tilde{\mu }}_i^{k+1} \Vert P_i \Vert _F \Big (\Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ), \end{aligned} \end{aligned}$$

    (70)

    where ${\tilde{\mu }}_i^{k+1} {:}{=}\lambda _i^k + \rho ^{k+1} |\frac{1}{2} ({\mathbf {x}}^k)^T P_i {\mathbf {x}}^k + {\mathbf {q}}_i^T {\mathbf {x}}^k + {\mathbf {c}}_i^T {\mathbf {u}}^k + r_i |\ge \mu _i^{k+1}$. If we can bound $\rho ^{k+1} {\tilde{\mu }}_i^{k+1} \le \frac{\epsilon _2}{m_1 \Vert P_i \Vert _F}$, then we can achieve

    $$\begin{aligned} \text {(b)}_i \le \frac{\epsilon _2}{m_1} \Big (\Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ). \end{aligned}$$

    (71)

    By substituting $a_i = |\frac{1}{2} ({\mathbf {x}}^k)^T P_i {\mathbf {x}}^k + {\mathbf {q}}_i^T {\mathbf {x}}^k + {\mathbf {c}}_i^T {\mathbf {u}}^k + r_i |\ge 0$, $b_i = \lambda _i^k \ge 0$ and $c_i = \frac{\epsilon _2}{m_1 \Vert P_i \Vert _F} > 0$, we can rewrite $\rho ^{k+1} {\tilde{\mu }}_i^{k+1} - \frac{\epsilon _2}{m_1 \Vert P_i \Vert _F}$ as $a_i(\rho ^{k+1})^2 + b_i \rho ^{k+1} - c_i$, which is simply a quadratic function of $\rho ^{k+1}$ with parameters $a_i$, $b_i$ and $c_i$. To bound $\rho ^{k+1} {\tilde{\mu }}_i^{k+1} \le \frac{\epsilon _2}{m_1 \Vert P_i \Vert _F}$ is equivalent to find proper values of $\rho ^{k+1}$ that keep the quadratic function stay below zero.

    • If $a_i = 0$ and $b_i = 0$, then $\rho ^{k+1} \in (0, +\infty )$.

    • If $a_i = 0$ and $b_i > 0$, then $\rho ^{k+1} \in (0, \frac{c_i}{b_i}]$.

    • If $a_i > 0$, then $\rho ^{k+1} \in (0, \frac{-b_i + \sqrt{b_i^2 + 4 a_i c_i}}{2 a_i}]$.

    Since $\rho ^{k+1} \le \rho _2({\mathbf {x}}^k, {\mathbf {u}}^k, \varvec{\lambda }^k) \le \rho _{2i}({\mathbf {x}}^k, {\mathbf {u}}^k, \varvec{\lambda }^k)$, it satisfies all the above three conditions, we then obtain (71), and hence (68).

  • If $\Vert P_i \Vert _F = 0$, then letting $\delta _i^2 = 1$ yields

    $$\begin{aligned} \begin{aligned} \text {(b)}_i \le&\rho ^{k+1} \mu _i^{k+1} \Big (\Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ) \\ \le&\rho ^{k+1} {\tilde{\mu }}_i^{k+1} \Big (\Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ). \end{aligned} \end{aligned}$$

    (72)

    Similarly, if we can bound $\rho ^{k+1} {\tilde{\mu }}_i^{k+1} \le \frac{\epsilon _2}{m_1}$, then we can also achieve (71). By substituting $a_i = |\frac{1}{2} ({\mathbf {x}}^k)^T P_i {\mathbf {x}}^k + {\mathbf {q}}_i^T {\mathbf {x}}^k + {\mathbf {c}}_i^T {\mathbf {u}}^k + r_i |\ge 0$, $b_i = \lambda _i^k \ge 0$ and $c_i = \frac{\epsilon _2}{m_1} > 0$, we can rewrite $\rho ^{k+1} {\tilde{\mu }}_i^{k+1} - \frac{\epsilon _2}{m_1}$ as $a_i(\rho ^{k+1})^2 + b_i \rho ^{k+1} - c_i$. The same analysis can be followed as discussed in the case of $\Vert P_i \Vert _F \not = 0$.

• (c) Next, we want to show that

  $$\begin{aligned} \text {(c)} \le \epsilon _3 \Big (\Vert \varvec{\lambda }^{k+1} - \varvec{\mu }^{k+1} \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ). \end{aligned}$$

  (73)

  By using P to denote $\left( \begin{array}{c}P_1 \\ \vdots \\ P_{m_1}\end{array}\right)$, we can rewrite

  $$\begin{aligned} \text {(c)} = \rho ^{k+1} \Big \{(\varvec{\lambda }^{k+1} - \varvec{\mu }^{k+1})^T \Big [I_{m_1 \times m_1} \otimes ({\mathbf {x}}^k + {\mathbf {y}}^{k+1})^T\Big ] P ({\mathbf {y}}^{k+1} - {\mathbf {x}}^k)\Big \}, \end{aligned}$$

  (74)

  where $\otimes$ denotes the Kronecker product; that is, given a matrix $A \in {\mathbb {R}}^{m_1 \times n_1}$ and a matrix $B \in {\mathbb {R}}^{m_2 \times n_2}$, $A \otimes B {:}{=}\left( \begin{array}{ccc}a_{11} B&{}\cdots &{}a_{1 m_1} B \\ \vdots &{}&{}\vdots \\ a_{m_1 1} B&{}\cdots &{}a_{m_1 n_1} B\end{array}\right)$. Applying (64) to (74) yields

  $$\begin{aligned} \begin{aligned} \text {(c)}&\le \rho ^{k+1} \Big (\frac{1}{2 \delta ^2} \Vert \varvec{\lambda }^{k+1} - \varvec{\mu }^{k+1} \Vert _2^2 \\&\quad + \frac{\delta ^2}{2} {\left| \left| \left| I_{m_1 \times m_1} \otimes ({\mathbf {x}}^k + {\mathbf {y}}^{k+1})^T \right| \right| \right| }_2^2 {\left| \left| \left| P \right| \right| \right| }_2^2 \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ) \\&\le \rho ^{k+1} \Big (\frac{1}{2 \delta ^2} \Vert \varvec{\lambda }^{k+1} - \varvec{\mu }^{k+1} \Vert _2^2 + \frac{\delta ^2}{2} \Vert {\mathbf {x}}^k + {\mathbf {y}}^{k+1} \Vert _2^2 \Vert P \Vert _F^2 \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ). \end{aligned} \end{aligned}$$

  (75)

  Since we have the property that ${\left| \left| \left| A \otimes B \right| \right| \right| }_2 = {\left| \left| \left| A \right| \right| \right| }_2 {\left| \left| \left| B \right| \right| \right| }_2$ (see Theorem 8 in [15]), the last inequality holds due to

  $$\begin{aligned} {\left| \left| \left| I_{m_1 \times m_1} \otimes ({\mathbf {x}}^k + {\mathbf {y}}^{k+1})^T \right| \right| \right| }_2^2 = {\left| \left| \left| I_{m_1 \times m_1} \right| \right| \right| }_2^2 {\left| \left| \left| ({\mathbf {x}}^k + {\mathbf {y}}^{k+1})^T \right| \right| \right| }_2^2, \end{aligned}$$

  (76)

  together with ${\left| \left| \left| I_{m_1 \times m_1} \right| \right| \right| }_2 = 1$ and ${\left| \left| \left| ({\mathbf {x}}^k + {\mathbf {y}}^{k+1})^T \right| \right| \right| }_2 \le \Vert ({\mathbf {x}}^k + {\mathbf {y}}^{k+1})^T \Vert _F = \Vert {\mathbf {x}}^k + {\mathbf {y}}^{k+1} \Vert _2$. Note that $\Vert P \Vert _F \not = 0$, otherwise the QCQP is simply a QP.

  • If $\Vert {\mathbf {x}}^k + {\mathbf {y}}^{k+1} \Vert _2 \not = 0$, then letting $\delta ^2 = \frac{1}{\Vert {\mathbf {x}}^k + {\mathbf {y}}^{k+1} \Vert _2 \Vert P \Vert _F}$ yields

    $$\begin{aligned} \begin{aligned} \text {(c)} \le&\frac{1}{2} \rho ^{k+1} \Vert {\mathbf {x}}^k + {\mathbf {y}}^{k+1} \Vert _2 \Vert P \Vert _F \Big (\Vert \varvec{\lambda }^{k+1} - \varvec{\mu }^{k+1} \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ) \\ \le&\frac{1}{2} \rho ^{k+1} \Vert {\mathbf {x}}^k + \tilde{{\mathbf {y}}}^{k+1} \Vert _2 \Vert P \Vert _F \Big (\Vert \varvec{\lambda }^{k+1} - \varvec{\mu }^{k+1} \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ), \end{aligned} \end{aligned}$$

    (77)

    where ${\tilde{y}}_j^{k+1} {:}{=}x_j^k + \rho \Big |\big [P_0 {\mathbf {x}}^k + {\mathbf {q}}_0 + \sum _{i=1}^{m_1} \lambda _i^k \big (P_i {\mathbf {x}}^k + {\mathbf {q}}_i\big ) + A^T \varvec{\gamma }^k\big ]_j \Big |\ge y_j^{k+1}$. If we can bound $\rho ^{k+1} \Vert {\mathbf {x}}^k + \tilde{{\mathbf {y}}}^{k+1} \Vert _2 \le \frac{2 \epsilon _3}{\Vert P^T \Vert _F}$, then (73) can be obtained. We first bound

    $$\begin{aligned}&\rho ^{k+1} \Vert {\mathbf {x}}^k + \tilde{{\mathbf {y}}}^{k+1} \Vert _2 - \frac{2 \epsilon _3}{\Vert P \Vert _F} \nonumber \\&\quad \le \rho ^{k+1} \Big [2 \Vert {\mathbf {x}}^k \Vert _2 + \rho ^{k+1} \Vert P_0 {\mathbf {x}}^k + {\mathbf {q}}_0 + \sum _{i=1}^{m_1} \lambda _i^k \big (P_i {\mathbf {x}}^k + {\mathbf {q}}_i\big ) + A^T \varvec{\gamma }^k \Vert _2\Big ] \nonumber \\&\qquad - \frac{2 \epsilon _3}{\Vert P \Vert _F}. \end{aligned}$$

    (78)

    By substituting $a = \Vert P_0 {\mathbf {x}}^k + {\mathbf {q}}_0 + \sum _{i=1}^{m_1} \lambda _i^k \big (P_i {\mathbf {x}}^k + {\mathbf {q}}_i\big ) + A^T \varvec{\gamma }^k \Vert _2 \ge 0$, $b = 2 \Vert {\mathbf {x}}^k \Vert _2 \ge 0$ and $c = \frac{2 \epsilon _3}{\Vert P \Vert _F} > 0$, we can bound $\rho ^{k+1} \Vert {\mathbf {x}}^k + \tilde{{\mathbf {y}}}^{k+1} \Vert _2 - \frac{2 \epsilon _3}{\Vert P \Vert _F}$ using $a (\rho ^{k+1})^2 + b \rho ^{k+1} - c$, which is simply a quadratic function of $\rho ^{k+1}$ with parameters a, b and c. Bounding $\rho ^{k+1} \Vert {\mathbf {x}}^k + \tilde{{\mathbf {y}}}^{k+1} \Vert _2 \le \frac{2 \epsilon _3}{\Vert P \Vert _F}$ can be guaranteed by finding the proper values of $\rho ^{k+1}$ that keep the quadratic function stay below zero.

    • If $a = 0$ and $b = 0$, then $\rho ^{k+1} \in (0, +\infty )$.

    • If $a = 0$ and $b > 0$, then $\rho ^{k+1} \in (0, \frac{c}{b}]$.

    • If $a > 0$, then $\rho ^{k+1} \in (0, \frac{-b + \sqrt{b^2 + 4 a c}}{2 a}]$.

    Since $\rho ^{k+1} \le \rho _3({\mathbf {x}}^k, \varvec{\lambda }^k, \varvec{\gamma }^k)$, it satisfies all the above three conditions, we obtain (73).

  • If $\Vert {\mathbf {x}}^k + {\mathbf {y}}^{k+1} \Vert _2 = 0$, then letting $\delta ^2 = 1$ yields

    $$\begin{aligned} (c) \le \frac{1}{2} \rho ^{k+1} \Big (\Vert \varvec{\lambda }^{k+1} - \varvec{\mu }^{k+1} \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ) \end{aligned}$$

    (79)

    Since $\rho ^{k+1} \le \rho _3({\mathbf {x}}^k, \varvec{\lambda }^k, \varvec{\gamma }^k) \le 2 \epsilon _3$, (73) is also obtained.

• (d) To show that

  $$\begin{aligned} \text {(d)} \le \epsilon _4 \Big (\Vert \varvec{\lambda }^{k+1} - \varvec{\mu }^{k+1} \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ), \end{aligned}$$

  (80)

  by letting $Q = \left( \begin{array}{c}{\mathbf {q}}_1^T \\ \vdots \\ {\mathbf {q}}_{m_1}^T\end{array}\right)$, we can rewrite that

  $$\begin{aligned} \text {(d)} = 2 \rho ^{k+1} \Big [(\varvec{\lambda }^{k+1} - \varvec{\mu }^{k+1})^T Q^T ({\mathbf {y}}^{k+1} - {\mathbf {x}}^k)\Big ]. \end{aligned}$$

  (81)

  Applying (64) to (81) yields

  $$\begin{aligned} \begin{aligned} \text {(d)} \le&2 \rho ^{k+1} \Big (\frac{1}{2 \delta ^2} \Vert \varvec{\lambda }^{k+1} - \varvec{\mu }^{k+1} \Vert _2^2 + \frac{\delta ^2}{2} {\left| \left| \left| Q \right| \right| \right| }_2^2 \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ) \\ \le&2 \rho ^{k+1} \big (\frac{1}{2 \delta ^2} \Vert \varvec{\lambda }^{k+1} - \varvec{\mu }^{k+1} \Vert _2^2 + \frac{\delta ^2}{2} \Vert Q \Vert _F^2 \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\big ). \end{aligned} \end{aligned}$$

  (82)

  • If $\Vert Q \Vert _F \not = 0$, then letting $\delta ^2 = \frac{1}{\Vert Q \Vert _F}$ yields

    $$\begin{aligned} \text {(d)} \le \rho ^{k+1} \Vert Q \Vert _F \Big (\Vert \varvec{\lambda }^{k+1} - \varvec{\mu }^{k+1} \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ). \end{aligned}$$

    (83)

    Since $\rho ^{k+1} \le \rho _4 = \frac{\epsilon _4}{\Vert Q \Vert _F}$, we obtain (80).

  • If $\Vert Q \Vert _F = 0$, then letting $\delta ^2 = 1$ yields

    $$\begin{aligned} \text {(d)} \le \rho ^{k+1} \Big (\Vert \varvec{\lambda }^{k+1} - \varvec{\mu }^{k+1} \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ). \end{aligned}$$

    (84)

    Since $\rho ^{k+1} \le \rho _4 = \epsilon _4$, (80) is also obtained.

• (e) Similarly, to show

  $$\begin{aligned} \text {(e)} \le \epsilon _4 \Big (\Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^k \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2\Big ), \end{aligned}$$

  (85)

  we can rewrite that

  $$\begin{aligned} \text {(e)} = 2 \rho ^{k+1} \Big [(\varvec{\mu }^{k+1} - \varvec{\lambda }^k)^T Q ({\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1})\Big ]. \end{aligned}$$

  (86)

  Applying (64) to (86) yields that

  $$\begin{aligned} \begin{aligned} \text {(e)} \le&2 \rho ^{k+1} \Big (\frac{1}{2 \delta ^2} \Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^k \Vert _2^2 + \frac{\delta ^2}{2} {\left| \left| \left| Q \right| \right| \right| }_2^2 \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2\Big ) \\ \le&2 \rho ^{k+1} \Big (\frac{1}{2 \delta ^2} \Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^k \Vert _2^2 + \frac{\delta ^2}{2} \Vert Q \Vert _F^2 \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2\Big ). \end{aligned} \end{aligned}$$

  (87)

  • If $\Vert Q \Vert _F \not = 0$, then letting $\delta ^2 = \frac{1}{\Vert Q \Vert _F}$ yields

    $$\begin{aligned} \text {(e)} \le \rho ^{k+1} \Vert Q \Vert _F \Big (\Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^k \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2\Big ). \end{aligned}$$

    (88)

    Since $\rho ^{k+1} \le \rho _4 = \frac{\epsilon _4}{\Vert Q \Vert _F}$, we obtain (85).

  • If $\Vert Q \Vert _F = 0$, then letting $\delta ^2 = 1$ yields

    $$\begin{aligned} \text {(e)} \le \rho ^{k+1} \Big (\Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^k \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2\Big ). \end{aligned}$$

    (89)

    Since $\rho ^{k+1} \le \rho _4 = \epsilon _4$, (85) is also obtained.

• (f) To show

  $$\begin{aligned} \text {(f)} \le \epsilon _5 \Big (\Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^k \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2\Big ), \end{aligned}$$

  (90)

  we can rewrite

  $$\begin{aligned} \text {(f)} = 2 \rho ^{k+1} \Big \{(\varvec{\mu }^{k+1} - \varvec{\lambda }^k)^T \Big [I_{m_1 \times m_1} \otimes ({\mathbf {x}}^k)^T\Big ] P ({\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1})\Big \}. \end{aligned}$$

  (91)

  Applying (64), we have that

  $$\begin{aligned} \begin{aligned} \text {(f)}&\le 2 \rho ^{k+1} \Big (\frac{1}{2 \delta ^2} \Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^k \Vert _2^2 \\&\quad + \frac{\delta ^2}{2} {\left| \left| \left| I_{m_1 \times m_1} \otimes ({\mathbf {x}}^k)^T \right| \right| \right| }_2^2 {\left| \left| \left| P \right| \right| \right| }_2^2 \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2\Big ) \\&\le 2 \rho ^{k+1} \Big (\frac{1}{2 \delta ^2} \Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^k \Vert _2^2 + \frac{\delta ^2}{2} \Vert {\mathbf {x}}^k \Vert _2^2 \Vert P \Vert _F^2 \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2\Big ). \end{aligned} \end{aligned}$$

  (92)

  Similarly, the last inequality holds due to

  $$\begin{aligned} {\left| \left| \left| I_{m_1 \times m_1} \otimes ({\mathbf {x}}^k)^T \right| \right| \right| }_2^2 = {\left| \left| \left| I_{m_1 \times m_1} \right| \right| \right| }_2^2 {\left| \left| \left| ({\mathbf {x}}^k)^T \right| \right| \right| }_2^2. \end{aligned}$$

  • If $\Vert {\mathbf {x}}^k \Vert _2 \not = 0$, then letting $\delta ^2 = \frac{1}{\Vert {\mathbf {x}}^k \Vert _2 \Vert P \Vert _F}$ yields

    $$\begin{aligned} \text {(f)} \le \rho ^{k+1} \Vert {\mathbf {x}}^k \Vert _2 \Vert P \Vert _F \Big (\Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^k \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2\Big ). \end{aligned}$$

    (93)

    Since $\rho ^{k+1} \le \rho _5({\mathbf {x}}^k) = \frac{\epsilon _5}{\Vert {\mathbf {x}}^k \Vert _2 \Vert P \Vert _F}$, we obtain (90).

  • If $\Vert {\mathbf {x}}^k \Vert _2 = 0$, then letting $\delta ^2 = 1$ yields

    $$\begin{aligned} \text {(f)} \le \rho ^{k+1} \Big (\Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^k \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2\Big ). \end{aligned}$$

    (94)

    Since $\rho ^{k+1} \le \rho _5({\mathbf {x}}^k) = \epsilon _5$, (90) is also obtained.

• (g) To show

  $$\begin{aligned} \text {(g)} \le \epsilon _6 \Big (\Vert \varvec{\lambda }^{k+1} - \varvec{\mu }^{k+1} \Vert _2^2 + \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^k \Vert _2^2\Big ), \end{aligned}$$

  (95)

  By letting $C = \left( \begin{array}{c}{\mathbf {c}}_1^T \\ \vdots \\ {\mathbf {c}}_{m_2}^T\end{array}\right)$, we can rewrite

  $$\begin{aligned} \text {(g)} = 2 \rho ^{k+1} \Big [(\varvec{\lambda }^{k+1} - \varvec{\mu }^{k+1})^T C ({\mathbf {v}}^{k+1} - {\mathbf {u}}^k)\Big ]. \end{aligned}$$

  (96)

  Applying (64), we have that

  $$\begin{aligned} \begin{aligned} \text {(g)} \le&2 \rho ^{k+1} \Big (\frac{1}{2 \delta ^2} \Vert \varvec{\lambda }^{k+1} - \varvec{\mu }^{k+1} \Vert _2^2 + \frac{\delta ^2}{2} {\left| \left| \left| C \right| \right| \right| }_2^2 \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^k \Vert _2^2\Big ) \\ \le&2 \rho ^{k+1} \Big (\frac{1}{2 \delta ^2} \Vert \varvec{\lambda }^{k+1} - \varvec{\mu }^{k+1} \Vert _2^2 + \frac{\delta ^2}{2} \Vert C \Vert _F^2 \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^k \Vert _2^2\Big ). \end{aligned} \end{aligned}$$

  (97)

  • If $\Vert C \Vert _F \not = 0$, then letting $\delta ^2 = \frac{1}{\Vert C \Vert _F}$ yields

    $$\begin{aligned} \text {(g)} \le \rho ^{k+1} \Vert C \Vert _F \Big (\Vert \varvec{\lambda }^{k+1} - \varvec{\mu }^{k+1} \Vert _2^2 + \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^k \Vert _2^2\Big ). \end{aligned}$$

    (98)

    Since $\rho ^{k+1} \le \rho _6 = \frac{\epsilon _6}{\Vert C \Vert _F}$, we obtain (95).

  • If $\Vert C \Vert _F = 0$, then letting $\delta ^2 = 1$ yields

    $$\begin{aligned} \text {(g)} \le \rho ^{k+1} \Big (\Vert \varvec{\lambda }^{k+1} - \varvec{\mu }^{k+1} \Vert _2^2 + \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^k \Vert _2^2\Big ). \end{aligned}$$

    (99)

    Since $\rho ^{k+1} \le \rho _6 = \epsilon _6$, (95) is also obtained.

• (h) Next, we want to show that

  $$\begin{aligned} \text {(h)} \le \epsilon _6 \Big (\Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^k \Vert _2^2 + \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^{k+1} \Vert _2^2\Big ). \end{aligned}$$

  (100)

  Similarly, we can rewrite

  $$\begin{aligned} \text {(h)} = 2 \rho ^{k+1} \Big [(\varvec{\mu }^{k+1} - \varvec{\lambda }^k)^T C ({\mathbf {v}}^{k+1} - {\mathbf {u}}^{k+1})\Big ]. \end{aligned}$$

  (101)

  Applying (64) on the above equality leads to

  $$\begin{aligned} \begin{aligned} \text {(h)} \le&2 \rho ^{k+1} \Big (\frac{1}{2 \delta ^2} \Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^k \Vert _2^2 + \frac{\delta ^2}{2} {\left| \left| \left| C \right| \right| \right| }_2^2 \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^{k+1} \Vert _2^2\Big ) \\ \le&2 \rho ^{k+1} \Big (\frac{1}{2 \delta ^2} \Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^k \Vert _2^2 + \frac{\delta ^2}{2} \Vert C \Vert _F^2 \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^{k+1} \Vert _2^2\Big ). \end{aligned} \end{aligned}$$

  (102)

  • If $\Vert C \Vert _F \not = 0$, then letting $\delta ^2 = \frac{1}{\Vert C \Vert _F}$ yields

    $$\begin{aligned} \text {(h)} \le \rho ^{k+1} \Vert C \Vert _F \Big (\Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^k \Vert _2^2 + \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^{k+1} \Vert _2^2\Big ). \end{aligned}$$

    (103)

    Since $\rho ^{k+1} \le \rho _6 = \frac{\epsilon _6}{\Vert C \Vert _F}$, we obtain (100).

  • If $\Vert C \Vert _F = 0$, then letting $\delta ^2 = 1$ yields

    $$\begin{aligned} \text {(h)} \le \rho ^{k+1} \Big (\Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^k \Vert _2^2 + \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^{k+1} \Vert _2^2\Big ). \end{aligned}$$

    (104)

    Since $\rho ^{k+1} \le \rho _6 = \epsilon _6$, (100) is also obtained.

• (i) To show

  $$\begin{aligned} \text {(i)} \le \epsilon _7 \Big (\Vert \varvec{\gamma }^{k+1} - \varvec{\nu }^{k+1} \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ), \end{aligned}$$

  (105)

  we apply (64) on the rewriting of (i), which leads to

  $$\begin{aligned} \begin{aligned} \text {(i)} \le&2 \rho ^{k+1} \Big (\frac{1}{2 \delta ^2} \Vert \varvec{\gamma }^{k+1} - \varvec{\nu }^{k+1} \Vert _2^2 + \frac{\delta ^2}{2} {\left| \left| \left| A \right| \right| \right| }_2^2 \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ) \\ \le&2 \rho ^{k+1} \Big (\frac{1}{2 \delta ^2} \Vert \varvec{\gamma }^{k+1} - \varvec{\nu }^{k+1} \Vert _2^2 + \frac{\delta ^2}{2} \Vert A \Vert _F^2 \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ). \end{aligned} \end{aligned}$$

  (106)

  • If $\Vert A \Vert _F \not = 0$, then letting $\delta ^2 = \frac{1}{\Vert A \Vert _F}$ yields

    $$\begin{aligned} \text {(i)} \le \rho ^{k+1} \Vert A \Vert _F \Big (\Vert \varvec{\gamma }^{k+1} - \varvec{\nu }^{k+1} \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ). \end{aligned}$$

    (107)

    Since $\rho ^{k+1} \le \rho _7 = \frac{\epsilon _7}{\Vert A \Vert _F}$, we obtain (105).

  • If $\Vert A \Vert _F = 0$, then letting $\delta ^2 = 1$ yields

    $$\begin{aligned} \text {(i)} \le \rho ^{k+1} \Big (\Vert \varvec{\gamma }^{k+1} - \varvec{\nu }^{k+1} \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2\Big ). \end{aligned}$$

    (108)

    Since $\rho ^{k+1} \le \rho _7 = \epsilon _7$, (105) is also obtained.

• (j) Similarly, to show

  $$\begin{aligned} \text {(j)} \le \epsilon _7 \Big (\Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^k \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2\Big ). \end{aligned}$$

  (109)

  we apply (64) on the rewriting of (j), which yields

  $$\begin{aligned} \begin{aligned} \text {(j)} \le&2 \rho ^{k+1} \Big (\frac{1}{2 \delta ^2} \Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^k \Vert _2^2 + \frac{\delta ^2}{2} {\left| \left| \left| A \right| \right| \right| }_2^2 \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2\Big ) \\ \le&2 \rho ^{k+1} \Big (\frac{1}{2 \delta ^2} \Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^k \Vert _2^2 + \frac{\delta ^2}{2} \Vert A \Vert _F^2 \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2\Big ). \end{aligned} \end{aligned}$$

  (110)

  • If $\Vert A \Vert _F \not = 0$, then letting $\delta ^2 = \frac{1}{\Vert A \Vert _F}$ yields

    $$\begin{aligned} \text {(j)} \le \rho ^{k+1} \Vert A \Vert _F \Big (\Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^k \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2\Big ). \end{aligned}$$

    (111)

    Since $\rho ^{k+1} \le \rho _7 = \frac{\epsilon _7}{\Vert A \Vert _F}$, we obtain (109).

  • If $\Vert A \Vert _F = 0$, then letting $\delta ^2 = 1$ yields

    $$\begin{aligned} \text {(j)} \le \rho ^{k+1} \Big (\Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^k \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2\Big ). \end{aligned}$$

    (112)

    Since $\rho ^{k+1} \le \rho _7 = \epsilon _7$, (109) is also obtained.

• (k) Next, to show

  $$\begin{aligned} \text {(k)} \le \epsilon _8 \Big (\Vert \varvec{\gamma }^{k+1} - \varvec{\nu }^{k+1} \Vert _2^2 + \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^k \Vert _2^2\Big ), \end{aligned}$$

  (113)

  we apply (64) on the rewriting of (k):

  $$\begin{aligned} \begin{aligned} \text {(k)} \le&2 \rho ^{k+1} \Big (\frac{1}{2 \delta ^2} \Vert \varvec{\gamma }^{k+1} - \varvec{\nu }^{k+1} \Vert _2^2 + \frac{\delta ^2}{2} {\left| \left| \left| B \right| \right| \right| }_2^2 \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^k \Vert _2^2\Big ) \\ \le&2 \rho ^{k+1} \Big (\frac{1}{2 \delta ^2} \Vert \varvec{\gamma }^{k+1} - \varvec{\nu }^{k+1} \Vert _2^2 + \frac{\delta ^2}{2} \Vert B \Vert _F^2 \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^k \Vert _2^2\Big ). \end{aligned} \end{aligned}$$

  (114)

  • If $\Vert B \Vert _F \not = 0$, then letting $\delta ^2 = \frac{1}{\Vert B \Vert _F}$ yields

    $$\begin{aligned} \text {(k)} \le \rho ^{k+1} \Vert B \Vert _F \Big (\Vert \varvec{\gamma }^{k+1} - \varvec{\nu }^{k+1} \Vert _2^2 + \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^k \Vert _2^2\Big ). \end{aligned}$$

    (115)

    Since $\rho ^{k+1} \le \rho _8 = \frac{\epsilon _8}{\Vert B \Vert _F}$, we obtain (113).

  • If $\Vert B \Vert _F = 0$, then letting $\delta ^2 = 1$ yields

    $$\begin{aligned} \text {(k)} \le \rho ^{k+1} \Big (\Vert \varvec{\gamma }^{k+1} - \varvec{\nu }^{k+1} \Vert _2^2 + \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^k \Vert _2^2\Big ). \end{aligned}$$

    (116)

    Since $\rho ^{k+1} \le \rho _8 = \epsilon _8$, (113) is also obtained.

• (l) Last, to show

  $$\begin{aligned} \text {(l)} \le \epsilon _8 \Big (\Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^k \Vert _2^2 + \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^{k+1} \Vert _2^2\Big ), \end{aligned}$$

  (117)

  we apply (64) on the rewriting of (l):

  $$\begin{aligned} \begin{aligned} \text {(l)} \le&2 \rho ^{k+1} \Big (\frac{1}{2 \delta ^2} \Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^k \Vert _2^2 + \frac{\delta ^2}{2} {\left| \left| \left| B \right| \right| \right| }_2^2 \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^{k+1} \Vert _2^2\Big ) \\ \le&2 \rho ^{k+1} \Big (\frac{1}{2 \delta ^2} \Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^k \Vert _2^2 + \frac{\delta ^2}{2} \Vert B \Vert _F^2 \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^{k+1} \Vert _2^2\Big ). \end{aligned} \end{aligned}$$

  (118)

  • If $\Vert B \Vert _F \not = 0$, then letting $\delta ^2 = \frac{1}{\Vert B \Vert _F}$ yields

    $$\begin{aligned} \text {(l)} \le \rho ^{k+1} \Vert B \Vert _F \Big (\Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^k \Vert _2^2 + \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^{k+1} \Vert _2^2\Big ). \end{aligned}$$

    (119)

    Since $\rho ^{k+1} \le \rho _8 = \frac{\epsilon _8}{\Vert B \Vert _F}$, we obtain (117).

  • If $\Vert B \Vert _F = 0$, then letting $\delta ^2 = 1$ yields

    $$\begin{aligned} \text {(l)} \le \rho ^{k+1} \Big (\Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^k \Vert _2^2 + \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^{k+1} \Vert _2^2\Big ). \end{aligned}$$

    (120)

    Since $\rho ^{k+1} \le \rho _8 = \epsilon _8$, (117) is also obtained.

The summation of terms $\text {(a)}$ to $\text {(l)}$ can now be bounded as:

$$\begin{aligned} \begin{aligned}&\text {(a)} + \sum _{i=1}^{m_1} \text {(b)}_i + \text {(c)} + \text {(d)} + \text {(e)} + \text {(f)} + \text {(g)} + \text {(h)} + \text {(i)} + \text {(j)} + \text {(k)} + \text {(l)} \\&\quad \le (\epsilon _1 + \epsilon _2 + \epsilon _4 + \epsilon _5 + \epsilon _7) \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2 \\&\qquad + (\epsilon _1 + \epsilon _2 + \epsilon _3 + \epsilon _4 + \epsilon _7) \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2 \\&\qquad + (\epsilon _6 + \epsilon _8) \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^{k+1} \Vert _2^2 + (\epsilon _6 + \epsilon _8) \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^k \Vert _2^2 \\&\qquad + (\epsilon _3 + \epsilon _4 + \epsilon _6) \Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^{k+1} \Vert _2^2 + (\epsilon _4 + \epsilon _5 + \epsilon _6) \Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^k \Vert _2^2 \\&\qquad + (\epsilon _7 + \epsilon _8) \Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^{k+1} \Vert _2^2 + (\epsilon _7 + \epsilon _8) \Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^k \Vert _2^2 \\&\quad \le (\sum _{s=1}^8 \epsilon _s) \Big [\Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2 \\&\qquad + \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^{k+1} \Vert _2^2 + \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^k \Vert _2^2 \\&\qquad + \Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^{k+1} \Vert _2^2 + \Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^k \Vert _2^2 \\&\qquad + \Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^{k+1} \Vert _2^2 + \Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^k \Vert _2^2\Big ] \\&\quad \le (1 - \epsilon _0) \Big [\Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2 \\&\qquad + \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^{k+1} \Vert _2^2 + \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^k \Vert _2^2 \\&\qquad + \Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^{k+1} \Vert _2^2 + \Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^k \Vert _2^2 \\&\qquad + \Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^{k+1} \Vert _2^2 + \Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^k \Vert _2^2\Big ]. \end{aligned} \end{aligned}$$

(121)

Substituting it back into (61), we have that for all $k \ge 0$,

$$\begin{aligned}&\Vert {\mathbf {x}}^{k+1} - {\mathbf {x}}^* \Vert _2^2 + \Vert {\mathbf {u}}^{k+1} - {\mathbf {u}}^* \Vert _2^2 + \Vert \varvec{\lambda }^{k+1} - \varvec{\lambda }^* \Vert _2^2 + \Vert \varvec{\gamma }^{k+1} - \varvec{\gamma }^* \Vert _2^2 \nonumber \\&\quad \le \Vert {\mathbf {x}}^k - {\mathbf {x}}^* \Vert _2^2 + \Vert {\mathbf {u}}^k - {\mathbf {u}}^* \Vert _2^2 + \Vert \varvec{\lambda }^k - \varvec{\lambda }^* \Vert _2^2 + \Vert \varvec{\gamma }^k - \varvec{\gamma }^* \Vert _2^2 \nonumber \\&\qquad - \epsilon _0 \Big [\Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2 + \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2 + \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^{k+1} \Vert _2^2 + \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^k \Vert _2^2 \nonumber \\&\qquad + \Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^{k+1} \Vert _2^2 + \Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^k \Vert _2^2 + \Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^{k+1} \Vert _2^2 + \Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^k \Vert _2^2\Big ], \end{aligned}$$

(122)

which implies for all $k \ge 0$:

$$\begin{aligned} 0 \le&\Vert {\mathbf {x}}^{k+1} - {\mathbf {x}}^* \Vert _2^2 + \Vert {\mathbf {u}}^{k+1} - {\mathbf {u}}^* \Vert _2^2 + \Vert \varvec{\lambda }^{k+1} - \varvec{\lambda }^* \Vert _2^2 + \Vert \varvec{\gamma }^{k+1} - \varvec{\gamma }^* \Vert _2^2 \nonumber \\ \le&\Vert {\mathbf {x}}^k - {\mathbf {x}}^* \Vert _2^2 + \Vert {\mathbf {u}}^k - {\mathbf {u}}^* \Vert _2^2 + \Vert \varvec{\lambda }^k - \varvec{\lambda }^* \Vert _2^2 + \Vert \varvec{\gamma }^k - \varvec{\gamma }^* \Vert _2^2 \nonumber \\ \le&\Vert {\mathbf {x}}^{k-1} - {\mathbf {x}}^* \Vert _2^2 + \Vert {\mathbf {u}}^{k-1} - {\mathbf {u}}^* \Vert _2^2 + \Vert \varvec{\lambda }^{k-1} - \varvec{\lambda }^* \Vert _2^2 + \Vert \varvec{\gamma }^{k-1} - \varvec{\gamma }^* \Vert _2^2 \nonumber \\ \le&\cdots \le \Vert {\mathbf {x}}^0 - {\mathbf {x}}^* \Vert _2^2 + \Vert {\mathbf {u}}^0 - {\mathbf {u}}^* \Vert _2^2 + \Vert \varvec{\lambda }^0 - \varvec{\lambda }^* \Vert _2^2 + \Vert \varvec{\gamma }^0 - \varvec{\gamma }^* \Vert _2^2. \end{aligned}$$

(123)

It further implies that the sequence $\{\Vert {\mathbf {x}}^k - {\mathbf {x}}^* \Vert _2^2 + \Vert {\mathbf {u}}^k - {\mathbf {u}}^* \Vert _2^2 + \Vert \varvec{\lambda }^k - \varvec{\lambda }^* \Vert _2^2 + \Vert \varvec{\gamma }^k - \varvec{\gamma }^* \Vert _2^2\}$ is monotonically decreasing and bounded below by 0; hence the sequence must be convergent to a limit, denoted by $\xi$:

$$\begin{aligned} \lim _{k \rightarrow +\infty } \Vert {\mathbf {x}}^k - {\mathbf {x}}^* \Vert _2^2 + \Vert {\mathbf {u}}^k - {\mathbf {u}}^* \Vert _2^2 + \Vert \varvec{\lambda }^k - \varvec{\lambda }^* \Vert _2^2 + \Vert \varvec{\gamma }^k - \varvec{\gamma }^* \Vert _2^2 = \xi . \end{aligned}$$

(124)

Taking the limit on both sides of (122) yields:

$$\begin{aligned} \begin{aligned}&\lim _{k \rightarrow +\infty } \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1} \Vert _2^2 = 0, \quad&\lim _{k \rightarrow +\infty } \Vert {\mathbf {y}}^{k+1} - {\mathbf {x}}^k \Vert _2^2 = 0, \\&\lim _{k \rightarrow +\infty } \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^{k+1} \Vert _2^2 = 0, \quad&\lim _{k \rightarrow +\infty } \Vert {\mathbf {v}}^{k+1} - {\mathbf {u}}^k \Vert _2^2 = 0, \\&\lim _{k \rightarrow +\infty } \Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^{k+1} \Vert _2^2 = 0, \quad&\lim _{k \rightarrow +\infty } \Vert \varvec{\mu }^{k+1} - \varvec{\lambda }^k \Vert _2^2 = 0, \\&\lim _{k \rightarrow +\infty } \Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^{k+1} \Vert _2^2 = 0, \quad&\lim _{k \rightarrow +\infty } \Vert \varvec{\nu }^{k+1} - \varvec{\gamma }^k \Vert _2^2 = 0. \end{aligned} \end{aligned}$$

(125)

Additionally, (124) also implies that $\{({\mathbf {x}}^k, {\mathbf {u}}^k, \varvec{\lambda }^k, \varvec{\gamma }^k)\}$ is a bounded sequence, and there exists a sub-sequence $\{({\mathbf {x}}^{k_j}, {\mathbf {u}}^{k_j}, \varvec{\lambda }^{k_j}, \varvec{\gamma }^{k_j})\}$ that converges to a limit point $({\mathbf {x}}^{\infty }, {\mathbf {u}}^{\infty }, \varvec{\lambda }^{\infty }, \varvec{\gamma }^{\infty })$. We next show that the limit point is indeed a saddle point and is also the unique limit point of $\{({\mathbf {x}}^k, {\mathbf {u}}^k, \varvec{\lambda }^k, \varvec{\gamma }^k)\}$. Given any ${\mathbf {x}} \in {\mathbb {X}}$ and ${\mathbf {u}} \in {\mathbb {R}}^{n_2}$, we have:

$$\begin{aligned} \begin{aligned}&2 \rho ^{k+1} \big [{\mathcal {L}}({\mathbf {x}}^{k+1}, {\mathbf {u}}^{k+1}, \varvec{\mu }^{k+1}, \varvec{\nu }^{k+1}) - {\mathcal {L}}({\mathbf {x}}, {\mathbf {u}}, \varvec{\mu }^{k+1}, \varvec{\nu }^{k+1})\big ] \\&\quad = 2 \rho ^{k+1} \Big \{\big [\frac{1}{2} ({\mathbf {x}}^{k+1})^T P_0 {\mathbf {x}}^{k+1} - \frac{1}{2} {\mathbf {x}}^T P_0 {\mathbf {x}}\big ] + {\mathbf {q}}_0^T ({\mathbf {x}}^{k+1} - {\mathbf {x}}) + {\mathbf {c}}_0^T ({\mathbf {u}}^{k+1} - {\mathbf {u}}) \\&\qquad + \sum _{i=1}^{m_1} \mu _i^{k+1} \big [\frac{1}{2} ({\mathbf {x}}^{k+1})^T P_i {\mathbf {x}}^{k+1} - \frac{1}{2} {\mathbf {x}}^T P_i {\mathbf {x}}\big ] + \sum _{i=1}^{m_1} \mu _i^{k+1} {\mathbf {q}}_i^T ({\mathbf {x}}^{k+1} - {\mathbf {x}}) \\&\qquad + \sum _{i=1}^{m_1} \mu _i^{k+1} {\mathbf {c}}_i^T ({\mathbf {u}}^{k+1} - {\mathbf {u}})\\&\qquad + \varvec{\nu }^{k+1} A({\mathbf {x}}^{k+1} - {\mathbf {x}}) + \varvec{\nu }^{k+1} B({\mathbf {u}}^{k+1} - {\mathbf {u}})\Big \} \\&\quad = \underbrace{2 \rho ^{k+1}\Big [- \frac{1}{2}({\mathbf {x}}^{k+1} - {\mathbf {x}})^T P_0 ({\mathbf {x}}^{k+1} - {\mathbf {x}}) - \sum _{i=1}^{m_1} \mu _i^{k+1} \frac{1}{2}({\mathbf {x}}^{k+1} - {\mathbf {x}})^T P_i ({\mathbf {x}}^{k+1} - {\mathbf {x}})\Big ]}_{(\varDelta )} \\&\qquad + 2 \rho ^{k+1} \Big [(P_0 {\mathbf {x}}^{k+1} + {\mathbf {q}}_0)^T ({\mathbf {x}}^{k+1} - {\mathbf {x}}) + {\mathbf {c}}_0^T ({\mathbf {u}}^{k+1} - {\mathbf {u}}) \\&\qquad + \sum _{i=1}^{m_1} \mu _i^{k+1} (P_i {\mathbf {x}}^{k+1} + {\mathbf {q}}_i)^T ({\mathbf {x}}^{k+1} - {\mathbf {x}}) + \sum _{i=1}^{m_1} \mu _i^{k+1} {\mathbf {c}}_i^T ({\mathbf {u}}^{k+1} - {\mathbf {u}}) \\&\qquad + \varvec{\nu }^{k+1} A({\mathbf {x}}^{k+1} - {\mathbf {x}}) + \varvec{\nu }^{k+1} B({\mathbf {u}}^{k+1} - {\mathbf {u}})\Big ] \\&\quad \le 2 \rho ^{k+1} \Big [(P_0 {\mathbf {x}}^{k+1} + {\mathbf {q}}_0)^T ({\mathbf {x}}^{k+1} - {\mathbf {x}}) + {\mathbf {c}}_0^T ({\mathbf {u}}^{k+1} - {\mathbf {u}}) \\&\qquad + \sum _{i=1}^{m_1} \mu _i^{k+1} (P_i {\mathbf {x}}^{k+1} + {\mathbf {q}}_i)^T ({\mathbf {x}}^{k+1} - {\mathbf {x}}) + \sum _{i=1}^{m_1} \mu _i^{k+1} {\mathbf {c}}_i^T ({\mathbf {u}}^{k+1} - {\mathbf {u}}) \\&\qquad + \varvec{\nu }^{k+1} A({\mathbf {x}}^{k+1} - {\mathbf {x}}) + \varvec{\nu }^{k+1} B({\mathbf {u}}^{k+1} - {\mathbf {u}})\Big ]. \end{aligned} \end{aligned}$$

(126)

The positive semi-definiteness of each $P_i$ for all $i = 0, 1, \dots , m$ guarantees the non-positiveness of $(\varDelta )$, which makes the last inequality hold. Applying Lemma 1 on (22c) with $\widehat{{\mathbf {z}}} = ({\mathbf {x}}^{k+1}, {\mathbf {u}}^{k+1})$, $\bar{{\mathbf {z}}} = ({\mathbf {x}}^k, {\mathbf {u}}^k)$ and ${\mathbf {z}} = ({\mathbf {x}}, {\mathbf {u}})$ yields:

$$\begin{aligned}&2 \rho ^{k+1}\Big \{(P_0 {\mathbf {y}}^{k+1} + {\mathbf {q}}_0)^T {\mathbf {x}}^{k+1} + {\mathbf {c}}_0^T {\mathbf {u}}^{k+1} + r_0 \nonumber \\&\qquad + \sum _{i=1}^{m_1} \mu _i^{k+1} \Big [(P_i {\mathbf {y}}^{k+1} + {\mathbf {q}}_i)^T {\mathbf {x}}^{k+1} + {\mathbf {c}}_i^T {\mathbf {u}}^{k+1} + r_i\Big ] \nonumber \\&\qquad + \varvec{\nu }^{k+1} (A {\mathbf {x}}^{k+1} + B {\mathbf {u}}^{k+1} - {\mathbf {b}})\Big \} \nonumber \\&\qquad - 2 \rho ^{k+1}\Big \{(P_0 {\mathbf {y}}^{k+1} + {\mathbf {q}}_0)^T {\mathbf {x}} + {\mathbf {c}}_0^T {\mathbf {u}} + r_0 \nonumber \\&\qquad + \sum _{i=1}^m \mu _i^{k+1} \Big [(P_i {\mathbf {y}}^{k+1} + {\mathbf {q}}_i)^T {\mathbf {x}} + {\mathbf {c}}_i^T {\mathbf {u}} + r_i\Big ] \nonumber \\&\qquad + \varvec{\nu }^{k+1} (A {\mathbf {x}} + B {\mathbf {u}} - {\mathbf {b}})\Big \} \nonumber \\&\quad \le \Vert {\mathbf {x}}^k - {\mathbf {x}} \Vert _2^2 - \Vert {\mathbf {x}}^{k+1} - {\mathbf {x}} \Vert _2^2 - \Vert {\mathbf {x}}^{k+1} - {\mathbf {x}}^k \Vert _2^2 \nonumber \\&\qquad + \Vert {\mathbf {u}}^k - {\mathbf {u}} \Vert _2^2 - \Vert {\mathbf {u}}^{k+1} - {\mathbf {u}} \Vert _2^2 - \Vert {\mathbf {u}}^{k+1} - {\mathbf {u}}^k \Vert _2^2 \nonumber \\&\qquad \le \big (\Vert {\mathbf {x}}^k - {\mathbf {x}}^{k+1} \Vert _2^2 + \Vert {\mathbf {x}}^{k+1} - {\mathbf {x}} \Vert _2^2\big ) - \Vert {\mathbf {x}}^{k+1} - {\mathbf {x}} \Vert _2^2 - \Vert {\mathbf {x}}^{k+1} - {\mathbf {x}}^k \Vert _2^2 \nonumber \\&\qquad + \big (\Vert {\mathbf {u}}^k - {\mathbf {u}}^{k+1} \Vert _2^2 + \Vert {\mathbf {u}}^{k+1} - {\mathbf {u}} \Vert _2^2\big ) - \Vert {\mathbf {u}}^{k+1} - {\mathbf {u}} \Vert _2^2 - \Vert {\mathbf {u}}^{k+1} - {\mathbf {u}}^k \Vert _2^2 = 0. \end{aligned}$$

(127)

Adding the above two inequalities yields

$$\begin{aligned}&2 \rho ^{k+1} \Big [{\mathcal {L}}({\mathbf {x}}^{k+1}, {\mathbf {u}}^{k+1}, \varvec{\mu }^{k+1}, \varvec{\nu }^{k+1}) - {\mathcal {L}}({\mathbf {x}}, {\mathbf {u}}, \varvec{\mu }^{k+1}, \varvec{\nu }^{k+1})\Big ] \nonumber \\&\quad + 2 \rho ^{k+1} \Big \{({\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1})^T P_0 ({\mathbf {x}}^{k+1} - {\mathbf {x}}) \nonumber \\&\quad + \sum _{i=1}^m \mu _i^{k+1} \big [({\mathbf {y}}^{k+1} - {\mathbf {x}}^{k+1})^T P_i ({\mathbf {x}}^{k+1} - {\mathbf {x}})\big ]\Big \} \le 0. \end{aligned}$$

(128)

Taking the limits over an appropriate sub-sequence $\{k_j\}$ on both sides and using (125), we have:

$$\begin{aligned} {\mathcal {L}}({\mathbf {x}}^{\infty }, {\mathbf {u}}^{\infty }, \varvec{\lambda }^{\infty }, \varvec{\gamma }^{\infty }) \le {\mathcal {L}}({\mathbf {x}}, {\mathbf {u}}, \varvec{\lambda }^{\infty }, \varvec{\gamma }^{\infty }), \quad \forall {\mathbf {x}} \in {\mathbb {X}}, \forall {\mathbf {u}} \in {\mathbb {R}}^{n_2}. \end{aligned}$$

(129)

Similarly, given any $\varvec{\lambda } \in {\mathbb {R}}_+^{m_1}$ and $\varvec{\gamma } \in {\mathbb {R}}^{m_2}$, applying Lemma 1 on (22d) with $\widehat{{\mathbf {z}}} = (\varvec{\lambda }^{k+1}, \varvec{\gamma }^{k+1})$, $\bar{{\mathbf {z}}} = (\varvec{\lambda }^k, \varvec{\gamma }^k)$ and ${\mathbf {z}} = (\varvec{\lambda }, \varvec{\gamma })$ yields

$$\begin{aligned}&2 \rho ^{k+1}\big [{\mathcal {L}}({\mathbf {y}}^{k+1}, {\mathbf {v}}^{k+1}, \varvec{\lambda }, \varvec{\gamma }) - {\mathcal {L}}({\mathbf {y}}^{k+1}, {\mathbf {v}}^{k+1}, \varvec{\lambda }^{k+1}, \varvec{\gamma }^{k+1})\big ] \nonumber \\&\quad \le \Vert \varvec{\lambda }^k - \varvec{\lambda } \Vert _2^2 - \Vert \varvec{\lambda }^{k+1} - \varvec{\lambda } \Vert _2^2 - \Vert \varvec{\lambda }^{k+1} - \varvec{\lambda }^k \Vert _2^2 \nonumber \\&\qquad + \Vert \varvec{\gamma }^k - \varvec{\gamma } \Vert _2^2 - \Vert \varvec{\gamma }^{k+1} - \varvec{\gamma } \Vert _2^2 - \Vert \varvec{\gamma }^{k+1} - \varvec{\gamma }^k \Vert _2^2 \nonumber \\&\quad \le \big (\Vert \varvec{\lambda }^k - \varvec{\lambda }^{k+1} \Vert _2^2 - \Vert \varvec{\lambda }^{k+1} + \varvec{\lambda } \Vert _2^2\big ) - \Vert \varvec{\lambda }^{k+1} - \varvec{\lambda } \Vert _2^2 - \Vert \varvec{\lambda }^{k+1} - \varvec{\lambda }^k \Vert _2^2 \nonumber \\&\qquad + \big (\Vert \varvec{\gamma }^k - \varvec{\gamma }^{k+1} \Vert _2^2 - \Vert \varvec{\gamma }^{k+1} + \varvec{\gamma } \Vert _2^2\big ) - \Vert \varvec{\gamma }^{k+1} - \varvec{\gamma } \Vert _2^2 - \Vert \varvec{\gamma }^{k+1} - \varvec{\gamma }^k \Vert _2^2 = 0. \end{aligned}$$

(130)

Taking the limits over an appropriate sub-sequence $\{k_j\}$ on both sides and using (125), we have:

$$\begin{aligned} {\mathcal {L}}({\mathbf {x}}^{\infty }, {\mathbf {u}}^{\infty }, \varvec{\lambda }, \varvec{\gamma }) \le {\mathcal {L}}({\mathbf {x}}^{\infty }, {\mathbf {u}}^{\infty }, \varvec{\lambda }^{\infty }, \varvec{\gamma }^{\infty }), \quad \forall \varvec{\lambda } \in {\mathbb {R}}_+^{m_1}, \forall \varvec{\gamma } \in {\mathbb {R}}^{m_2}. \end{aligned}$$

(131)

Therefore, we show that $({\mathbf {x}}^{\infty }, {\mathbf {u}}^{\infty }, \varvec{\lambda }^{\infty }, \varvec{\gamma }^{\infty })$ is indeed a saddle point of the Lagrangian function ${\mathcal {L}}({\mathbf {x}}, {\mathbf {u}}, \varvec{\lambda }, \varvec{\gamma })$. Then (124) implies that

$$\begin{aligned} \lim _{k \rightarrow +\infty } \Vert {\mathbf {x}}^k - {\mathbf {x}}^{\infty } \Vert _2^2 + \Vert {\mathbf {u}}^k - {\mathbf {u}}^{\infty } \Vert _2^2 + \Vert \varvec{\lambda }^k - \varvec{\lambda }^{\infty } \Vert _2^2 + \Vert \varvec{\gamma }^k - \varvec{\gamma }^{\infty } \Vert _2^2 = \xi . \end{aligned}$$

(132)

Since we have argued (after Eq. (125)) that there exists a bounded sequence of $\{({\mathbf {x}}^k, {\mathbf {u}}^k, \varvec{\lambda }^k, \varvec{\gamma }^k)\}$ that converges to $({\mathbf {x}}^{\infty }, {\mathbf {u}}^{\infty }, \varvec{\lambda }^{\infty }, \varvec{\gamma }^{\infty })$; that is, there exists $\{k_j\}$ such that $\lim _{k_j \rightarrow +\infty } \Vert {\mathbf {x}}^{k_j} - {\mathbf {x}}^{\infty } \Vert _2^2 + \Vert {\mathbf {u}}^{k_j} - {\mathbf {u}}^{\infty } \Vert _2^2 + \Vert \varvec{\lambda }^{k_j} - \varvec{\lambda }^{\infty } \Vert _2^2 + \Vert \varvec{\gamma }^{k_j} - \varvec{\gamma }^{\infty } \Vert _2^2 = 0$, which then implies that $\xi = 0$. Therefore, we show that $\{({\mathbf {x}}^k, {\mathbf {u}}^k, \varvec{\lambda }^k, \varvec{\gamma }^k)\}$ converges globally to a saddle point $({\mathbf {x}}^{\infty }, {\mathbf {u}}^{\infty }, \varvec{\lambda }^{\infty }, \varvec{\gamma }^{\infty })$. $\Box$