A symmetric version of the generalized alternating direction method of multipliers for two-block separable convex programming

This paper introduces a symmetric version of the generalized alternating direction method of multipliers for two-block separable convex programming with linear equality constraints, which inherits the superiorities of the classical alternating direction method of multipliers (ADMM), and which extends the feasible set of the relaxation factor α of the generalized ADMM to the infinite interval \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$[1,+\infty)$\end{document}[1,+∞). Under the conditions that the objective function is convex and the solution set is nonempty, we establish the convergence results of the proposed method, including the global convergence, the worst-case \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$\mathcal{O}(1/k)$\end{document}O(1/k) convergence rate in both the ergodic and the non-ergodic senses, where k denotes the iteration counter. Numerical experiments to decode a sparse signal arising in compressed sensing are included to illustrate the efficiency of the new method.


Introduction
We consider the two-block separable convex programming with linear equality constraints, where the objective function is the sum of two individual functions with decoupled variables: where θ i : R n i → R (i = , ) are closed proper convex functions; A i ∈ R l×n i (i = , ) and b ∈ R l , and X i ⊆ R n i (i = , ) are given nonempty closed convex sets. The linear constrained convex problem () is a unified framework of many problems arising in real world, including compressed sensing, image restoration, and statistical learning, and so forth (see, for example, [-]). An important special case of () is the following linear inverse problem: where A ∈ R m×n and y ∈ R m are given matrix and vector, μ >  is a regularization parameter and x  is the  -norm of a vector x defined as x  = n i= |x i |. Then setting x  := Axy, x  := x, () can be converted into the following two-block separable convex programming: which is a special case of problem () with the following specifications: A  := A, b := y.

Existing algorithms
In their seminal work, Glowinski et al. [] and Gabay et al. [] independently developed the alternating direction method of multipliers (ADMM), which is an influential first-order method for solving problem (). ADMM can be regarded as an application of the Douglas-Rachford splitting method (DRSM) [] to the dual of (), or a special case of the proximal point algorithm (PPA) [, ] in the cyclic sense. We refer to [] for a more detailed relationship. With any initial vectors x   ∈ X  , λ  ∈ R l , the iterative scheme of ADMM reads where λ ∈ R l is the Lagrangian multiplier and β >  is a penalty parameter. The main characteristics of ADMM are that it in full exploits the separable structure of problem (), and that it updates the variables x  , x  , λ in an alternating order by solving a series of low-dimensional sub-problems with only one unknown variable.
In the past few decades, ADMM has received a revived interest, and it has become a research focus in optimization community, especially in the (non)convex optimization. Many efficient ADMM-type methods have been developed, including the proximal ADMM [, ], the generalized ADMM [], the symmetric ADMM [], the inertial ADMM [], and some proximal ADMM-type methods [-]. Specifically, the proximal ADMM attaches some proximal terms to the sub-problems of ADMM (). The generalized ADMM updates the variables x  and λ by including a relaxation factor α ∈ (, ), and α ∈ (, ) is often advantageous to speed up its performance. The symmetric ADMM updates the Lagrangian multiplier λ twice at each iteration and includes two relaxation factors α ∈ (, ), β ∈ (, ). Recent researches of the symmetric ADMM can be found in [, , ]. The inertial ADMM unifies the basic ideas of the inertial PPA and ADMM, which utilizes the latest two iterates to generate the new iterate, therefore it can be viewed as a multistep method. For the proximal ADMM, the objective functions of its sub-problems are often strongly convex, which are often easier to be solved than those of (). However, a new challenge has arisen for the proximal ADMM-type methods. It is how to choose a proper proximal matrix. In fact, most proximal ADMM-type methods need to estimate the matrix norm A i A i (i = , ), which demands lots of calculations, especially for large n i (i = , ). Quite recently, some customized Douglas-Rachford splitting algorithms [-], and the proximal ADMM-type methods with indefinite proximal regularization are developed [, ], which dissolve the above problem to some extent. All the above mentioned ADMM-type methods are generalizations of the classical ADMM, because they all reduce to the iterative scheme () by choosing some special parameters. For more new development of the ADMM-type methods, including the convergence rate, acceleration techniques, its generalization for solving multi-block separable convex programming and nonconvex, nonsmooth programming, we refer to [-].

Contributions and organization
We are going to further study the generalized ADMM. Note that the first sub-problem in the generalized ADMM is irrelevant to the relaxation factor α. That is, the updating formula for x  does not incorporate the relaxation factor α explicitly. Furthermore, α ∈ (, ) is often advantageous for the generalized ADMM []. Therefore, in this paper, we are going to propose a new generalized ADMM, whose both sub-problems incorporate the relaxation factor α directly. The new method generalizes the method proposed in [] by relaxing the feasible set of α from the interval [, ) to the infinite interval [, +∞), and can be viewed as a symmetric version of the generalized ADMM.
The rest of the paper is organized as follows. In Section , we summarize some necessary preliminaries and characterize problem () by a mixed variational inequality problem. In Section , we describe the new symmetric version of the generalized ADMM and establish its convergence results in detail. In Section , some compressed sensing experiments are given to illustrate the efficiency of the proposed method. Some conclusions are drawn in Section .

Preliminaries
In this section, some necessary preliminaries which are useful for further discussions are presented, and to make our analysis more succinct, some positive definite or positive semidefinite block matrices are defined and their properties are investigated.
For two real matrices A ∈ R s×m , B ∈ R n×s , the Kronecker product of A and B is defined as A⊗B = (a ij B). Let · p (p = , ) denote the standard definition of p -norm; in particular, · = ·  . For any two vectors x, y ∈ R n , x, y or x y denote their inner product, and for any symmetric matrix G ∈ R n×n , the symbol G  (resp., G ) denotes that G is positive definite (resp., semi-definite). For any x ∈ R n and G , the G-norm x G of the vector x is defined as √ x Gx. The effective domain of a closed proper function f : X → (-∞, +∞] is defined as dom(f ) := {x ∈ X |f (x) < +∞}, and the symbol ri(C) denotes the set of all relative interior points of a given nonempty convex set C. Furthermore, we use the following notations: Then, for a convex function f : R n → R, we have the following basic inequality: where ∂f (y) = {ξ ∈ R n : f (ȳ) ≥ f (y) + ξ ,ȳy , for allȳ ∈ R n } denotes the subdifferential of f (·) at the point y.
Throughout the paper, we make the following standard assumptions for problem ().
Assumption . The matrices A i (i = , ) are full-column rank.
Assumption . The generalized Slater condition holds, i.e., there is a point (

The mixed variational inequality problem
Under Assumption ., it follows from Theorem . and Theorem . where Then, for the nonempty convex set X i and

Lemma . For any vector x
From the subgradient inequality (), one has Thus, where the second inequality comes from From this and Theorem .
. This completes the proof.
Remark . Based on () and Lemma ., the vector x * = (x *  , x *  ) ∈ R n  +n  is an optimal solution to problem () if and only if there exists a vector λ * ∈ R l such that Moreover, any λ * ∈ R l satisfying () is an optimal solution to the dual of problem (). Obviously, () can be written as the following mixed variational inequality problem, denoted by VI(W, F, θ ): Find a vector w * ∈ W such that The solution set of VI(W, F, θ ), denoted by W * , is nonempty by Assumption . and Remark .. It is easy to verify that the linear function F(·) is not only monotone but also satisfies the following desired property:

Three matrices and their properties
To present our analysis in a compact way, now let us define some matrices. For any R i ∈ R n i ×n i (i = , ) , set and for α ∈ [, +∞), set The above defined three matrices M, Q, H satisfy the following properties.
Proof Set t = α  -α + , which is positive for any α ∈ R. By (), we have Obviously, the first part is positive semi-definite, and we only need to prove the second part is also positive semi-definite. In fact, it can written as The middle matrix in the above expression can be further written as where ⊗ denotes the matrix Kronecker product. The matrix Kronecker product has a nice property: for any two matrices X and Y , the eigenvalue of X ⊗ Y equals the product of λ(X)λ(Y ), where λ(X) and λ(Y ) are the eigenvalues of X and Y , respectively. Therefore, we only need to show the -by- matrix Therefore, the matrix H is positive semi-definite. The proof is then complete.
Then the first assertion is proved. For (), by some simple manipulations, we obtain We now break up the proof into two cases. First, if α = , then Therefore, () holds. Second, if α ∈ (, +∞), then () The middle matrix in the above expression can be further written as the right-hand side of () is also positive semi-definite. Thus, we have Substituting () into () and by the expression of M HM, we obtain (). The lemma is proved.

Algorithm and convergence results
In this section, we first describe the symmetric version of the generalized alternating direction method of multipliers (SGADMM) for VI(W, F, θ ) formally, and then we prove its global convergence in a contraction perspective and establish its worst-case O(/k) convergence rate in both the ergodic and the non-ergodic senses step by step, where k denotes the iteration counter.
Remark . Obviously, the iterative scheme () reduces to the generalized ADMM when α = , and further reduces to () when R i =  (i = , ). That is to say, if the parameters α =  and R i =  (i = , ), then the classical ADMM is recovered. Since the convergence results of the (proximal) ADMM have been established in the literature [, , ], in the following, we only consider α ∈ (, +∞).

Global convergence
For further analysis, we need to define an auxiliary sequence {ŵ k } as follows:

Lemma . Let {λ k+ } and {λ k } be the two sequences generated by SGADMM. Then
Proof From the definition of λ k+ , we get Then () is proved. For (), we havê Therefore () is also right. This completes the proof.
Thus, based on () and (), the two sequences {w k } and {ŵ k } satisfies the following relationship: where M is defined in ().
The following lemma shows that the stopping criterion () of SGADMM is reasonable.
Proof By invoking the optimality condition of the three sub-problems in (), we have the following mixed variational inequality problems: for any w = (x  , x  , λ) ∈ W, Then, adding the above three inequalities and by (), (), we get Then by (), we obtain Then, by () (the definition of Q), the above inequality can be rewritten as and λ k = λ k+ , then by (), we have λ k+ =λ k . Thenλ k = λ k . Thus, we have Q w k -ŵ k = , which together with () implies that This indicates that the vectorŵ k is a solution of VI (W, F, θ ). This completes the proof.
Lemma . Let {w k } and {ŵ k } be two sequences generated by SGADMM. Then, for any w ∈ W, we have Proof Applying the identity This together with () and () implies that Now let us deal with the last term in (), which can be written as (using ()).
Substituting the above inequality into (), the assertion of this lemma is proved.
Theorem . Let {w k } and {ŵ k } be two sequences generated by SGADMM. Then, for any w ∈ W, we have Proof First, combining () and (), we get From the monotonicity of F(·), we have Adding the above two inequalities, we obtain the assertion (). The proof is completed.
With the above theorem in hand, we are ready to establish the global convergence of SGADMM for solving VI(W, F, θ ).
Theorem . Let {w k } be the sequence generated by SGADMM. If α > , R i + βA i A i  (i = , ), then the corresponding sequence {w k } converges to some w ∞ , which belongs to W * .
Proof Setting w = w * in (), we have where the second inequality follows from w * ∈ W * . Thus, we have By α >  and the positive semi-definite of H, the above inequality implies that Thus, by the definition of H, we have where is positive definite by R  + βA  A  . From () again, we have Thus, Then, taking the limit along the subsequence {w k j } in () and using (), for any w ∈ W, we obtain which indicates that w ∞ is a solution of VI(W, F, θ ). Then, since w * in () is arbitrary, we can set w * = w ∞ and conclude that the whole generated sequence {w k } converges by R i + βA i A i  (i = , ). This completes the proof.

Convergence rate
Now, we are going to prove the worst-case O(/t) convergence rate of SGADMM in both the ergodic and the non-ergodic senses.
Theorem . Let {w k } and {ŵ k } be the sequences generated by SGADMM, and set and its closed-form solution is given by where, for any c > , shrink c (·) is defined as Similarly, for problem (), its first two sub-problems resulting by SGADMM are as follows.
• With the given x k  and λ k , the x  -sub-problem in () is (here R  = ) x k+  = argmin and its closed-form solution is given by • With the updated x k+  , the x  -sub-problem in () is (here and its closed-form solution is given by Obviously, the above two iterative schemes both need to compute A A and A y, which is quite time consuming if n is large. However, noting that these two terms are invariant during the iteration process, therefore we need only compute them once before all iterations. Regarding the penalty parameter β and the constant α in SGADMM, any β >  and α ≥  can ensure the convergence of SGADMM in theory. There are two traditional methods to determine them in practice. One is the tentative method, which is easy to execute. The other is the self-adaptive adjustment method, which needs much computation. In this experiment, for β and α, we use the tentative method to determine their suitable values. For β, Xiao et al. [] set β = mean(abs(y)) for ADMM. Motivated by this choice, we set β = mean(abs(y))/(α -) in our algorithm. As for the parameter α, we have pointed out in Remark . that larger values of α may be beneficial for our algorithm. Here, we use () to do a little experiment to test this. We choose different values of α in the interval [, ]. Specifically, we choose α ∈ {., ., . . . , }. Other data about this experiment are as follows: the proximal parameter τ is set as τ = .(α -)β A A ; the observed signal y is set as y = Ax + . × randn(m, ) in Matlab; the sensing matrix A and the original signal x are generated bȳ and x = zeros(n, ); p = randperm(n); x p( : k) = randn(k, ).
Then the observed signal y is further set as (R ) - y. The initial points are set as x   = A y, λ  = Ax   . In addition, we set the regularization parameter μ = ., and the dimensions of the problem are set as n = ,, m = , k = , where k denotes the number of the non-zeros in the original signal x. To evaluate the quality of the recovered signal, let us define the quantity 'RelErr' as follows: wherex denotes the recovered signal. The stopping criterion is where f k denotes the function value of () at the iterate x k .

Numerical results
The numerical results are graphicly shown in Figure . Clearly, the numerical results in Figure  indicate that Remark . is reasonable. Both CPU time and number of iterations are descent with respect to α. Then, in the following, we set α = ., which is a moderate choice for α. Now, let us graphically show the recovered results of SGADMM for () and (). The proximal parameter τ is set as τ = .(α -)β A A for (), and τ = . A A for (). The initial points are set as x   = A y, λ  = Ax   for (), and x   = A y, λ  = x   for (). Other parameters are set the same as above. Figure  reports the numerical results of SGADMM for () and ().
The bottom two subplots in Figure  indicate that our new method SGADMM can be used to solve () and ().
In the following, we do some numerical comparisons to illustrate the advantage of our new method and to analyze which one is more suitable to compressed sensing () between the two models () and (). SGADMM for () is denoted by SGADMM, SGADMM for () is denoted by SGADMM. We also compare SGADMM with the classical ADMM.  The numerical results are listed in Table , where 'Time' denotes the CPU time (in seconds), and 'Iter' denotes the number of iterations required for the whole recovering process, m = floor(γ n), k = floor(σ m). The numerical results are the average of the numerical results of ten runs with different combinations of γ and σ .

Discussion
The numerical results in Table  indicate that: () by the criterion 'RelErr' , all methods successfully solved all the cases; () by the criteria 'Time' and 'Iter' , SGADMM performs better than the other two methods. Especially the number of iterations of SGADMM is about at most two-thirds of the other two methods. This experiment also indicate that the model () is also an effective model for compressed sensing, and sometimes it is more efficient than the model (), though they are equivalent in theory. In conclusion, by choosing some relaxation factor α ∈ [, +∞), SGADMM may be more efficient than the classical ADMM.

Conclusions
In this paper, we have proposed a symmetric version of the generalized ADMM (SGADMM), which generalizes the feasible set of the relaxation factor α from (, ) to [, +∞). Under the same conditions, we have proved the convergence results of the new method. Some numerical results illustrate that it may perform better than the classical ADMM. In the future, we shall study SGADMM with α ∈ (, ) to perfect the theoretical system.