A Parallel Splitting Augmented Lagrangian Method for Two-Block Separable Convex Programming with Application in Image Processing

The augmented Lagrangian method (ALM) is one of the most successful ﬁrst-order methods for convex programming with linear equality constraints. To solve the two-block separable convex minimization problem, we always use the parallel splitting ALM method. In this paper, we will show that no matter how small the step size and the penalty parameter are, the convergence of the parallel splitting ALM is not guaranteed. We propose a new convergent parallel splitting ALM (PSALM), which is the regularizing ALM’s minimization subproblem by some simple proximal terms. In application this new PSALM is used to solve video background extraction problems and our numerical results indicate that this new PSALM is eﬃcient.


Introduction
Many problems arising from machine learning, such as compressive sensing [1,2], the video background extraction problem [3][4][5], batch images alignment [6,7], and transform invariant low-rank textures [8,9] can be formulated as separable convex programming with linear constraints. In this paper, we consider the following two-block separable convex programming: where i � 1, 2, A i ∈ R ℓ×n i , the ℓ × n i matrix on R; b ∈ R ℓ , the ℓ-dimensional vector on R; X i is the subset of R n i . Here, θ i : X i ⟼ R(i � 1, 2) are the proper lower semicontinuous convex functions, i.e., lim inf x⟶x θ i (x) � θ i (x), for all x ∈ dom θ i ≠ ∅, for all x, y ∈ X i , λ ∈ [0, 1].
(2) e solution set of problem (1), denoted by X * , is assumed to be nonempty. Note that theoretical results to problem (1) can easily be extended to matrix variables.
Let β > 0 be a penalty parameter, and the augmented Lagrangian functions of problem (1) can be written as where λ ∈ R ℓ is the Lagrangian multiplier. It is well-known that the augmented Lagrangian method (ALM) [10] is one of the most successful first-order methods for convex programming with linear constraints. Applying it to problem (1), we can obtain the following procedure: where α ∈ (0, 2) is the step size. It has been proved that the sequence generated by the iterative scheme (4) is Fejér monotone and has the global convergence. In many practical problems, the proximal mapping prox ]θ i of θ i , defined as prox ]θ i x i ≔ arg min has closed-form expression. For example, consider the video background extraction problem given by min L,S ‖L‖ * + τ‖S‖ 1 , where σ i (L) are the singular values of the m × n matrix L, U V ⊤ is the singular value decomposition of L, and s ij are elements of the m × n matrix S. en, by [11], the proximal mapping (5) is rewritten as Similarly, by [12], the proximal mapping (5) is given as where S ] (E) is defined by and sign(·) is the sign function. However, the proximal mapping of ‖L‖ * + τ‖S‖ 1 does not admit closed-form expression so that the iterative scheme (4) for problem (5) has to resort some inner solver to compute its minimization problem in each iteration, which inevitably creates a chain of problems, such as an efficient solver to solve the minimization problem and stopping criterion to ensure the global convergence of inexact version of the iterative scheme (5). erefore, we should not ignore the separable structure of the objective function of problem (1). at is, we'd better split the minimization subproblem in (5) into some smaller scale subproblems to fully utilize the individual property of θ i . e alternating direction method of multipliers (ADMM) can decompose a complicated problem into some small-scale subproblems and solve these subproblems in a sequential manner or a parallel manner. Focusing on the way of splitting the augmented Lagrangian function in the spirit of the well-known alternating direction method (ADM), Tao and Yuan [4] have proposed a variant of the alternating splitting augmented Lagrangian method (ASALM) with convergent property, which can solve three-block separable convex programming. He et al. [13] propose a splitting method for solving a separable convex minimization problem with linear constraints, where the objective function is expressed as the sum of m individual functions without coupled variables. Aybat and Iyenga [14] have designed a variant of ADMM with increasing penalty for stable principal component pursuit (ADMIP) to solve the SPCP problem. Its preliminary computational tests show that ADMIP works very well in practice and outperforms ASALM which is a state-of-the-art ADMM algorithm for the SPCP problem with a constant penalty parameter.
In this paper, following the abovementioned procedure, we split the minimization problem of ALM in the Jacobian manner and get the following iterative scheme: e iterative scheme (11) takes advantages of the separable structure and parallel architectures. Unfortunately, it suffers from the weakness that its convergence cannot be guaranteed under the convex assumption of θ i and X * ≠ ∅.
To verify this, we shall apply the iterative scheme (11) to solve problem (1) given in Section 2 and show that the generated sequence diverges even β ⟶ 0 + . So, in Section 2, we propose a new parallel splitting ALM for problem (1), which is not only convergent but also keeps the good properties of the original ALM. Our numerical results in Section 3 indicate that the new PSALM is efficient. e remainder of this paper is organized as follows. In Section 2, an example is given showing that the iterative scheme (11) maybe diverge, and a new parallel splitting ALM for problem (1) is proposed, which has the global convergence in both ergodic and nonergodic senses. For illustration of our main results, two numerical examples are given in Section 3 and a brief conclusion is drawn in Section 4.

Algorithm and Convergence Results
In this section, we will show that the iterative scheme (11) with any small positive α and β can be diverge so that a new parallel splitting ALM for problem (1) is proposed and the corresponding convergence results are established.
In what follows, R n will stand for the n-dimensional Euclidean space, and set By the characterization of optimality condition for convex optimization, problem (1) can be readily characterized as the following mixed variational inequalities denoted by MVI (W, F, θ): Because F(w) is a linear mapping with the skew-symmetric coefficient matrix, it satisfies the following property: e solution set of MVI (W, F, θ) is denoted by W * , which is nonempty under the assumption X * ≠ ∅.
Based on the prototype algorithm proposed by He and Yuan in [15], we can introduce the following prototype algorithm to solve MVI (W, F, θ): A prototype algorithm for MVI (W, F, θ) is denoted by ProAlo [Prediction] For given w k , find w k ∈ W and Q satisfying where the matrix Q has the property (Q + Q ⊤ ) which is positive definite [Correction] Determine a nonsingular matrix M, a scalar α > 0, and generate the new iterate w k+1 via Similar to [15], we have the following convergence condition, which ensures that the ProAlo is globally convergent and has worst-case O(1/t) convergence rate in ergodic and nonergodic senses.
Convergence Condition Now we give an example to show that the iterative scheme (11) maybe diverge for problem (1).

Example 1. Consider the linear equation [16]
Obviously, the linear equation (18) is a special case of problem (1) with the specifications:

augmented Lagrangian function of problem (18) is given by
en, applying the iterative scheme (11) to solve problem (18) gives the following procedure: where ree eigenvalues of the matrix P are is the spectral radius of P. Hence, the iterative scheme (20) with α, β > 0 is divergent, which also indicates that the iterative scheme (11) diverge for problem (1) under the traditional assumption. In the following, we shall design a new parallel splitting ALM for problem (1), which maximally inherits the structure of the iterative scheme (11).

Algorithm 1. new PSALM
Step 1. Given an initial point Mathematical Problems in Engineering Step 3. If max then stop; otherwise, replace k + 1 by k, and go to Step 2.

Remark 1.
e new PSALM iterative scheme (23) not only preserves the main advantage of the ALM iterative scheme (11) in treating the objective functions θ 1 and θ 2 individually, but also numerically accelerates (11) with some values of the step size α, e.g., α ∈ (1, 2), which can be achieved when ] > 1.
In the following, we further make the following assumption about problem (1).

Assumption 1.
e matrices A 1 and A 2 in problem (1) are both full column ranks.
To establish the convergence results of new PSALM, we first cast it to a special case of the ProAlo. To accomplish this, let us define an auxiliary sequence w k � (x k 1 , x k 2 , λ k ) as follows: is together with the updated formula of λ in (23) gives us, where the matrix M is defined as Lemma 1. Let w k be the sequence generated by the new PSALM and w k be defined as in (24), and it holds that where the matrix Q is defined by Proof. By the first-order optimality condition of the two minimization problems in (23), for any x 1 ∈ X 1 andx 2 ∈ X 2 , we have (30) e definition of the variable λ k in (24) gives (31) Adding the above three inequalities, using the notation of θ(x) and F(w), we can get the compact form (28) and this completes the proof.
Substituting it into the right-hand side of (28), we have So, w k ∈ W * , and the stopping criterion of the PSALM is reasonable.
With inequalities (26) and (28) in hand, to establish the convergence results of the PSALM, we only need to ensure the matrices M and Q defined as in (27) and (29) satisfy the convergence condition. (27)

Lemma 2. Let the matrices Q and M be defined as in
which is positive definite from ] > 0 and Assumption 1. (ii) From the definition of the matrices Q and M, we have which is symmetric and positive definite by Assumption 1. (iii) Similarly, from the definition of Q, M, we have where From Assumption 1, we only need to prove the matrix R(α) is positive definite. In fact, it can be written as where ⊗ denotes the matrix Kronecker product. us, we only need to prove the 3 order matrix is positive definite. Its three eigenvalues are )/2. en, solving the following three inequalities we get 0 < α < 2]/1 + ]. erefore, the matrix G is positive definite for any α ∈ (0, 2]/1 + ]). In addition, from its expression, the matrix G(α) is obviously symmetric. is completes the proof.
Lemma 2 indicates that the matrices Q and M defined as in (27) and (29) satisfy convergence condition, and according to the result in [15], we get the following convergence results of new PSALM. □ Theorem 1 (global convergence). Let w k be the sequence generated by new PSALM. en, it converges to some w ∞ , which belongs to W * . Theorem 2 (ergodic convergence rate). Let w k be the sequence generated by new PSALM, w k be the corresponding sequence defined as in (24). Set Then, for any integer t ≥ 0, we have Theorem 3 (nonergodic convergence rate). Let w k be the sequence generated by new PSALM. en, for any w * ∈ W * and integer t ≥ 0, we have where c 0 > 0 is a constant. Note that the right-hand side of (41) is independent of the distance between the initial iterate w 0 and the solution set W * . erefore, (41) is not a reasonable criterion to measure the nonergodic convergence rate. In the following, we give a refined result to measure the nonergodic convergence rate of the sequence w k generated by new PSALM.
Lemma 4. Let w k be the sequence generated by new PSALM. en, for any w ∈ W, we have Mathematical Problems in Engineering e left subplot of Figure 1 graphically depicts the numerical results when ] � 5 and α � 0.1: 0.1: 1.5. It reveals that when ] is fixed, the larger α is, the better of the new PSALM's performance is. e right subplot of Figure 1 graphically depicts the numerical results when ] � 0.1: 0.1: 5 and α � (1.9]/1 + ]). It reveals that when ν takes some values near 1, the new PSALM always performs best. is is in conformity with the intuition because the large value of this parameter makes the proximal term ]β/2‖A 1 x 1 − A 1 x k 1 ‖ 2 taking a too heavy weight in the objective function L β (x 1 , x k 2 , λ k ) + ]β/2‖A 1 x 1 − A 1 x k 1 ‖ 2 , leads to a very small increment of the variable x 1 , and finally decreases the convergence speed of the new PSALM. On the contrary, the small value of this parameter also makes the step size α � 1.9]/1 + ] to be very small. erefore, how to choose a suitable combination of α and ] is a challenging issue.

Remark 3.
e step size α is at the denominator of the righthand side terms of (45). erefore, the bigger α is, the smaller the right-hand side terms of (45) are, and the new PSALM may need less iteration to generate an ε-approximate solution of problem (1).

Numerical Results
In this section, we demonstrate the potential efficiency of this new PSALM by solving the video background extraction problem (6).

Simulation Study.
Firstly, we generate a low-rank matrix L * ∈ R m×m and a sparse matrix S * ∈ R m×m . Secondly, we get an observation matrix D � L * + S * be an observation matrix.
irdly, we want to recover the low-rank matrix L * and the sparse matrix S * by problem (6) with D as the input matrix and L and S as the unknown matrices. In the following, let r and spr represent the low-rank ratio of L * and the ratios of the number of nonzeros entries of S * (i.e., ‖S * ‖ 0 /(pq)), respectively. We use the following Matlab script to generate matrix D: (i) L � randn(m, r) * randn(r, n) (ii) S � zeros(m, n); q � randperm(m * n); K � round(spr * m * n); S(q(1: K)) � randn(K, 1) (iii) D � L + S We set m � n � 100 and tested (60) We empirically set τ � 0.1 in problem (6), and α � 1.3, β � 0.05/mean(abs(D(:))), ] � 2 in the PSALM. e initial iterate (L 0 , S 0 , Λ 0 ) � (0, 0, 0), and use the stopping criterion or the number of iterations exceeds 1000. Let L and S be the numerical solution of problem (6) obtained by the PSALM, and we measure the quality of recovery by the relative error to (L * , S * ), which is defined by In Table 1, we report the recovery results on simulated matrices with size m � n � 100. Note that for the recovered sparse matrix S, to eliminate the roundabout error, we have reset the component whose the absolute value is less than 0.01 to be 0, i.e., S(abs(S) < 0.01) � 0. From this table, it can be seen that (i) the rank of the low-rank matrix L * can be   exactly recovered, while the sparse matrix S * is recovered approximately; (ii) though S * is not recovered completely, the relative error RelErr indicates that the precision of the recovered S is high.
In Figure 2, we present the convergence result for problem (6) with r � 5 and spr � 0.1. From this figure, it can be observed that when the number of iterations k � 80, the relative error RelErr achieves at a stable state. It decreases almost linearly with k before k � 80, while it almost does not decrease any more after k � 80.

An Application Example.
In this section, we apply the new PSALM(NEW − PSALM) to solve a concrete problem of model (6): the video background extraction problem [17], which aims to subtract the background from (the low-rank matrix L) a video clip (stacking as the matrix D) and meanwhile detect the moving objects (the sparse matrix S). We set τ � 1/( �������� � max m, n { } ) in problem (6). We have downloaded the video clip: Highway from the Internet. For comparison, we also applied the iteration method in [4] (denoted by TY − VASALM) and the iteration method in [13] (denoted by HTY − IMA) to solve this problem. Figure 3 presents the extracted backgrounds and foregrounds images by TY − VASALM, HTY − IMA, and NEW − PSALM. Clearly, all methods can extract satisfactory results Table 2 lists the detail experimental results, where the number of iterations (Iteration), CPU time in seconds (CPU), the rank of the recovered final low-rank matrix (Rank), and the number of nonzeros of the recovered final sparse matrix (Sparse) are reported. From the results given in Table 2, we see that NEW − PSALM is faster than the other two methods in terms of both iteration numbers and CPU time. In particular, NEW − PSALM outperforms all the other two methods in recovering low rank and sparse components. ese results clearly illustrate the efficiency and robustness of NEW − PSALM.

Conclusion
In this paper, we have proposed a new parallel splitting augmented Lagrangian method(NEW − PSALM) for twoblock separable convex programming and have established its various convergence results, including global convergence, ergodic, and nonerogdic convergence rate. Numerical results indicate that it is efficient for the video background extraction problem.
Practical experiments indicate that the ALM and its various variants with the step size α ∈ (1, 2) can often accelerate the convergence speed, while in the new PSALM, to ensure the step size α ∈ (1, 2), the parameter ν must larger than 1, such as α ∈ (0, 1.9) only if ] ≥ 19. However, as we have mentioned in Example 2, large ν often decreases the convergence speed of the new PSALM. erefore, the ALM type methods with less restrictions imposed on ] and α deserve further researching, and we leave this problem as a future research work.

Data Availability
e data used to support the findings of this study are available from http://www.SceneBackgroundModeling.net.

Conflicts of Interest
e authors declare that there are no conflicts of interest regarding the publication of this paper.  Mathematical Problems in Engineering 9