On Convergence of Inexact Augmented Lagrangians for Separable and Equality Convex QCQP Problems without Constraint Qualification

The classical convergence theory of the augmented Lagrangian method has been developed under the assumption that the solutions satisfy a constraint qualification. The point of this note is to show that the constraint qualification can be limited to the constraints that are not enforced by the Lagrange multipliers. In particular, it follows that if the feasible set is non-empty and the inequality constraints are convex and separable, then the convergence of the algorithm is guaranteed without any additional assumptions. If the feasible set is empty and the projected gradients of the Lagrangians are forced to go to zero, then the iterates are shown to converge to the nearest well posed problem.


Introduction
We are interested in the minimization of a convex quadratic function subject to possibly nonlinear separable convex inequality constraints and linear equality constraints where b ∈ R n , h i are differentiable and convex functions, A is an n × n Symmetric Positive Definite (SPD) matrix, and B ∈ R m×n , B = O.If not specified otherwise, we assume Ω SE = ∅ and admit dependent rows of B, but we do not require that B is a full column rank matrix, so that KerB = {o}.Observe that some more general Quadratic Programming (QP) or QCQP problems can be reduced to Eq. ( 1) by duality, a suitable shift of variables, or by a modification of f .The problem arises in the solution of many engineering problems, e.g., in the design of electromagnetic brakes [16] or contact mechanics [13].
Here we restrict our attention to the Augmented Lagrangian method (also called the method of multipliers) proposed by Hestenes (see [19]) and Powell (see [21]) for the solution of nonlinear optimization problems with equality constraints.The algorithms can be considered as a modification of the exterior penalty method which enables to reduce the original equality constrained problem to the series of unconstrained problems without increasing the penalty parameter to infinity.The algorithm is described, e.g., in the classical monographs [3] and [18].It has been soon observed that the algorithm is useful also for the solution of more general problems, such as the problems with bound and equality constraints.An important instance of such approach was used by Conn, Gold, and Toint (see [6]) in their software LANCELOT (see [7]).An up to date description of the general algorithm can be found in [4].We are interested in a modification of the algorithm called SMALSE-M (Semimonotonic Augmented Lagrangians for Separable and Equality Constraints).Unlike the LANCELOT, it uses the adaptive precision control introduced by [11] and keeps the regularization parameter constant (see [10], or [9]).The algorithm was proved to generate the iterates with the feasibility error and projected gradients converging to zero (see [12] or [15]).Though some kind of optimality has been proved for SMALSE-M, no proof of convergence of the iterates themselves was given.
Our main goal here is to show that the iterates generated by SMALSE-M converge to the solution without assuming any constraint qualification and to prove the convergence of the Lagrange multipliers under the assumption of the regularity of the solution.Let us recall that the optimality properties of SMALSE-M were exploited in the development of scalable algorithms for the solution of contact problems with friction [13] and [14].If the feasible set is empty and the projected gradients of the Lagrangians are forced to go to zero, then the iterates of a slightly modified algorithm are shown to converge to the solution of a nearest well posed problem.Notice that a similar results were obtained by Gilbert and Chiche [5] or Gilbert and Joannopoulis [17].The paper is an extension of the results that appeared in [14].

KKT Conditions
Since Ω SE is closed, convex and nonempty and f is assumed to be strictly convex, the solution of problem Eq. ( 1) exists and is necessarily unique ( [9], Proposition 2.5).If the intersection of Ω SE with the interior of Ω S is nonempty, then the solution satisfies the Slater constraint qualification and can be characterized by means of the augmented Lagrangian whose gradient reads If the solution satisfies the Slater constraint qualification, then a feasible vector x ∈ Ω SE is a solution of Eq. (1) if and only if there is λ ∈ R m and µ ∈ R s such that ∇ x L(x, λ, µ, ) = o, µ i ≥ 0, and µ i h i (x i ) = 0, i = 1, . . ., s.
Having effective algorithms for the solution of QCQP problems with separable constraints, it is convenient to use explicitly the Lagrange multipliers only for the equality constraints, i.e., to use and the gradient g = ∇ x L of the reduced augmented Lagrangian g = g(x, λ, ) defined by To simplify the notation, we use when we can specify the remaining arguments from the context.
To rewrite the KKT conditions Eq. ( 5) in a form suitable for our exposition, we introduce some auxiliary notations.Let S denote the set of all indices of the constraints so that For any x ∈ R n , we define the active set of x by Its complement is called a free set.For x ∈ Ω S , we define the outer unit normal n by The components of the gradient that violate the KKT conditions Eq. ( 5) in the free and active set are called the free gradient ϕ and the chopped gradient β, respectively.They are defined by where we use the notation . If we define the projected gradient by it is easy to check that x ∈ Ω SE is a solution of Eq. ( 1) if and only if there is λ ∈ R m such that We shall need yet another simple property of the projected gradient.
Proof: First observe that Using the definition of the projected gradient, we get To finish the proof, it is sufficient to observe that for i ∈ A(x) due to the convexity of h i .

SMALSE-M
The complete SMALSE-M algorithm reads as follows.
In Step 1 we can use any algorithm for minimizing a strictly convex quadratic function subject to separable constraints as long as it guarantees the convergence of the projected gradient to zero.The next lemma recalls that Alg. 1 is well defined.Typical choice of parameters is η k = 0.1 b , β = 0.1, M 0 = 10, and = A .
and g P (y k , λ, ) converges to the zero vector.Then {y k } either converges to the unique solution x of problem Eq. ( 1), or there is an index k such that Proof: If Eq. ( 27) does not hold for any k, then for any k.Since g P (y k , λ, ) converges to the zero vector by the assumption, it follows that By k converges to zero.Thus B y = o and Algorithm 1 Semimonotonic augmented Lagrangians for separable and equality constrained QCQP problems (SMALSE-M).
Given an SPD matrix A ∈ R n×n , B ∈ R m×n , n-vector b, constraints h.
The tools used for the analysis of SMALSE-M were sufficient to prove only the convergence of the feasibility error and the projected gradient [12].The results relevant for our exposition read as follows.

Boundedness
The first step toward the proof of convergence of our SMALSE-M Alg. 1 is to show that the iterates x k are bounded.
Proposition 2. Let {x k } and {λ k } be generated by Alg. 1 for the solution of Eq. ( 1) with η > 0, 0 < β < 1, M 0 > 0, > 0, and λ 0 ∈ R m .For each i ∈ {1, . . ., s}, let I i denote the indices of the components of x i , where x i is the argument of the constraint function h i , and let us define Assume that the boundary of Ω S is bounded, i.e., there is C > 0 such that for any Then {x k } is bounded.Moreover, if the solution of Eq. ( 1) is regular, then also {λ k } is bounded.
Proof: Since there is only a finite number of different subsets F of the set of all indices S = {1, . . ., s}, and {x k } is bounded if and only if {x k F } is bounded, we can restrict our attention to the analysis of infinite subsequences {x k F : F(x k ) = F} that are defined by the nonempty subsets F of S.
Let F ⊆ S, F = ∅, let K = {k : F(x k ) = F} be infinite, and denote We get and Since for k ∈ K and both g P (x k , λ k , ) and Bx k converge to zero by the definition of x k in Step 1 of Alg. 1 and Eq. ( 31), the right-hand side of Eq. ( 36) is bounded.Since H F F is nonsingular, it is easy to check that the matrix of the system Eq.( 36) is nonsingular when B * F is a full row rank matrix.It simply follows that both {x k } and {λ k } are bounded provided the matrix of the system Eq.( 36) is nonsingular.
If B * F is not a full row rank matrix, then its rank r satisfies r < m, and by the singular value decomposition formula there are orthogonal matrices and the diagonal matrix with the nonzero diagonal entries σ 1 > 0, . . ., σ r > 0 such that B * F = UΣV T .Taking and we have B * F = Û D VT and we can define a full row rank matrix that satisfies for any vector x We shall assign to any λ ∈ R m the vector λ = ÛT λ, so that BT F λ = B T F λ. Using the latter identity and Eq.(36), we get the system with a nonsingular matrix.The right-hand side of Eq. ( 43) being bounded due to B * F x k F = B * F x k F , we conclude that the set {x k F : F(x k ) = F} is bounded.See also [15] or [12].
Remark 1.Notice that the assumption on the boundary of Ω S does not imply the compactness of Ω SE .

Convergence
Now we are ready to prove the main convergence results.To describe them, let F = F( x) denote the set of the indices of variables that are involved in the free constraints of the unique solution x and recall that x is a regular solution of Eq. ( 1) if B * F is a full row rank matrix ( * denotes the full set of indices).
(ii) If the solution x of Eq. ( 1) is regular, then {λ k } converges to the uniquely determined vector λ of Lagrange multipliers of Eq. ( 1).
(iii) Let us define for any If the feasible set Ω SE is empty and η k → 0, then {x k } converges to the solution x of the nearest feasible problem with shifted equality constraints, where Proof: (i) Since the iterates x k are bounded due to Prop. 2, it follows that there is a cluster point x of {x k } and there is K ⊆ N such that Moreover, since x k ∈ Ω S and by Eq. ( 31) it follows that Bx = o and x ∈ Ω SE .
To show that x solves Eq. ( 1), let d ∈ R n satisfy x+d ∈ Ω SE , so that Bd = o, and set We used the relation Taking the limits, we get Thus x solves Eq. ( 1).The solution x of Eq. ( 1) being unique, it follows that x k converges to x = x.
(ii) Let us recall that F = F( x) and denote H = A + B T B. Since we have just proved that {x k } converges to x, there is k 1 such that F ⊆ F{x k } for k ≥ k 1 and converges to zero.It follows that the sequence is bounded.Moreover, if λ is any vector of Lagrange multipliers, then and converges to zero.
If the solution x of Eq. ( 1) is regular, then B T * F is a full column rank matrix, i.e., R m = ImB * F , and there is the unique Lagrange multiplier λ for problem Eq. ( 1).Moreover, since it follows that where σ F min denotes the smallest nonzero singular value of B * F .The convergence of the right-hand side of Eq. (56) to zero thus implies that λ k converges to λ.
(iii) As in the proof of (i), there is a cluster point x of x k .Using the other arguments of the proof of (i), we get that x solves the problem to find min f (x) subject to x ∈ Ω S and Bx = Bx.Moreover, if the feasible set Ω SE is empty, then the multipliers λ k are necessarily unbounded.After some simple manipulations, we get that the vector n with the nonzero components satisfies n ∈ ImB T .Due to the convexity, it follows that x is the nearest point of Ω S to Ω E .

Numerical Results
Now where (59)   The solution of the problem is x = (0, 0).Lagrange multipliers for the equality constraint during the iteration process of Alg. 1 are depicted in Fig. 2. The values of the vectors x during the iteration process of the same algorithm are depicted in Fig. 3.The green circle marks the initial iteration and the red circle indicates the last iteration.
Example 2. In the second example the feasible set Ω SE is empty.There is no point of the feasible set which satisfies any constraint qualification (Slater constraint qualification, regularity condition, etc.).The optimization problem reads as follows: where If η k → 0, then Alg. 1 converges to the vector x = (9, 0) which is the closest point of Ω S to Ω E .
Figure 4 shows the geometric interpretation of the equality and inequality constraints.The values of the Lagrange multipliers for the equality constraint during the iteration process of Alg. 1 are depicted in Fig. 5.The values of the vectors x k during the iteration process of the same algorithm are depicted in Fig. 6.The green circle marks the initial iteration and the red circle indicates the last iteration.
lim k→∞ g P (x k , λ k , ) = o and lim k→∞ Bx k = o.(31) c 2017 ADVANCES IN ELECTRICAL AND ELECTRONIC ENGINEERING

Fig. 2 :
Fig. 2: Example 1, values of the Lagrange multipliers during the iteration process.

Figure 1 Fig. 3 :
Figure1shows the geometric interpretation of the equality and inequality constraints.The values of the

Fig. 5 :
Fig. 5: Example 2, values of the Lagrange multipliers during the iteration process.

Fig. 6 :
Fig. 6: Example 2, values of the vectors x during the iteration process.
we are ready to show some numerical examples which illustrate the preceding results.The feasible set in the first example contains only one point and the solution of the minimization problem in Exm. 1 is not regular.The feasible set in Exm. 2 is empty and Alg. 1 converges to the closest point of the set Ω S to the set Ω E (see Exm. 2 for formal definition).The sequence of the Lagrange multipliers for the equality constraint is unbounded in both cases.Example 1.In the first example the feasible set Ω SE is restricted to the point.There is no point of the feasible set which satisfies any constraint qualification (Slater constraint qualification, regularity condition, etc.).The optimization problem reads as follows: