The two-stage iteration algorithms based on the shortest distance for low-rank matrix completion☆
Introduction
From the pioneering work on low-rank approximation by Fazel [10] as well as on matrix completion by Candès and Recht [7], there has been a lot of study (see [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26] and references therein) both from theoretical and algorithmic aspects on the problem of recovering a low-rank matrix from partial entries-also known as matrix completion. The problem occurs in many areas of engineering and applied science such as model reduction [18], machine learning [1], [2], control [20], pattern recognition [9], imaging inpainting [3] and computer vision [23] and so on. There is a rapidly growing interest for this issue. Explicitly seeking the lowest rank matrix consistent with the known entries is mathematically expressed as: where the matrix is the underlying matrix to be reconstructed, Ω is a random subset of indices for the known entries, and is the associated sampling orthogonal projection operator which acquires only the entries indexed by .
Since a manifold of rank r matrices can be factorized into a bi-linear form where and a few algorithms have been presented to solve (1) while the rank r was known or can be estimated [18], [19]. The general problem (1), however, is non-convex and is NP-hard [12] due to the rank objective. Vandereycken [24] applied the Riemannian optimization to the problem by minimizing the least square distance on the sampling set over the Riemannian manifold of matrix Z0. Then a Riemannian geometry method and a Riemannian trust-region method were given by Mishra et al. [21] and Boumal et al. [5], respectively. Evidently, the computation of a gradient is expensive in method, and then several methods resulted in alternating optimization and XY were raised [8], [15], [22].
On the other hand, Candés and Rechat [7] replaced the rank objective in (1) with its convex relaxation, the nuclear norm ‖Z‖* which is the sum of all singular values of matrix Z, that is Alternative to the convex optimization, there have been many algorithms which are designed to attempt to solve for the global minimum of (1) directly; many of them are adaptations of algorithms for compressed sensing, such as the hard thresholding algorithms [4], [14], [16], the singular value thresholding (SVT) method as well as its variants [6], [13], [25]. However, the computations of a partial singular value decomposition (SVD) were required at each iteration in the most direct implementation of these algorithms. The computational cost of computing the SVD has complexity of O(n3) when the rank r and matrix-size n are proportional, resulting in computing the SVD to be the dominant computational cost at each iteration and then limits their applicability for large n. In addition, Lin et al. [17] proposed an augmented Lagrange multiplier (ALM) method which performs better both in theory and algorithms than the others that with a Q-linear convergence speed globally.
Recently, based on the simple factorization where and mentioned above, rather than solving (1), algorithms are designed to compute the non-convex problem with In fact, the model (3) replaced the rank objective in (1) with the distance between a matrix and an r-dimensional manifold for the fixed-rank problem. Algorithms for the solution of (3) with the distance objective usually follow an alternating minimization scheme, with PowerFactorization [11] and LMaFit [26] two representatives. Tanner and Wei [22] proposed an alternating steepest descent (ASD), and a scaled variant (ScaledASD) methods in 2016. In so doing ASD and ScaledASD [22] are able to recover matrices of substantially higher rank than can LMaFit [26]. However, the rank of most of completing is unknown such that we have to estimate it in advance or approximate it from a lower rank until satisfying . In this study, we first define the distance between a matrix and the r-dimensional manifold and then come up with a class of two-stage iteration algorithms for the case that the rank r was unknown. The rank is increased either one-by-one until the optimal rank r is obtained for a lower rank (say, be estimated) model or by combining l jumping-space with one-by-one until the optimal rank r is obtained for a larger rank (say, be estimated) model. The inner iteration finds the matrix which is up to the shortest or approximation shortest distance and the outer iteration finds the optimal r-dimensional manifold with two kinds criterion. The convergence theory of the new algorithms are studied.
The rest of the paper is organized as follows. A class of two-stage iteration algorithms for the case that the rank r was unknown is proposed in Section 2. The convergence of the new algorithms are discussed in Section 3. The numerical experiments are shown and comparison to algorithms in Section 4. Finally, we end the paper with a concluding remark in Section 5.
Here are some necessary notations and preliminaries. is used to denote the m × n real matrix set, and the n-dimensional real vector set. XT denotes the transpose of the matrix or vector X. The Frobenius norm is denoted by ‖X‖F. For a matrix dim(X) is always used to represent dimensions of the manifold of fixed-rank matrix X and rank(X) represents the rank of a matrix X. Let denote the indices of the observed entries of the matrix X, denote the indices of the missing entries. Then be the orthogonal projection operator on the span of matrices vanishing outside of Ω. So that the (i, j)th component of is equal to Xij when (i, j) ∈ Ω, and zero otherwise. Also, stands for the r-dimensional manifold of fixed-rank matrices.
The one-to-one correspondence between a matrix and its projection enables us to devise a notation of distance between a matrix and an r-dimensional manifold as follows.
Definition 1.1 For a matrix is called distance between a matrix Y and an r-dimensional manifold . That is essentially the distance between matrix Y and its projection onto the r-dimensional manifold . We trivially give the distance between a feasible matrix and its projection onto the r-dimensional manifold for introducing the new iteration algorithms. For we called as a distance between a feasible matrix Y and an r-dimensional manifold . Evidently, it should be noted that min d(Yr) > 0 if r < min rank(Z) and if . To obtain the min d(Yr), some algorithms are presented by combining the model (1) and (4).
Section snippets
Scaled alternating steepest descent (ScaledASD) method
In order to solve the non-convex problem (3), the alternating steepest descent (ASD) method [22] applies steepest gradient descent to f(X, Y) in (3) alternatively with respect to X and Y. If f(X, Y) is written as fY(X) when Y is held constant and fX(Y) when X is held constant, the directions of gradient The steepest descent stepsizes along the gradient descent directions and were denoted by
Convergence analysis
In this section, the convergence theory of Algorithms 3 and 4 will be discussed in detail and that of Algorithm 2 can be a simple modification in [22].
The following Lemmas will be need in our supplement study.
Lemma 3.1 (Lemma 4.2 of [22]) Assume that the matrix sequence (Xi, Yi) generated by Algorithm 1 is nonsingular (i.e., full rank) during all the iterations. Let be a subsquence of (Xi, Yi) converging to a stationary and nonsingular pair (X*, Y*). Then is bounded and satisfies
Numerical experiments
In this section, we conduct experiments of randomly drawn rank r matrices using the model where and from the normal distribution . A random subset Ω are sampled uniformly at random, and p is defined as its fraction of the entries in Z0. For conciseness, the tests presented consider square matrices as is typical in the study.
The comparison results of the four algorithms (Two-stage I, Two-stage II, Two-stage III, and ALM) are displayed in Tables 1–6. There, we denote
Concluding remarks
For general matrix completion problems, we have proposed a class of two-stage iteration methods, Two-stage I-III. The inner iteration is the scaled alternating steepest descent algorithm for the fixed-rank matrix completion problem, the outer iterations can be used two iteration criterions: the gradient norm and the distance between the feasible part with the corresponding part of completed low-rank matrix. Numerical experiments show that Two-stage II-III methods outperform the famous ALM
References (26)
- et al.
Shape and motion from image streams under orthography : a factorization method
Int. J. Comput. Vis.
(1992) - et al.
Solving a low-rank factorization model for matrix completion by a non-linear successive over-relaxation algorithm
Math. Program. Comput.
(2012) - et al.
Unconvering shared structures in malticalass classification
Proceeding of the Twenty-Fourth International Conference on Machine Learning
(2007) - et al.
Multi-task feature learning
Adv. Neural Inf. Process. Syst.
(2007) - et al.
Multi-task feature learning, image inpainting
Comput. Gr.
(2000) - et al.
CGIHT: Conjugate Gradient Iterative Hard Thresholding for Compressed Sensing and Matrix Completion
(2014) - et al.
RTRMC: a Riemannian trust-region method for low-rank matrix completion
- et al.
A singular value thresholding method for matrix completion
SIAM J. Optim.
(2010) - et al.
Exact matrix completion via convex optimization
Found. Comput. Math.
(2009) - et al.
Matrix completion via an alternating direction method
IMA J. Numer. Anal.
(2012)
Matrix Methods in Data Mining and Pattern Recognization
Matrix rank minimization with applications
Rank-constrained solutions to linear matrix equations using powerfactorization
IEEE Signal Process. Lett.
Cited by (6)
Toeplitz matrix completion via smoothing augmented Lagrange multiplier algorithm
2019, Applied Mathematics and ComputationCitation Excerpt :For example, the singular value theresholding (SVT) method as well as its variants [7,18,37], an accelerated proximal gradient (APG) method [32], the augmented Lagrange multiplier (ALM) method [23] etc. Numerous details derivations on MC problem can be refer to the [5–13,16,18–23,25,28,31,32,34–39] and the references given therein. In this study, we focus on the Toeplitz matrix completion (TMC) problem, one of the most important MC problems.
Accelerated low rank matrix approximate algorithms for matrix completion
2019, Computers and Mathematics with ApplicationsA Singular Value Thresholding with Diagonal-Update Algorithm for Low-Rank Matrix Completion
2020, Mathematical Problems in EngineeringToeplitz matrix completion via a low-rank approximation algorithm
2020, Journal of Inequalities and ApplicationsA new method based on the manifold-alternative approximating for low-rank matrix completion
2018, Journal of Inequalities and Applications
- ☆
This work is supported by NSF of China (11371275) and NSF of Shanxi Province (201601D011004).