Improved SVD-based initialization for nonnegative matrix factorization using low-rank correction
Introduction
Nonnegative matrix factorization (NMF) is the problem of approximating a input nonnegative matrix X as the product of two nonnegative matrices: Given and an integer r, find and such that X ≈ WH. NMF allows to reconstruct data using a purely additive model: each column of X is a nonnegative linear combination of the columns of W. For this reason, it is widely employed in research fields like image processing and computer vision [8], [21], data mining and document clustering [6], hyperspectral image analysis [19], [25], signal processing [31] and computational biology [20]; see also [5], [9] and the references therein.
To measure the quality of the NMF approximation, a distance metric should be chosen. In this paper, we focus on the most widely used one, namely the Frobenius norm, leading to the following optimization problemwhere is Frobenius norm of a matrix M. Most algorithms tackling (1) use standard non-linear optimization schemes such as block coordinate descent methods hence initialization of the factors (W, H) is crucial in practice as it will influence
- (i)
the number of iterations needed for an algorithm to converge (in fact, if the initial point is closer to a local minimum, it will require less iterations to converge to it), and
- (ii)
the final solution to which the algorithm will converge.
Note that, due to the NP-hardness of NMF [27], no polynomial-time algorithm currently exist that can obtain a globally optimal solution in general. Many approaches have been proposed for NMF initialization, for example based on k-means and spherical k-means by Wild et al. [29], on fuzzy c-means by Rezaei et al. [23], on nature inspired heuristic algorithms by Janecek and Tan [13], on Lanczos bidiagonalization by Wang et al. [28], on subtractive clustering by Casalino et al. [4], on independant component analysis by Kitamura and Ono [14], on the successive projection algorithm by Sauwen et al. [24], and on rank-one approximations by Liu and Tan [18], to name a few; see also Langville et al. [15].
In this paper, we focus on SVD-based initializations for NMF. Two of the most widely used methods are NNDSVD [2] and SVD-NMF [22] which are described in the next section. These methods suffer from the fact that the approximation error of the initial factors (W, H) increases as the rank increases which is not a desirable property for NMF initializations. Our key contribution is to provide a new SVD-based initialization that does not suffer from this shortcoming while (i) it generates sparse factors which not only provide storage efficiency [10] but also provide better part-based representations [4], [7] and resilience to noise [26], [30], and (ii) it only requires a truncated SVD of rank as opposed to a truncated SVD of rank r for the other SVD-based initializations.
Outline of the paper This paper is organized as follows. Section 2 will discuss our proposed solution in details, highlighting the differences with existing SVD-based initializations. In Section 3, we evaluate our proposed solution against other SVD-based initializations on dense and sparse data sets. Section 4 concludes the paper.
Section snippets
Nonnegative SVD with low-rank correction, a new SVD-based NMF initialization
The truncated SVD is a low-rank matrix approximation technique that approximates a given matrix as a sum of r rank-one terms made of singular triplets, where . Each singular triplet (ui, vi, σi) (1 ≤ i ≤ r) consists of two column vectors ui and vi which are the left and the right singular vectors, respectively, associated with the ith singular value (which we assume are sorted in nonincreasing order). We havewhere (.)T is the transpose of given matrix
Numerical experiments
In this section, we compare NNSVD-LRC with four NMF initializations. The first two are the SVD based NMF initializations presented in Section 2, namely NNDSVD and SVD-NMF. The third one, CR1-NMF, is a recent hybrid method combining clustering and the computation of rank-one SVDs [18]. The fourth one, SPKM, is one of the first proposed initialization for NMF using spherical k-means [29]. The code for CR1-NMF and SPKM are available from https://github.com/zhaoqiangliu/cr1-nmf.
All tests are
Conclusion
In this paper, we presented a novel SVD-based NMF initialization. Our motivation was to address the shortcomings of previously proposed SVD-based NMF initializations. Our newly proposed method, referred to as nonnegative singular value decomposition with low-rank correction (NNSVD-LRC), has the following advantages
- 1.
the initial error decreases as the factorization r increases,
- 2.
the sparsity of the initial factors (W, H) is close to 50%,
- 3.
it is computationally cheaper as it only requires the
Acknowledgement
The authors thank the anonymous reviewers for their insightful comments which helped improve the paper. The financial support of HEC Pakistan is highly acknowledged for granting PhD scholarship to the first author. NG acknowledges the support of the European Research Council (ERC starting grant no. 679515).
References (33)
- et al.
SVD based initialization: a head start for nonnegative matrix factorization
Pattern Recognit.
(2008) - et al.
Subtractive clustering for seeding non-negative matrix factorizations
Inf. Sci.
(2014) - et al.
Using underapproximations for sparse nonnegative matrix factorization
Pattern Recognit.
(2010) - et al.
Brain tumor detection by various image segmentation techniques with introduction to non negative matrix factorization
Brain
(2015) - et al.
An efficient initialization method for nonnegative matrix factorization
J. Appl. Sci.
(2011) - et al.
Graph regularized and sparse nonnegative matrix factorization with hard constraints for data representation
Neurocomputing
(2016) On the complexity of nonnegative matrix factorization
SIAM J. Optim.
(2010)- et al.
Multitask sparse nonnegative matrix factorization for joint spectral-spatial hyperspectral imagery denoising
IEEE Trans. Geosci. Remote Sens.
(2015) - et al.
Nonnegative matrices in the mathematical sciences
SIAM
(1994) - et al.
Resolving the sign ambiguity in the singular value decomposition
J. Chemom.
(2008)