Elsevier

Pattern Recognition Letters

Volume 122, 1 May 2019, Pages 53-59
Pattern Recognition Letters

Improved SVD-based initialization for nonnegative matrix factorization using low-rank correction

https://doi.org/10.1016/j.patrec.2019.02.018Get rights and content

Highlights

  • We propose a novel SVD-based NMF initialization.

  • The initial error decreases as the factorization increases.

  • The average sparsity of the initial factors is close to 50%.

  • It is computationally cheaper than other SVD-based initializations.

  • It improves the solution via a highly efficient low-rank correction step.

Abstract

Due to the iterative nature of the most nonnegative matrix factorization (NMF) algorithms, initialization is a key aspect as it significantly influences both the convergence and the final solution obtained. Many initialization schemes have been proposed for NMF, among which one of the most popular class of methods are based on the singular value decomposition (SVD) and clustering. However, these SVD-based initializations as well as clustering based initializations (if they dense their right factor H), do not satisfy a rather natural condition, namely that the error should decrease as the rank of factorization increases. In this paper, we propose a novel SVD-based NMF initialization to specifically address this shortcoming by taking into account the SVD factors that were discarded to obtain a nonnegative initialization. This method, referred to as nonnegative SVD with low-rank correction (NNSVD-LRC), allows us to significantly reduce the initial error at a negligible additional computational cost using the low-rank structure of the discarded SVD factors. NNSVD-LRC has two other advantages compared to other NMF initializations: (1) it provably generates sparse initial factors, and (2) it is faster as it only requires to compute a truncated SVD of rank r2+1 where r is the factorization rank of the sought NMF decomposition (as opposed to a rank-r truncated SVD for other methods). We show on several standard dense and sparse data sets that our new method competes favorably with state-of-the-art SVD-based and clustering based initializations for NMF.

Introduction

Nonnegative matrix factorization (NMF) is the problem of approximating a input nonnegative matrix X as the product of two nonnegative matrices: Given XR0m×n and an integer r, find WR0m×r and HR0r×n such that X ≈ WH. NMF allows to reconstruct data using a purely additive model: each column of X is a nonnegative linear combination of the columns of W. For this reason, it is widely employed in research fields like image processing and computer vision [8], [21], data mining and document clustering [6], hyperspectral image analysis [19], [25], signal processing [31] and computational biology [20]; see also [5], [9] and the references therein.

To measure the quality of the NMF approximation, a distance metric should be chosen. In this paper, we focus on the most widely used one, namely the Frobenius norm, leading to the following optimization problemminWRm×r,HRr×nXWHF2suchthatW0andH0,where MF=i,jMi,j2 is Frobenius norm of a matrix M. Most algorithms tackling (1) use standard non-linear optimization schemes such as block coordinate descent methods hence initialization of the factors (W, H) is crucial in practice as it will influence

  • (i)

    the number of iterations needed for an algorithm to converge (in fact, if the initial point is closer to a local minimum, it will require less iterations to converge to it), and

  • (ii)

    the final solution to which the algorithm will converge.

Note that, due to the NP-hardness of NMF [27], no polynomial-time algorithm currently exist that can obtain a globally optimal solution in general. Many approaches have been proposed for NMF initialization, for example based on k-means and spherical k-means by Wild et al. [29], on fuzzy c-means by Rezaei et al. [23], on nature inspired heuristic algorithms by Janecek and Tan [13], on Lanczos bidiagonalization by Wang et al. [28], on subtractive clustering by Casalino et al. [4], on independant component analysis by Kitamura and Ono [14], on the successive projection algorithm by Sauwen et al. [24], and on rank-one approximations by Liu and Tan [18], to name a few; see also Langville et al. [15].

In this paper, we focus on SVD-based initializations for NMF. Two of the most widely used methods are NNDSVD [2] and SVD-NMF [22] which are described in the next section. These methods suffer from the fact that the approximation error XWHF2 of the initial factors (W, H) increases as the rank increases which is not a desirable property for NMF initializations. Our key contribution is to provide a new SVD-based initialization that does not suffer from this shortcoming while (i) it generates sparse factors which not only provide storage efficiency [10] but also provide better part-based representations [4], [7] and resilience to noise [26], [30], and (ii) it only requires a truncated SVD of rank r2+1, as opposed to a truncated SVD of rank r for the other SVD-based initializations.

Outline of the paper This paper is organized as follows. Section 2 will discuss our proposed solution in details, highlighting the differences with existing SVD-based initializations. In Section 3, we evaluate our proposed solution against other SVD-based initializations on dense and sparse data sets. Section 4 concludes the paper.

Section snippets

Nonnegative SVD with low-rank correction, a new SVD-based NMF initialization

The truncated SVD is a low-rank matrix approximation technique that approximates a given matrix XRm×n as a sum of r rank-one terms made of singular triplets, where 1rrank(X). Each singular triplet (ui, vi, σi) (1 ≤ i ≤ r) consists of two column vectors ui and vi which are the left and the right singular vectors, respectively, associated with the ith singular value (which we assume are sorted in nonincreasing order). We haveXXr=i=1rσiuiviT=UrΣrVrT,where (.)T is the transpose of given matrix

Numerical experiments

In this section, we compare NNSVD-LRC with four NMF initializations. The first two are the SVD based NMF initializations presented in Section 2, namely NNDSVD and SVD-NMF. The third one, CR1-NMF, is a recent hybrid method combining clustering and the computation of rank-one SVDs [18]. The fourth one, SPKM, is one of the first proposed initialization for NMF using spherical k-means [29]. The code for CR1-NMF and SPKM are available from https://github.com/zhaoqiangliu/cr1-nmf.

All tests are

Conclusion

In this paper, we presented a novel SVD-based NMF initialization. Our motivation was to address the shortcomings of previously proposed SVD-based NMF initializations. Our newly proposed method, referred to as nonnegative singular value decomposition with low-rank correction (NNSVD-LRC), has the following advantages

  • 1.

    the initial error decreases as the factorization r increases,

  • 2.

    the sparsity of the initial factors (W, H) is close to 50%,

  • 3.

    it is computationally cheaper as it only requires the

Acknowledgement

The authors thank the anonymous reviewers for their insightful comments which helped improve the paper. The financial support of HEC Pakistan is highly acknowledged for granting PhD scholarship to the first author. NG acknowledges the support of the European Research Council (ERC starting grant no. 679515).

References (33)

  • A. Cichocki et al.

    Nonnegative Matrix and Tensor Factorizations: Applications to Exploratory Multi-Way Data Analysis and Blind Source Separation

    (2009)
  • R. Du et al.

    Hybrid clustering based on content and connection structure using joint nonnegative matrix factorization

    J. Global Optim.

    (2017)
  • M. Elad et al.

    On the role of sparse and redundant representations in image processing

    Proc. IEEE

    (2010)
  • T. Ensari

    Character recognition analysis with nonnegative matrix factorization

    Int. J. Comput.

    (2016)
  • N. Gillis

    The why and how of nonnegative matrix factorization

    Regularization, Optimization, Kernels, and Support Vector Machines

    (2014)
  • N. Gillis et al.

    Accelerated multiplicative updates and hierarchical ALS algorithms for nonnegative matrix factorization

    Neural Comput.

    (2012)
  • Cited by (0)

    View full text