Fast Nonnegative Matrix Factorization and Completion Using Nesterov Iterations

Abstract

In this paper, we aim to extend Nonnegative Matrix Factorization with Nesterov iterations (Ne-NMF)—well-suited to large-scale problems—to the situation when some entries are missing in the observed matrix. In particular, we investigate the Weighted and Expectation-Maximization strategies which both provide a way to process missing data. We derive their associated extensions named W-NeNMF and EM-W-NeNMF, respectively. The proposed approaches are then tested on simulated nonnegative low-rank matrix completion problems where the EM-W-NeNMF is shown to outperform state-of-the-art methods and the W-NeNMF technique.

A Proof of Lemmas 1 and 2

Following the structure of the proof in [9], the proof of Lemma 1 is shown by noticing that

(19)

Let us now focus on the proof of Lemma 2. The Lipschitz continuity of $\nabla _F \mathcal {J}_W$ can be shown by extending the proof in [9] to the weighted situation considered in this paper. However, the key point lies in the estimation of the “best” Lipschitz constant. Indeed, let us first recall that Q is a Lipschitz constant of $\nabla _F \mathcal {J}_W (G,.)$ if, for any matrices $F_1$ and $F_2$,

$$\begin{aligned} \left| \left| \nabla _F \mathcal {J}_W(G,F_1)-\nabla _F \mathcal {J}_W(G,F_2) \right| \right| _\mathcal {F}\le Q\cdot \left| \left| F_1-F_2 \right| \right| _\mathcal {F}. \end{aligned}$$

(20)

From Eq. (7), it is obvious that the larger is the majoring constant Q, the smaller is the associated gradient step size and thus the convergence speed.

Considering the gradient expression (14), we derive

$$\begin{aligned} \left| \left| \nabla _F \mathcal {J}_W(G,F_1)-\nabla _F \mathcal {J}_W(G,F_2) \right| \right| _\mathcal {F} = \left| \left| G^T\cdot \left( W^2\circ \left( G\cdot \left( F_1-F_2\right) \right) \right) \right| \right| _\mathcal {F}. \end{aligned}$$

(21)

The singular value decomposition of G yields

$$\begin{aligned} \left| \left| \nabla _F \mathcal {J}_W(G,F_1)-\nabla _F \mathcal {J}_W(G,F_2) \right| \right| _\mathcal {F} \le \left| \left| G \right| \right| _2 \left| \left| W^2\circ \left( G\cdot \left( F_1-F_2\right) \right) \right| \right| _\mathcal {F}. \end{aligned}$$

(22)

At this stage, we need to put W and G out of the norm to obtain a Lipschitz constant. This can be done by assuming that any column and any row of M contains at least one element—i.e., any column and row of W contains at least one 1—and thus noticing that

$$\begin{aligned} \left| \left| W^2\circ \left( G\cdot \left( F_1-F_2\right) \right) \right| \right| _\mathcal {F} \le \left| \left| G \cdot \left( F_1-F_2\right) \right| \right| _\mathcal {F}, \end{aligned}$$

(23)

which provides, using another singular value decomposition of G:

$$\begin{aligned} \left| \left| \nabla _F \mathcal {J}_W(G,F_1)-\nabla _F \mathcal {J}_W(G,F_2) \right| \right| _\mathcal {F} \le \left| \left| G \right| \right| _2^2 \left| \left| F_1-F_2 \right| \right| _\mathcal {F}, \end{aligned}$$

(24)

i.e., the Lipschitz constant for W-NeNMF is L.