Randomized Algorithms for Low-Rank Matrix Factorizations: Sharp Performance Bounds

Abstract

The development of randomized algorithms for numerical linear algebra, e.g. for computing approximate QR and SVD factorizations, has recently become an intense area of research. This paper studies one of the most frequently discussed algorithms in the literature for dimensionality reduction—specifically for approximating an input matrix with a low-rank element. We introduce a novel and rather intuitive analysis of the algorithm in [6], which allows us to derive sharp estimates and give new insights about its performance. This analysis yields theoretical guarantees about the approximation error and at the same time, ultimate limits of performance (lower bounds) showing that our upper bounds are tight. Numerical experiments complement our study and show the tightness of our predictions compared with empirical observations.

Appendix

We use well-known bounds to control the expectation of the extremal singular values of a Gaussian matrix. These bounds are recalled in [8], though known earlier.

Lemma 4.1

If $m>n$ and $A$ is a $m \times n$ matrix with i.i.d. $\mathcal {N}(0,1)$ entries, then

$$\begin{aligned} \sqrt{m} - \sqrt{n} \le \mathbb {E}\sigma _{\min }(A)\le \mathbb {E}\sigma _{\max }(A) \le \sqrt{m} + \sqrt{n}. \end{aligned}$$

Next, we control the expectation of the norm of the pseudo-inverse $A^\dagger$ of a Gaussian matrix $A$.

Lemma 4.2

In the setup of Lemma 4.1, we have

$$\begin{aligned} \frac{1}{\sqrt{m-n}} \le \mathbb {E}\Vert A^\dagger \Vert \le e \frac{\sqrt{m}}{m-n}. \end{aligned}$$

Proof

The upper bound is the same as is used in [3] and follows from the work of [1]. For the lower bound, set $B = (A^*A)^{-1}$ which has an inverse Wishart distribution, and observe that

$$\begin{aligned} \Vert A^\dagger \Vert ^2 = \Vert B\Vert \ge B_{11}, \end{aligned}$$

where $B_{11}$ is the entry in the $(1,1)$ position. It is known that $B_{11} \mathop {=}\limits ^{d} 1/Y$, where $Y$ is distributed as a chi-square variable with $d = m - n + 1$ degrees of freedom [5, Page 72]. Hence,

$$\begin{aligned} \mathbb {E}\Vert A^\dagger \Vert \ge \mathbb {E}\frac{1}{\sqrt{Y}} \ge \frac{1}{\sqrt{\mathbb {E}Y}} = \frac{1}{\sqrt{m-n+1}}. \end{aligned}$$

$\square$

The limit laws below are taken from [10] and [2].

Lemma 4.3

Let $A_{m,n}$ be a sequence of $m \times n$ matrix with i.i.d. $\mathcal {N}(0,1)$ entries such that $\lim _{n \rightarrow \infty } m/n = c \ge 1$. Then

$$\begin{aligned} \frac{1}{\sqrt{n}} \sigma _{\min }(A_{m,n})&\,{a.s. \over \rightarrow }\,\sqrt{c} - 1\\ \frac{1}{\sqrt{n}} \sigma _{\max }(A_{m,n})&\,{a.s. \over \rightarrow }\,\sqrt{c}+1. \end{aligned}$$