Sparse trace norm regularization

Abstract

We study the problem of estimating multiple predictive functions from a dictionary of basis functions in the nonparametric regression setting. Our estimation scheme assumes that each predictive function can be estimated in the form of a linear combination of the basis functions. By assuming that the coefficient matrix admits a sparse low-rank structure, we formulate the function estimation problem as a convex program regularized by the trace norm and the $\ell _1$-norm simultaneously. We propose to solve the convex program using the accelerated gradient (AG) method; we also develop efficient algorithms to solve the key components in AG. In addition, we conduct theoretical analysis on the proposed function estimation scheme: we derive a key property of the optimal solution to the convex program; based on an assumption on the basis functions, we establish a performance bound of the proposed function estimation scheme (via the composite regularization). Simulation studies demonstrate the effectiveness and efficiency of the proposed algorithms.

Appendix

1.1 Operators $\mathcal S _0$ and $\mathcal S _1$

We define two operators, namely $\mathcal S _0$ and $\mathcal S _1$, on an arbitrary matrix pair (of the same size) based on Lemma $3.4$ in Recht et al. (2010), as summarized in the following lemma.

Lemma 1

Given any ${\varTheta }$ and ${\varDelta }$ of size $h \times k$, let ${rank} ({\varTheta }) = r$ and denote the SVD of ${\varTheta }$ as

$$\begin{aligned} {\varTheta } = U \left[ \begin{array}{c@{\quad }c} {\varSigma } &{} \mathbf{0} \\ \mathbf{0} &{} \mathbf{0} \end{array} \right] V^T, \end{aligned}$$

(26)

where $U \in \mathbb R ^{h \times h}$ and $V \in \mathbb R ^{k \times k}$ are orthogonal, and ${\varSigma } \in \mathbb R ^{r \times r}$ is diagonal consisting of the non-zero singular values on its main diagonal. Let

$$\begin{aligned} {\widehat{{\varDelta }}} = U^T {\varDelta } V = \left[ \begin{array}{c@{\quad }c} {{\widehat{{\varDelta }}}}_{11} &{} {{\widehat{{\varDelta }}}}_{12} \\ {{\widehat{{\varDelta }}}}_{21} &{} {{\widehat{{\varDelta }}}}_{22} \end{array} \right] , \end{aligned}$$

(27)

where ${{\widehat{{\varDelta }}}}_{11} \in \mathbb R ^{r \times r}$, ${{\widehat{{\varDelta }}}}_{12} \in \mathbb R ^{r \times (k-r)}$, ${{\widehat{{\varDelta }}}}_{21} \in \mathbb R ^{(h-r) \times r}$, and ${{\widehat{{\varDelta }}}}_{22} \in \mathbb R ^{(h-r) \times (k-r)}$. Define $\mathcal S _0$ and $\mathcal S _1$ as

$$\begin{aligned} \mathcal S _0 ({\varTheta }, {\varDelta }) = U \left[ \begin{array}{c@{\quad }c} {{\widehat{{\varDelta }}}}_{11} &{} {{\widehat{{\varDelta }}}}_{12} \\ {{\widehat{{\varDelta }}}}_{21} &{} \mathbf{0} \end{array} \right] V^T, \quad \mathcal S _1 ({\varTheta }, {\varDelta }) = U \left[ \begin{array}{c@{\quad }c} \mathbf{0} &{} \mathbf{0} \\ \mathbf{0} &{} {{\widehat{{\varDelta }}}}_{22} \end{array} \right] V^T. \end{aligned}$$

Then the following conditions hold: ${rank} \left( \mathcal S _0 ({\varTheta }, {\varDelta }) \right) \!\le \! 2 r$, ${\varTheta } \mathcal S _1 ({\varTheta }, {\varDelta })^T\!\!=\!0, {\varTheta }^T \mathcal S _1 ({\varTheta }, {\varDelta }) = 0$.

The result presented in Lemma 1 implies a condition under which the trace norm on a matrix pair is additive. From Lemma 1 we can easily verify that

$$\begin{aligned} \Vert {\varTheta } + \mathcal S _1 ({{\varTheta }, {\varDelta }})\Vert _* = \Vert {\varTheta }\Vert _* + \Vert \mathcal S _1 ({\varTheta }, {\varDelta })\Vert _*, \end{aligned}$$

(28)

for arbitrary ${\varTheta }$ and ${\varDelta }$ of the same size. To avoid clutter notation, we denote $\mathcal S _0 ({\varTheta }, {\varDelta })$ by $\mathcal S _0 ({\varDelta })$, and $\mathcal S _1 ({\varTheta }, {\varDelta })$ by $\mathcal S _1 ({\varDelta })$ throughout this paper, as the appropriate ${\varTheta }$ can be easily determined from the context.

1.2 Bound on trace norm

As a consequence of Lemma 1, we derive a bound on the trace norm of the matrices of interest as summarized below.

Corollary 1

Given an arbitrary matrix pair ${\widehat{{\varTheta }}}$ and ${\varTheta }$, let ${\varDelta } = {\widehat{{\varTheta }}} - {\varTheta }$. Then

$$\begin{aligned} \Vert {\widehat{{\varTheta }}} - {\varTheta } \Vert _* + \Vert {\varTheta }\Vert _* - \Vert {\widehat{{\varTheta }}}\Vert _* \le 2 \Vert \mathcal S _0 ({\varDelta })\Vert _*. \end{aligned}$$

Proof

From Lemma 1 we have ${\varDelta } = \mathcal S _0 ({\varDelta }) + \mathcal S _1 ({\varDelta })$ for the matrix pair ${\varTheta }$ and ${\varDelta }$. Moreover,

$$\begin{aligned} \Vert {\widehat{{\varTheta }}}\Vert _*&= \Vert {\varTheta } + \mathcal S _0 ({\varDelta }) + \mathcal S _1 ({\varDelta }) \Vert _* \ge \Vert {\varTheta } + \mathcal S _1 ({\varDelta }) \Vert _* - \Vert \mathcal S _0 ({\varDelta }) \Vert _* \nonumber \\&= \Vert {\varTheta } \Vert _* + \Vert \mathcal S _1 ({\varDelta }) \Vert _* - \Vert \mathcal S _0 ({\varDelta }) \Vert _*, \end{aligned}$$

(29)

where the inequality above follows from the triangle inequality and the last equality above follows from Eq. (28). Using the result in Eq. (29), we have

$$\begin{aligned} \Vert {\widehat{{\varTheta }}} - {\varTheta } \Vert _* + \Vert {\varTheta }\Vert _* - \Vert {\widehat{{\varTheta }}}\Vert _*&\le \Vert {\varDelta }\Vert _* + \Vert {\varTheta }\Vert _* - \Vert {\varTheta } \Vert _* - \Vert \mathcal S _1 ({\varDelta }) \Vert _* + \Vert \mathcal S _0 ({\varDelta }) \Vert _* \\&\le 2 \Vert \mathcal S _0 ({\varDelta })\Vert _*. \end{aligned}$$

We complete the proof of this corollary.

1.3 Bound on $\ell _1$-norm

Analogous to the bound on the trace norm in Corollary 1, we also derive a bound on the $\ell _1$-norm of the matrices of interest in the following lemma. For arbitrary matrices ${\varTheta }$ and ${\varDelta }$, we denote by $J ({\varTheta }) =\{(i,j) \}$ the coordinate set (the location set of nonzero entries) of ${\varTheta }$, and by $J({\varTheta })_\bot$ the associated complement (the location set of zero entries); we denote by ${\varDelta }_{J({\varTheta })}$ the matrix of the same entries as ${\varDelta }$ on the set $J({\varTheta })$ and of zero entries on the set $J({\varTheta })_\bot$. We now present a result associated with $J({\varTheta })$ and $J({\varTheta })_\bot$ in the following lemma. Note that a similar result for the vector case is presented in Bickel et al. (2009).

Lemma 2

Given a matrix pair ${\widehat{{\varTheta }}}$ and ${\varTheta }$ of the same size, the inequality below always holds

$$\begin{aligned} \Vert {\widehat{{\varTheta }}} - {\varTheta } \Vert _1 + \Vert {\varTheta } \Vert _1 - \Vert {\widehat{{\varTheta }}} \Vert _1 \le 2 \Vert {\widehat{{\varTheta }}}_{J({\varTheta })} - {\varTheta }_{J({\varTheta })} \Vert _1. \end{aligned}$$

Proof

It can be verified that the inequality

$$\begin{aligned} \Vert {\varTheta }_{J({\varTheta })} \Vert _1 - \Vert {\widehat{{\varTheta }}}_{J({\varTheta })} \Vert _1 \le \Vert ({\widehat{{\varTheta }}} - {\varTheta })_{J({\varTheta })} \Vert _1, \end{aligned}$$

and the equalities

$$\begin{aligned} {\varTheta }_{J({\varTheta })_{\bot }} = \mathbf{0}, \,\,\, \Vert ({\widehat{{\varTheta }}} - {\varTheta })_{J({\varTheta })_\bot } \Vert _1 - \Vert {\widehat{{\varTheta }}}_{J({\varTheta })_\bot } \Vert _1 = \mathbf{0}, \end{aligned}$$

hold. Therefore we can derive

$$\begin{aligned} \Vert {\widehat{{\varTheta }}} - {\varTheta } \Vert _1 + \Vert {\varTheta } \Vert _1 - \Vert {\widehat{{\varTheta }}} \Vert _1&= \Vert ({\widehat{{\varTheta }}} - {\varTheta })_{J({\varTheta })} \Vert _1 + \Vert ({\widehat{{\varTheta }}} - {\varTheta })_{J({\varTheta })_\bot } \Vert _1 \\&\quad + \Vert {\varTheta }_{J({\varTheta })} \Vert _1 \!+\! \Vert {\varTheta }_{J({\varTheta })_\bot } \Vert _1 - \Vert {\widehat{{\varTheta }}}_{J({\varTheta })} \Vert _1 - \Vert {\widehat{{\varTheta }}}_{J({\varTheta })_\bot } \Vert _1 \\&\le 2 \Vert ({\widehat{{\varTheta }}} - {\varTheta })_{J({\varTheta })} \Vert _1. \end{aligned}$$

This completes the proof of this lemma.

1.4 Concentration inequality

Lemma 3

Let $\sigma _{{X(l)}}$ be the maximum singular value of the matrix $\mathcal G _X \in \mathbb R ^{n \times h}$; let $W \in \mathbb R ^{n \times k}$ be the matrix of i.i.d entries as $w_{ij} \sim \mathcal N (0, \sigma _w^2 )$. Let $\lambda = {2 \sigma _{{X(l)}} \sigma _w \sqrt{n}} \left( 1 + \sqrt{\frac{k}{n}} + t \right) / {N}$. Then

$$\begin{aligned} \Pr \left( \frac{1}{N} \Vert W^T \mathcal G _X \Vert _2 \le \frac{\lambda }{2} \right) \ge 1 - \exp \left( - \frac{1}{2} n t^2 \right) . \end{aligned}$$

Proof

It is known (Szarek 1991) that a Gaussian matrix ${\widehat{W}} \in \mathbb R ^{n \times k}$ with $n \ge k$ and $\hat{w}_{ij} \sim \mathcal N (0, {1}/{n})$ satisfies

$$\begin{aligned} \mathrm{Pr} \left( \Vert {\widehat{W}}\Vert _2 > 1 + \sqrt{\frac{k}{n}} + t \right) \le \exp \left( - \frac{1}{2} n t^2 \right) , \end{aligned}$$

(30)

where $t$ is a universal constant. From the definition of the largest singular value, there exists a vector $b \in \mathbb R ^h$ of length $1$, i.e., $\Vert b\Vert _2 = 1$, such that $\Vert W^T \mathcal G _X\Vert _2 = \Vert W^T \mathcal G _X b\Vert _2 \le \Vert W \Vert _2 \Vert \mathcal G _X b\Vert _2 \le \sigma _{X(l)}\Vert W\Vert _2$. Since $w_{ij} / \left( \sigma _w \sqrt{n} \right) \sim \mathcal N (0, 1 / n)$, we have

$$\begin{aligned} \mathrm{Pr} \left( \frac{1}{N} \Vert W^T \mathcal G _X \Vert _2 > \frac{\lambda }{2} \right) \le \mathrm{Pr} \left( \frac{1}{N} {\sigma _{{X(l)}}} \left\| W \right\| _2 > \frac{\lambda }{2} \right) . \end{aligned}$$

Applying the result in Eq. (30) into the inequality above, we complete the proof of this lemma.