Accelerating adaptive online learning by matrix approximation

Abstract

Adaptive subgradient methods are able to leverage the second-order information of functions to improve the regret and have become popular for online learning and optimization. According to the amount of information used, these methods can be divided into diagonal-matrix version (ADA-DIAG) and full-matrix version (ADA-FULL). In practice, ADA-DIAG is the most commonly adopted instead of ADA-FULL, because ADA-FULL is computationally intractable in high dimensions though it has smaller regret when gradients are correlated. In this paper, we propose to employ techniques of matrix approximation to accelerate ADA-FULL and develop two methods based on random projections. Compared with ADA-FULL, at each iteration, our methods reduce the space complexity from $O(d^2)$ to $O(\tau d)$ and the time complexity from $O(d^3)$ to $O(\tau ^2 d)$ where d is the dimensionality of the data and $\tau \ll d$ is the number of random projections. Experimental results about online convex optimization and training convolutional neural networks show that our methods are comparable to ADA-FULL and outperform other state-of-the-art algorithms including ADA-DIAG.

A Theoretical analysis

In this section, we provide proofs of Theorems 1 and 2.

1.1 Supporting results

The following results are used throughout our analysis.

Lemma 1

(Proposition 3 of [7]). Let sequence $\{\varvec{\beta }_t\}$ be generated by ADA-DP. We have

$$\begin{aligned} R(T)&\le \frac{1}{\eta }\sum \limits _{t=1}^{T-1}\left[ B_{\Psi _{t+1}}(\varvec{\beta }^*,\varvec{\beta }_{t+1})-B_{\Psi _{t}}(\varvec{\beta }^*,\varvec{\beta }_{t+1})\right] \\&\quad +\frac{1}{\eta }B_{\Psi _1}(\varvec{\beta }^*,\varvec{\beta }_1)+\frac{\eta }{2}\sum \limits _{t=1}^{T}\Vert f^\prime _t(\varvec{\beta }_t)\Vert ^2_{\Psi _{t}^*}. \end{aligned}$$

Lemma 2

Let ${X}_t=\sum _{i=1}^t{\mathbf {x}}_i{\mathbf {x}}_i^\top$ and ${A}^\dagger$ denote the pseudo-inverse of A, then

$$\begin{aligned} \sum \limits _{t=1}^T\left\langle {\mathbf {x}}_t,({X}_t^{1/2})^{\dagger }{\mathbf {x}}_t\right\rangle&\le 2\sum \limits _{t=1}^T\left\langle {\mathbf {x}}_t,({X}_T^{1/2})^\dagger {\mathbf {x}}_t\right\rangle \\&=2\mathrm{tr}({X}_T^{1/2}). \end{aligned}$$

Lemma 2 can be proved in the same way as Lemma 10 of [7].

Theorem 3

(Theorem 2.3 of [32]). Let $0<\epsilon ,\delta <1$ and ${S}=\frac{1}{\sqrt{k}}{R}\in {\mathbb {R}}^{k\times n}$ where the entries of R are independent standard normal random variables. Then if $k=\Theta (\frac{d+\log (1/\delta )}{\epsilon ^{2}})$, then for any fixed $n\times d$ matrix A, with probability $1-\delta$, simultaneously for all ${\mathbf {x}}\in {\mathbb {R}}^d$,

$$\begin{aligned} (1-\epsilon )\Vert {A}{\mathbf {x}}\Vert _2^2\le \Vert {S}{A}{\mathbf {x}}\Vert _2^2\le (1+\epsilon )\Vert {A}{\mathbf {x}}\Vert _2^2. \end{aligned}$$

Based on the above theorem, we derive the following corollary.

Corollary 1

Let $0<\epsilon ,\delta <1$ and each entry of ${\mathbf {r}}_t\in {\mathbb {R}}^{\tau }$ is a Gaussian random variable independently drawn from ${\mathcal {N}}(0,1/\tau )$. Then, if $\tau =\Omega (\frac{r+\log (T/\delta )}{\epsilon ^{2}})$, with probability $1-\delta$, simultaneously for all $t=1,\ldots ,T$,

$$\begin{aligned} (1-\epsilon ){C}_t^\top {C}_t\preceq {S}_t^\top {S}_t\preceq (1+\epsilon ){C}_t^\top {C}_t. \end{aligned}$$

Theorem 4

(Theorem 10 of [35]). Let ${C}=\mathrm{diag}(c_1,\ldots ,c_p)$ and ${S}=\mathrm{diag}(s_1,\ldots ,s_p)$ be $p\times p$ diagonal matrices, where $c_i\ne 0$ and $c_i^2+s_i^2=1$ for all i. Let ${R}\in {\mathbb {R}}^{p\times n}$ be a Gaussian random matrix. Let ${M}={C}^2+\frac{1}{n}{S}{R}{R}^\top {S}$ and $r=\sum _is_i^2$.

$$\begin{aligned}&\Pr (\lambda _1({M})\ge 1+t)\le q\cdot \exp \left( -\frac{cnt^2}{\max _i(s_i^2)r}\right) ,\\&\Pr (\lambda _p({M})\le 1-t)\le q\cdot \exp \left( -\frac{cnt^2}{\max _i(s_i^2)r}\right) , \end{aligned}$$

where the constant c is at least 1 / 32, and q is the rank of S.

Based on the above theorem, we derive the following corollary.

Corollary 2

Let $c\ge 1/32$, $\alpha >0$, $\sigma _{ti}^2=\lambda _i({C}_t^\top {C}_t)$, ${\tilde{r}}_t=\sum _i\frac{\sigma ^2_{ti}}{\alpha +\sigma ^2_{ti}}$, ${\tilde{r}}_*=\max \limits _{k\le t\le T}{\tilde{r}}_t$ and ${\sigma ^2_{*1}}=\max \limits _{1\le t\le T}{\sigma ^2_{t1}}$. Let ${K}_t = \alpha {I}_d+{C}_t^\top {C}_t$, $\tilde{{K}}_t = \alpha {I}_d+{S}_t^\top {S}_t$ and $\tilde{{I}}_t={K}_t^{-1/2}\tilde{{K}}_t{K}_t^{-1/2}$. If $\tau \ge \frac{{\tilde{r}}_*{\sigma ^2_{*1}}}{c\epsilon ^2(\alpha +{\sigma ^2_{*1}})}\log \frac{2dT}{\delta }$, with probability at least $1-\delta$, simultaneously for all $t=1,\ldots ,T$,

$$\begin{aligned} (1-\epsilon ){I}_d\preceq \tilde{{I}}_t\preceq (1+\epsilon ){I}_d. \end{aligned}$$

1.2 Proof of Theorem 1

Let $\widetilde{{X}}_t$ denote ${S}_t^\top {S}_t$. First, we consider bounding the first term in the upper bound of Lemma 1. With probability $1-\delta$, we have

$$\begin{aligned}&B_{\Psi _{t+1}}(\varvec{\beta }^*,\varvec{\beta }_{t+1})-B_{\Psi _{t}}(\varvec{\beta }^*,\varvec{\beta }_{t+1})\\&\quad =\frac{1}{2}\left\langle \varvec{\beta }^*-\varvec{\beta }_{t+1},(\widetilde{{X}}_{t+1}^{1/2}-\widetilde{{X}}_t^{1/2})(\varvec{\beta }^*-\varvec{\beta }_{t+1})\right\rangle \\&\quad \le \frac{1}{2}\left\langle \varvec{\beta }^*-\varvec{\beta }_{t+1},\sqrt{1+\epsilon }{X}_{t+1}^{1/2}(\varvec{\beta }^*-\varvec{\beta }_{t+1})\right\rangle \\&\qquad -\frac{1}{2}\left\langle \varvec{\beta }^*-\varvec{\beta }_{t+1},\sqrt{1-\epsilon }{X}_{t}^{1/2}(\varvec{\beta }^*-\varvec{\beta }_{t+1})\right\rangle \\&\quad \le \frac{1}{2}\left\langle \varvec{\beta }^*-\varvec{\beta }_{t+1},({X}_{t+1}^{1/2}-{X}_{t}^{1/2})(\varvec{\beta }^*-\varvec{\beta }_{t+1})\right\rangle \\&\qquad +\frac{\epsilon }{4}\left\langle \varvec{\beta }^*-\varvec{\beta }_{t+1},({X}_{t+1}^{1/2}+{X}_{t}^{1/2})(\varvec{\beta }^*-\varvec{\beta }_{t+1})\right\rangle \\&\quad \le \frac{1}{2}\Vert \varvec{\beta }^*-\varvec{\beta }_{t+1}\Vert _2^2\Vert ({X}_{t+1}^{1/2}-{X}_{t}^{1/2})\Vert \\&\qquad +\frac{\epsilon }{4}\left\langle \varvec{\beta }^*-\varvec{\beta }_{t+1},({X}_{t+1}^{1/2}+{X}_{t}^{1/2})(\varvec{\beta }^*-\varvec{\beta }_{t+1})\right\rangle \\&\quad \le \frac{1}{2}\Vert \varvec{\beta }^*-\varvec{\beta }_{t+1}\Vert _2^2\mathrm{tr}({X}_{t+1}^{1/2}-{X}_{t}^{1/2})\\&\qquad +\frac{\epsilon }{4}\left\langle \varvec{\beta }^*-\varvec{\beta }_{t+1},({X}_{t+1}^{1/2}+{X}_{t}^{1/2})(\varvec{\beta }^*-\varvec{\beta }_{t+1})\right\rangle \end{aligned}$$

where the first inequality is due to Corollary 1.

Thus, we can get

$$\begin{aligned} \begin{aligned}&\sum \limits _{t=1}^{T-1}\left[ B_{\Psi _{t+1}}(\varvec{\beta }^*,\varvec{\beta }_{t+1})-B_{\Psi _{t}}(\varvec{\beta }^*,\varvec{\beta }_{t+1})\right] \\&\quad \le \frac{1}{2}\sum \limits _{t=1}^{T-1}\Vert \varvec{\beta }^*-\varvec{\beta }_{t+1}\Vert _2^2\mathrm{tr}({X}_{t+1}^{1/2}-{X}_{t}^{1/2})\\&\qquad +\frac{\epsilon }{4}\sum \limits _{t=1}^{T-1}\left\langle \varvec{\beta }^*-\varvec{\beta }_{t+1},({X}_{t+1}^{1/2}+{X}_{t}^{1/2})(\varvec{\beta }^*-\varvec{\beta }_{t+1})\right\rangle \\&\quad \le \frac{1}{2}\max \limits _{t\le T}\Vert \varvec{\beta }^*-\varvec{\beta }_{t}\Vert _2^2\mathrm{tr}({X}_{T}^{1/2})-\frac{1}{2}\Vert \varvec{\beta }^*-\varvec{\beta }_{1}\Vert _2^2\mathrm{tr}({X}_{1}^{1/2})\\&\qquad +\frac{\epsilon }{2}\max \limits _{t\le T}\Vert \varvec{\beta }^*-\varvec{\beta }_{t}\Vert _2^2\sum \limits _{t=1}^{T}\Vert {X}_{t}^{1/2}\Vert \\&\qquad -\frac{\epsilon }{4}\Vert \varvec{\beta }^*-\varvec{\beta }_{1}\Vert _2^2\mathrm{tr}({X}_{1}^{1/2}). \end{aligned} \end{aligned}$$

(3)

Note that $\varvec{\beta }_1={\mathbf {0}}$, then

$$\begin{aligned} \begin{aligned} B_{\Psi _1}(\varvec{\beta }^*,\varvec{\beta }_1)&=\frac{1}{2}\left\langle \varvec{\beta }^*,(\sigma {I}_d+\widetilde{{X}}_1^{1/2})\varvec{\beta }^*\right\rangle \\&\le \frac{1}{2}\sigma \Vert \varvec{\beta }^{*}\Vert _2^2+\frac{2+\epsilon }{4}\Vert \varvec{\beta }^{*}\Vert _2^2\mathrm{tr}({X}_{1}^{1/2}) \end{aligned} \end{aligned}$$

(4)

where the inequality is due to Corollary 1.

Then, we consider the bound of $\sum _{t=1}^{T}\Vert f^\prime _t(\varvec{\beta }_t)\Vert ^2_{\Psi _{t}^*}$. With probability $1-\delta$, we have

$$\begin{aligned}&\frac{1}{2}\Vert f^\prime _t(\varvec{\beta }_t)\Vert ^2_{\Psi _{t}^*}=\left\langle {\mathbf {g}}_t,(\sigma {I}_d+\widetilde{{X}}_t^{1/2})^{-1}{\mathbf {g}}_t\right\rangle \\&\quad \le \frac{1}{\sqrt{1-\epsilon }}\left\langle {\mathbf {g}}_t,({X}_{t}^{\dagger })^{1/2}{\mathbf {g}}_t\right\rangle =\frac{l^\prime (\varvec{\beta }_t^\top {\mathbf {x}}_t)^2}{\sqrt{1-\epsilon }}\left\langle {\mathbf {x}}_t,({X}_{t}^{\dagger })^{1/2}{\mathbf {x}}_t\right\rangle \end{aligned}$$

where the inequality is due to Corollary 1. According to Lemma 2, we have

$$\begin{aligned} \begin{aligned}&\sum \limits _{t=1}^{T}\Vert f^\prime _t(\varvec{\beta }_t)\Vert ^2_{\Psi _{t}^*}\\&\quad \le \sum \limits _{t=1}^{T}\frac{2l^\prime (\varvec{\beta }_t^\top {\mathbf {x}}_t)^2}{\sqrt{1-\epsilon }}\left\langle {\mathbf {x}}_t,({X}_{t}^{\dagger })^{1/2}{\mathbf {x}}_t\right\rangle \\&\quad \le \max \limits _{t\le T}l^\prime (\varvec{\beta }_t^\top {\mathbf {x}}_t)^2\frac{2}{\sqrt{1-\epsilon }}\sum \limits _{t=1}^{T}\left\langle {\mathbf {x}}_t,({X}_{t}^{\dagger })^{1/2}{\mathbf {x}}_t\right\rangle \\&\quad \le \frac{4}{\sqrt{1-\epsilon }}\max \limits _{t\le T}l^\prime (\varvec{\beta }_t^\top {\mathbf {x}}_t)^2\mathrm{tr}({X}_T^{1/2}). \end{aligned} \end{aligned}$$

(5)

We complete the proof by substituting (3), (4) and (5) into Lemma 1.

1.3 Proof of Theorem 2

Inspired by the proof of Theorem 1, we can derive Theorem 2 by, respectively, bounding each term in the upper bound of Lemma 1. Before that, we need to derive the lower and upper bounds of $({S}_t^\top {S}_t)^{1/2}$ based on Corollary 2.

Let the SVD of ${C}_t^\top$ be ${C}_t^\top = {U}{\Sigma }{V}^\top$ where ${U}\in {\mathbb {R}}^{d\times d},{\Sigma }\in {\mathbb {R}}^{d\times d},{V}\in {\mathbb {R}}^{t\times d}$. Let ${K}_t = \alpha {I}_d+{C}_t^\top {C}_t$, $\tilde{{K}}_t = \alpha {I}_d+{S}_t^\top {S}_t$ and $\tilde{{I}}_t={K}_t^{-1/2}\tilde{{K}}_t{K}_t^{-1/2}$. According to Corollary 2, with probability at least $1-\delta$, simultaneously for all $t=1,\ldots ,T$,

$$\begin{aligned} {S}_t^\top {S}_t&=\tilde{{K}}_t - \alpha {I}_d={K}_t^{1/2}\tilde{{I}}_t {K}_t^{1/2}- \alpha {I}_d\\&\preceq (1+\epsilon ){K}_t-\alpha {I}_d=(1+\epsilon ){C}_t^\top {C}_t+\epsilon \alpha {I}_d\\&={U}\big ((1+\epsilon ){\Sigma }{\Sigma }+\epsilon \alpha {I}_d\big ){U}^\top \end{aligned}$$

and

$$\begin{aligned} {S}_t^\top {S}_t+\epsilon \alpha {I}_d&=\tilde{{K}}_t - \alpha {I}_d+\epsilon \alpha {I}_d\\&={K}_t^{1/2}\tilde{{I}}_t{K}_t^{1/2}- \alpha {I}_d+\epsilon \alpha {I}_d\\&\succeq (1-\epsilon ){K}_t-\alpha {I}_d+\epsilon \alpha {I}_d\\&=(1-\epsilon ){C}_t^\top {C}_t. \end{aligned}$$

Then simultaneously for all $t=1,\ldots ,T$, we have

$$\begin{aligned} \begin{aligned} ({S}_t^\top {S}_t)^{1/2}&\preceq \sqrt{1+\epsilon }{U}({\Sigma }{\Sigma })^{1/2}{U}^\top +\sqrt{\epsilon \alpha }{U}{I}_d{U}^\top \\&=\sqrt{1+\epsilon }{X}_{t}^{1/2}+\sqrt{\epsilon \alpha }{I}_d \end{aligned} \end{aligned}$$

(6)

and

$$\begin{aligned} \begin{aligned} ({S}_t^\top {S}_t)^{1/2}&=({S}_t^\top {S}_t)^{1/2}+\sqrt{\epsilon \alpha }{I}_d-\sqrt{\epsilon \alpha }{I}_d\\&\succeq \big (({S}_t^\top {S}_t)+\epsilon \alpha {I}_d\big )^{1/2}-\sqrt{\epsilon \alpha }{I}_d\\&\succeq \sqrt{1-\epsilon }{X}_{t}^{1/2}-\sqrt{\epsilon \alpha }{I}_d. \end{aligned} \end{aligned}$$

(7)

Then we consider bounding the first term in the upper bound of Lemma 1. Let $\widetilde{{X}}_t$ denote ${S}_t^\top {S}_t$. Simultaneously for all $t=1,\ldots ,T$, we have

$$\begin{aligned}&B_{\Psi _{t+1}}(\varvec{\beta }^*,\varvec{\beta }_{t+1})-B_{\Psi _{t}}(\varvec{\beta }^*,\varvec{\beta }_{t+1})\\&\quad =\frac{1}{2}\left\langle \varvec{\beta }^*-\varvec{\beta }_{t+1},(\widetilde{{X}}_{t+1}^{1/2}-\widetilde{{X}}_t^{1/2})(\varvec{\beta }^*-\varvec{\beta }_{t+1})\right\rangle \\&\quad \le \frac{1}{2}\left\langle \varvec{\beta }^*-\varvec{\beta }_{t+1},\sqrt{1+\epsilon }{X}_{t+1}^{1/2}(\varvec{\beta }^*-\varvec{\beta }_{t+1})\right\rangle \\&\qquad -\frac{1}{2}\left\langle \varvec{\beta }^*-\varvec{\beta }_{t+1},\sqrt{1-\epsilon }{X}_{t}^{1/2})(\varvec{\beta }^*-\varvec{\beta }_{t+1})\right\rangle \\&\qquad +\frac{1}{2}\left\langle \varvec{\beta }^*-\varvec{\beta }_{t+1},2\sqrt{\epsilon \alpha }{I}_{d}(\varvec{\beta }^*-\varvec{\beta }_{t+1})\right\rangle \\&\quad =\frac{1}{2}\left\langle \varvec{\beta }^*-\varvec{\beta }_{t+1},\sqrt{1+\epsilon }{X}_{t+1}^{1/2}(\varvec{\beta }^*-\varvec{\beta }_{t+1})\right\rangle \\&\qquad -\frac{1}{2}\left\langle \varvec{\beta }^*-\varvec{\beta }_{t+1},\sqrt{1-\epsilon }{X}_{t}^{1/2})(\varvec{\beta }^*-\varvec{\beta }_{t+1})\right\rangle \\&\qquad +\sqrt{\epsilon \alpha }\Vert (\varvec{\beta }^*-\varvec{\beta }_{t+1})\Vert _2^2\\&\quad \le \frac{1}{2}\Vert \varvec{\beta }^*-\varvec{\beta }_{t+1}\Vert _2^2\mathrm{tr}({X}_{t+1}^{1/2}-{X}_{t}^{1/2})\\&\qquad +\frac{\epsilon }{4}\left\langle \varvec{\beta }^*-\varvec{\beta }_{t+1},({X}_{t+1}^{1/2}+{X}_{t}^{1/2})(\varvec{\beta }^*-\varvec{\beta }_{t+1})\right\rangle \\&\qquad +\sqrt{\epsilon \alpha }\Vert (\varvec{\beta }^*-\varvec{\beta }_{t+1})\Vert _2^2 \end{aligned}$$

where the first inequality is due to (6), (7) and the last inequality has been proved in the proof of Theorem 1.

Thus, we can get

$$\begin{aligned} \begin{aligned}&\sum \limits _{t=1}^{T-1}\left[ B_{\Psi _{t+1}}(\varvec{\beta }^*,\varvec{\beta }_{t+1})-B_{\Psi _{t}}(\varvec{\beta }^*,\varvec{\beta }_{t+1})\right] \\&\quad \le \frac{1}{2}\max \limits _{t\le T}\Vert \varvec{\beta }^*-\varvec{\beta }_{t}\Vert _2^2\mathrm{tr}({X}_{T}^{1/2})-\frac{1}{2}\Vert \varvec{\beta }^*-\varvec{\beta }_{1}\Vert _2^2\mathrm{tr}({X}_{1}^{1/2})\\&\qquad +\frac{\epsilon }{2}\max \limits _{t\le T}\Vert \varvec{\beta }^*-\varvec{\beta }_{t}\Vert _2^2\sum \limits _{t=1}^{T}\Vert {X}_{t}^{1/2}\Vert \\&\qquad -\frac{\epsilon }{4}\Vert \varvec{\beta }^*-\varvec{\beta }_{1}\Vert _2^2\mathrm{tr}({X}_{1}^{1/2})\\&\qquad +\sqrt{\epsilon \alpha }(T-1)\max \limits _{t\le T}\Vert \varvec{\beta }^*-\varvec{\beta }_{t}\Vert _2^2. \end{aligned} \end{aligned}$$

(8)

Note that $\varvec{\beta }_1={\mathbf {0}}$, then

$$\begin{aligned} \begin{aligned} B_{\Psi _1}(\varvec{\beta }^*,\varvec{\beta }_1)&=\frac{1}{2}\left\langle \varvec{\beta }^*,(\sigma {I}_d+\widetilde{{X}}_1^{1/2})\varvec{\beta }^*\right\rangle \\&\le \frac{1}{2}\sigma \Vert \varvec{\beta }^{*}\Vert _2^2+\frac{2+\epsilon }{4}\Vert \varvec{\beta }^{*}\Vert _2^2\mathrm{tr}({X}_{1}^{1/2})\\&\quad +\frac{1}{2}\sqrt{\epsilon \alpha }\Vert \varvec{\beta }^*\Vert _2^2. \end{aligned} \end{aligned}$$

(9)

Before considering the upper bound of $\sum _{t=1}^{T}\Vert f^\prime _t(\varvec{\beta }_t)\Vert ^2_{\Psi _{t}^*}$, we need to derive the upper bound of ${H}_t^{-1}$.

Let the SVD of ${S}_t^\top$ be ${S}_t^\top = {U}{\Sigma }{V}^\top$ where ${U}\in {\mathbb {R}}^{d\times d},{\Sigma }\in {\mathbb {R}}^{d\times d},{V}\in {\mathbb {R}}^{t\times d}$. We also have, for all $t=1,\ldots ,T$,

$$\begin{aligned} {H}_t&=\sigma {I}_d+({S}^\top _{t}{S}_{t})^{1/2}={U}\big (\sigma {I}_d+({\Sigma }{\Sigma })^{1/2}\big ){U}^\top \\&\succeq {U}\big (\alpha {I}_d+({\Sigma }{\Sigma })\big )^{1/2}{U}^\top =(\alpha {I}_d+{S}^\top _{t}{S}_{t})^{1/2} \end{aligned}$$

due to $\sigma \ge \sqrt{\alpha }\ge \sqrt{\lambda _i({S}_t^\top {S}_t)+\alpha }-\sqrt{\lambda _i({S}_t^\top {S}_t)}$ for all $i=1,\ldots ,d$.

Then according to Corollary 2, with probability at least $1-\delta$, simultaneously for all $t=1,\ldots ,T$,

$$\begin{aligned} {H}_t^{-1}&\preceq \big ((\alpha {I}_d+{S}^\top _{t}{S}_{t})^{1/2}\big )^{-1}=\big (({K}_t^{1/2}\tilde{{I}}_t{K}_t^{1/2})^{-1}\big )^{1/2}\\&\preceq \frac{1}{\sqrt{1-\epsilon }}({K}_t^{-1})^{1/2}=\frac{1}{\sqrt{1-\epsilon }}\big ((\alpha {I}_d+{X}_{t})^{-1}\big )^{1/2}. \end{aligned}$$

Thus, we can get

$$\begin{aligned} \Vert f^\prime _t(\varvec{\beta }_t)\Vert ^2_{\Psi _{t}^*}&=2\left\langle {\mathbf {g}}_t,{H}_t^{-1}{\mathbf {g}}_t\right\rangle \\&\le \frac{2}{\sqrt{1-\epsilon }}\left\langle {\mathbf {g}}_t,\big ((\alpha {I}_d+{X}_{t})^{-1}\big )^{1/2}{\mathbf {g}}_t\right\rangle \\&=\frac{2l^\prime (\varvec{\beta }_t^\top {\mathbf {x}}_t)^2}{\sqrt{1-\epsilon }}\left\langle {\mathbf {x}}_t,({X}_{t}^{\dagger })^{1/2}{\mathbf {x}}_t\right\rangle . \end{aligned}$$

According to Lemma 2, we have

$$\begin{aligned} \begin{aligned}&\sum \limits _{t=1}^{T}\Vert f^\prime _t(\varvec{\beta }_t)\Vert ^2_{\Psi _{t}^*}\\&\quad \le \frac{2}{\sqrt{1-\epsilon }}\max \limits _{t\le T}l^\prime (\varvec{\beta }_t^\top {\mathbf {x}}_t)^2\sum \limits _{t=1}^{T}\left\langle {\mathbf {x}}_t,({X}_{t}^{\dagger })^{1/2}{\mathbf {x}}_t\right\rangle \\&\quad \le \frac{4}{\sqrt{1-\epsilon }}\max \limits _{t\le T}l^\prime (\varvec{\beta }_t^\top {\mathbf {x}}_t)^2\mathrm{tr}({X}_T^{1/2}). \end{aligned} \end{aligned}$$

(10)

We complete the proof by substituting (8), (9) and (10) into Lemma 1.

1.4 Proof of Corollary 1

Let ${C}_t={U}{\Sigma } {V}^\top$ be the singular value decomposition of ${C}_t$. Notice that ${U}\in {\mathbb {R}}^{t\times r}, {\Sigma } {V}^\top \in {\mathbb {R}}^{r\times d}$. According to Theorem 3, we have if $\tau =\Theta (\frac{r+\log (1/\delta )}{\epsilon ^{2}})$, then simultaneously $\forall {\mathbf {x}} \in {\mathbb {R}}^{r}$, with probability $1-\delta$,

$$\begin{aligned} (1-\epsilon )\Vert {U}{\mathbf {x}}\Vert _2^2\le \Vert {R_t}{U}{\mathbf {x}}\Vert _2^2\le (1+\epsilon )\Vert {U}{\mathbf {x}}\Vert _2^2 \end{aligned}$$

Let ${\mathbf {y}}\in {\mathbb {R}}^d$ be arbitrary vector, then ${C}_t{\mathbf {y}}={U}{\Sigma }{V}^\top {\mathbf {y}}={U}{\mathbf {x}}$ where ${\mathbf {x}}={\Sigma }{V}^\top {\mathbf {y}}\in {\mathbb {R}}^{r}$.

Then we have

$$\begin{aligned} {\mathbf {y}}^\top {S}_t^\top {S}_t{\mathbf {y}}&={\mathbf {y}}^\top {C}_t^\top {R}^\top _t{R}_t{C}_t{\mathbf {y}}=\Vert {R_t}{U}{\mathbf {x}}\Vert _2^2\\&\le (1+\epsilon )\Vert {U}{\mathbf {x}}\Vert _2^2=(1+\epsilon ){\mathbf {y}}^\top {C}_t^\top {C}_t{\mathbf {y}} \end{aligned}$$

and

$$\begin{aligned} {\mathbf {y}}^\top {S}_t^\top {S}_t{\mathbf {y}}&={\mathbf {y}}^\top {C}_t^\top {R}^\top _t{R}_t{C}_t{\mathbf {y}}=\Vert {R_t}{U}{\mathbf {x}}\Vert _2^2\\&\ge (1-\epsilon )\Vert {U}{\mathbf {x}}\Vert _2^2=(1-\epsilon ){\mathbf {y}}^\top {C}_t^\top {C}_t{\mathbf {y}}. \end{aligned}$$

Then, we have $(1-\epsilon ){C}_t^\top {C}_t\preceq {S}_t^\top {S}_t\preceq (1+\epsilon ){C}_t^\top {C}_t$ with probability $1-\delta$, provided $\tau =\Omega (\frac{r+\log (1/\delta )}{\epsilon ^{2}})$. Using the union bound, we have if $\tau =\Omega (\frac{r+\log (T/\delta )}{\epsilon ^{2}})$, with probability $1-\delta$, simultaneously for all $t=1,\ldots ,T$,

$$\begin{aligned} (1-\epsilon ){C}_t^\top {C}_t\preceq {S}_t^\top {S}_t\preceq (1+\epsilon ){C}_t^\top {C}_t. \end{aligned}$$

1.5 Proof of Corollary 2

Define the SVD of ${C}_t^\top$ as ${C}_t^\top = {U}{\Sigma }{V}^\top$ where ${U}\in {\mathbb {R}}^{d\times d},{\Sigma }\in {\mathbb {R}}^{d\times d},{V}\in {\mathbb {R}}^{t\times d}$. Then we have ${K}_t = {U}(\alpha {I}_d+{\Sigma }{\Sigma }^\top ){U}^\top$ and

$$\begin{aligned} \tilde{{I}}_t&={K}_t^{-1/2}\tilde{{K}}_t{K}_t^{-1/2}={K}_t^{-1/2}(\alpha {I}_d+{C}_t^\top {R}_t^\top {R}_t{C}_t){K}_t^{-1/2}\\&={U}\Big ((\alpha {I}_p+{\Sigma }{\Sigma })^{-1/2}{\Sigma }{V}^\top {R}_t^\top {R}_t{V}{\Sigma }(\alpha {I}_d+{\Sigma }{\Sigma }^\top )^{-1/2}\\&\quad +\alpha {I}_d(\alpha {I}_d+{\Sigma }{\Sigma })^{-1}\Big ){U}^\top \\&={U}\Big ((\alpha {I}_p+{\Sigma }{\Sigma })^{-1/2}{\Sigma }{R}{R}^\top {\Sigma }(\alpha {I}_d+{\Sigma }{\Sigma }^\top )^{-1/2}\\&\quad +\alpha {I}_d(\alpha {I}_d+{\Sigma }{\Sigma })^{-1}\Big ){U}^\top \end{aligned}$$

where ${R}={V}^\top {R}_t^\top \in {\mathbb {R}}^{d\times \tau }$ is a Gaussian random matrix due to that V is an orthogonal matrix and ${R}_t^\top$ is a Gaussian random matrix. Let $c_i^2=\frac{\alpha }{\alpha +{\sigma ^2_{ti}}}$ and $s_i^2=\frac{{\sigma ^2_{ti}}}{\alpha +{\sigma ^2_{ti}}}$. Then according to Theorem 4, with probability at least $1-\delta$,

$$\begin{aligned} (1-\epsilon ){I}_d\preceq \tilde{{I}}_t\preceq (1+\epsilon ){I}_d \end{aligned}$$

provided $\tau \ge \frac{{\tilde{r}}_t{\sigma ^2_{t1}}}{c\epsilon ^2(\alpha +{\sigma ^2_{t1}})}\log \frac{2d}{\delta }$ where the constant c is at least 1 / 32. Using the union bound, we complete the proof.