Randomized Kaczmarz algorithm with averaging and block projection

Abstract

The randomized Kaczmarz algorithm is a simple iterative method for solving linear systems of equations. This study proposes a variant of the randomized Kaczmarz algorithm by combining block projection and weighted averaging techniques. Here, block projection quickly decreases iterative errors, and averaging reduces randomness and enables parallel computation simultaneously. Their combination can balance the convergence rate, convergence horizon, and computational complexity. In addition, three adaptive weights are designed to balance multiple block calculations and accelerate the proposed method. Exponential convergence is established for general linear systems (overdetermined or underdetermined, full-rank or deficient-rank, and consistent or inconsistent). Numerical simulations explain and verify the results.

Appendices

Appendices

Proof of Lemma 3.1

For a real number sequence $\{a_i\}$, we can verify that

$$\begin{aligned} \sum _{t=n}^k \Big ( \prod _{i=t+1}^k (1-a_i)\Big ) a_t = 1- \prod _{i=n}^k (1-a_i)\quad (k\ge n). \end{aligned}$$

(7.1)

As k goes to infinity, the left term in (7.1) converges to one.

Denote $\varPhi (k+1,t)= (1-a_k) \varPhi (k,t)$, $\varPhi _{t,t}=1$, $\forall k\ge t$. Using (7.1), we have

$$\begin{aligned} x_{k+1}&= \varPhi (k+1,0) x_0 + \sum _{t=0}^k \varPhi (k+1,t+1) \frac{a_tb_t}{a_t}\nonumber \\&=\varPhi (k+1,0) x_0 + \sum _{t=0}^k \Big (\prod _{i=t+1}^k (1-a_i)\Big ) \frac{a_tb_t}{a_t}. \end{aligned}$$

(7.2)

It is easy to see that $\lim _{k\rightarrow \infty }\varPhi (k,t)=0$, $\forall t$. Applying the Toeplitz Lemma [16] to the second term on the right-hand side of (7.2), we obtain that $x_{k}$ converges to $\lim _{k\rightarrow \infty } \frac{b_k}{a_k}$ if the limit exists. $\square$

Proof of Theorem 3.5

From (3.5), we have

$$\begin{aligned} {\textbf{x}}_k-{\textbf{x}}_*=\left( {\textbf{I}}-\alpha \sum _{j=1}^\tau \omega _j {\textbf{X}}_j\right) \left( {\textbf{x}}_{k-1}-{\textbf{x}}_*\right) +\alpha \sum _{j=1}^\tau \omega _j \left( {\textbf{A}}^j\right) ^T{\textbf{Z}}_j\theta ^j\nonumber . \end{aligned}$$

The disturbed term is

$$\begin{aligned} \sum _{j=1}^\tau \omega _j ({\textbf{A}}^j)^T{\textbf{Z}}_j\theta ^j =&\begin{pmatrix} \omega _1({\textbf{A}}^1)^T{\textbf{Z}}_1,\cdots ,\omega _\tau ({\textbf{A}}^\tau )^T{\textbf{Z}}_\tau \end{pmatrix} \begin{pmatrix} (\theta ^1)^T,\cdots ,(\theta ^\tau )^T \end{pmatrix}^T\\ =&\begin{pmatrix} \omega _1({\textbf{A}}^1)^T{\textbf{Z}}_1,\cdots ,\omega _\tau ({\textbf{A}}^\tau )^T{\textbf{Z}}_\tau \end{pmatrix} \theta . \end{aligned}$$

$\square$

Note that

$$\begin{aligned} {\mathbb {E}}[{\textbf{Z}}_j]=&\sum _{i=1}^{l_j}p_{j,i} {\textbf{I}}_{\{i\}}^T ({\textbf{A}}^j_{\{i\}}({\textbf{A}}^j_{\{i\}})^T{)^{-1} }{\textbf{I}}_{\{i\}} =\sum _{i=1}^{l_j}\frac{\Vert {\textbf{A}}^j_{\{i\}}\Vert ^2}{\Vert {\textbf{A}}^j\Vert _F^2} \frac{{\textbf{I}}_{\{i\}}^T{\textbf{I}}_{\{i\}}}{\Vert {\textbf{A}}^j_{\{i\}}\Vert ^2} =\frac{{\textbf{I}}}{\Vert {\textbf{A}}^j\Vert _F^2}, \end{aligned}$$

where one should notice that ${\textbf{A}}^j_{\{i\}}({\textbf{A}}^j_{\{i\}})^T$ is a nonzero number because ${\textbf{A}}^j_{\{i\}}$ is a nonzero row vector.

Taking expectation on the disturbed term, we have

$$\begin{aligned}&{\mathbb {E}}\left[ \sum _{j=1}^\tau \omega _j \left( {\textbf{A}}^j\right) ^T{\textbf{Z}}_j\theta ^j\right] \\&\quad =\begin{pmatrix} \omega _1\left( {\textbf{A}}^1\right) ^T{\mathbb {E}}\left[ {\textbf{Z}}_1\right] ,\omega _2 \left( {\textbf{A}}^2\right) ^T{\mathbb {E}}\left[ {\textbf{Z}}_2\right] ,\ldots ,\omega _\tau \left( {\textbf{A}}^\tau \right) ^T{\mathbb {E}}\left[ {\textbf{Z}}_\tau \right] \end{pmatrix}\theta \\&\quad =\Vert {\textbf{A}}\Vert _F^{-2} \begin{pmatrix} \left( {\textbf{A}}^1\right) ^T, \left( {\textbf{A}}^2\right) ^T,\ldots , \left( {\textbf{A}}^\tau \right) ^T \end{pmatrix}\theta =\Vert {\textbf{A}}\Vert _F^{-2} {\textbf{A}}^T\theta ={\textbf{0}}. \end{aligned}$$

Finally, note that ${\textbf{X}}_j=({\textbf{A}}^j)^T{\textbf{Z}}_j{\textbf{A}}^j$, $\omega _j=\Vert {\textbf{A}}^j\Vert _F ^2/\Vert {\textbf{A}}\Vert _F ^2$, we have

$$\begin{aligned} \begin{aligned} {\mathbb {E}}[{{\textbf {x}}}_k-{{\textbf {x}}}_*] =&\left( {{\textbf {I}}}-\alpha \sum _{j=1}^\tau \frac{\Vert {{\textbf {A}}}^j\Vert _F^2 }{\Vert {{\textbf {A}}}\Vert _F^2} {\mathbb {E}}[{{\textbf {X}}}_j]\right) {\mathbb {E}}[{{\textbf {x}}}_{k-1}-{{\textbf {x}}}_*]\\ =&\left( {{\textbf {I}}}-\frac{\alpha }{\Vert {{\textbf {A}}}\Vert _F^2} {{\textbf {A}}}^T{{\textbf {A}}}\right) {\mathbb {E}}[{{\textbf {x}}}_{k-1}-{{\textbf {x}}}_*]. \end{aligned} \end{aligned}$$

Then, if $\alpha \le 1$, we obtain $\Vert {\mathbb {E}}[{{\textbf {x}}}_k-{{\textbf {x}}}_*]\Vert \le (1-\frac{\alpha \sigma _r({{\textbf {A}}}^T{{\textbf {A}}})}{\Vert {{\textbf {A}}}\Vert _F^2})\Vert {\mathbb {E}}[{{\textbf {x}}}_{k-1}-{{\textbf {x}}}_*]\Vert$.

Proof of Lemma 4.1

Denote

$$\begin{aligned} V&=\frac{\sum _{i=1}^\tau a_i^{q+2}}{\sum _{j=1}^\tau a_j^q} -\left( \frac{\sum _{i=1}^\tau a_i^{q+1}}{\sum _{j=1}^\tau a_j^q}\right) ^2\\&= \sum _{i=1}^\tau \frac{ a_i^{q}}{\sum _{j=1}^\tau a_j^q}a_i^2 -\big (\sum _{i=1}^\tau \frac{ a_i^{q}}{\sum _{j=1}^\tau a_j^q}a_i\big )^2\ge 0. \end{aligned}$$

Then, we obtain

$$\begin{aligned} u(q+1)&= \sum _{i=1}^\tau \frac{a_i^{q+1}}{\sum _{j=1}^\tau a_j^{q+1}} a_i = \frac{\sum _{j=1}^\tau a_j^{q+2}}{\sum _{j=1}^\tau a_j^{q+1}}\\&= \frac{\frac{\sum _{j=1}^\tau a_j^{q+2}}{\sum _{j=1}^\tau a_j^{q}} \sum _{j=1}^\tau a_j^{q} -\left( \frac{\sum _{j=1}^\tau a_j^{q+1}}{\sum _{j=1}^\tau a_j^{q}}\right) ^2\sum _{j=1}^\tau a_j^{q}+\left( \frac{\sum _{j=1}^\tau a_j^{q+1}}{\sum _{j=1}^\tau a_j^{q}}\right) ^2\sum _{j=1}^\tau a_j^{q}}{\sum _{j=1}^\tau a_j^{q+1}}\\&= \frac{\left( \frac{\sum _{j=1}^\tau a_j^{q+2}}{\sum _{j=1}^\tau a_j^{q}} -\left( \frac{\sum _{j=1}^\tau a_j^{q+1}}{\sum _{j=1}^\tau a_j^{q}}\right) ^2\right) \sum _{j=1}^\tau a_j^{q}+\left( \frac{\sum _{j=1}^\tau a_j^{q+1}}{\sum _{j=1}^\tau a_j^{q}}\right) ^2\sum _{j=1}^\tau a_j^{q}}{\sum _{j=1}^\tau a_j^{q+1}}\\&=\frac{V\sum _{j=1}^\tau a_j^{q}+\left( \sum _{j=1}^\tau a_j^{q+1}\right) ^2/\left( \sum _{j=1}^\tau a_j^{q}\right) }{\sum _{j=1}^\tau a_j^{q+1}}\\&=\frac{V\left( \sum _{j=1}^\tau a_j^{q}\right) ^2+\left( \sum _{j=1}^\tau a_j^{q+1}\right) ^2}{\left( \sum _{j=1}^\tau a_j^{q+1}\right) ^2}\frac{\sum _{j=1}^\tau a_j^{q+1}}{\sum _{j=1}^\tau a_j^{q}}\\&=\left( \frac{V\left( \sum _{j=1}^\tau a_j^{q}\right) ^2}{\left( \sum _{j=1}^\tau a_j^{q+1}\right) ^2} +1\right) \frac{\sum _{j=1}^\tau a_j^{q+1}}{\sum _{j=1}^\tau a_j^{q}}\\&\ge \frac{\sum _{j=1}^\tau a_j^{q+1}}{\sum _{j=1}^\tau a_j^{q}} =\sum _{i=1}^\tau \frac{a_i^{q}}{\sum _{j=1}^\tau a_j^{q}}a_i =u(q). \end{aligned}$$

$\square$