An efficient normalized LMS algorithm

Abstract

The task of adaptive estimation in the presence of random and highly nonlinear environment such as wireless channel estimation and identification of non-stationary system etc. has been always challenging. The least mean square (LMS) algorithm is the most popular algorithm for adaptive estimation and it belongs to the gradient family, thus inheriting their low computational complexity and their slow convergence. To deal with this issue, an efficient normalization of the LMS algorithm is proposed in this work which is achieved by normalizing the input signal with an intelligent mixture of weighted signal and error powers which results in a variable step-size type algorithm. The proposed normalization scheme can provide both significant faster convergence in initial adaptation phase while maintaining a lower steady-state mean-square-error compared to the conventional normalized LMS (NLMS) algorithm. The proposed algorithm is tested on adaptive denoising of signals, estimation of unknown channel, and tracking of random walk channel and its performance is compared with that of the standard LMS and NLMS algorithms. Mean and mean-square performance of the proposed algorithm is investigated in both stationary and non-stationary environments. We derive the closed-form expressions of various performance measures by evaluating multi-dimensional moments. This is done by statistical characterization of required random variables by employing the approach of Indefinite Quadratic Forms. Simulation and experimental results are presented to corroborate our theoretical claims.

Appendices

Appendix A: Evaluation of integral (40)

In this appendix, we evaluate the integral in (40). By knowing the fact that $\mathbf{u} \sim {\mathcal {C}}{\mathcal {N}}~(\mathbf{0},\varvec{\varLambda })$, the pdf $p(\mathbf{u})$ is given by

$$\begin{aligned} p(\mathbf{u})=\frac{1}{\pi ^{2M} |\varvec{\varLambda }|}e^{-\mathbf{u}\varvec{\varLambda }^{-1}{} \mathbf{u}^H}, \end{aligned}$$

(73)

We substitute the expression for $p(\mathbf{u})$ in (40). In addition, we employ the following integral representation of the unit step function [20]

$$\begin{aligned} {\tilde{u}}(x)=\frac{1}{2\pi }\int _{-\infty }^{\infty } \frac{e^{x(j\omega +\beta )}}{(j\omega +\beta )} d\omega . \end{aligned}$$

(74)

As a result, the CDF of $Y_{k{\bar{k}}}$ will take the form

$$\begin{aligned}&F_{Y_{k{\bar{k}}}}(y)=\frac{1}{2\pi ^{2M+1}|\varvec{\varLambda }|}\int _{-\infty }^{\infty } e^{-\mathbf{u}\varvec{\varLambda }^{-1}{} \mathbf{u}^H} \int _{-\infty }^{\infty }\nonumber \\&\quad \frac{e^{(\epsilon y+ y ||\mathbf{u}||_{\mathbf{D}}^2-||\mathbf{u}||_{\mathbf{D}_{k{\bar{k}}}}^2)(j\omega +\beta )}}{(j\omega +\beta )}~d\omega ~d\mathbf{u}. \end{aligned}$$

(75)

By collecting similar terms, the above CDF can be set up as

$$\begin{aligned} F_{Y_{k{\bar{k}}}}(y)= & {} \frac{1}{2\pi ^{2M+1}|\varvec{\varLambda }|}\nonumber \\&\int _{-\infty }^{\infty }\int _{-\infty }^{\infty } e^{-\mathbf{u}\big (\varvec{\varLambda }^{-1}-(y\mathbf{D}-\mathbf{D}_{k{\bar{k}}})(j\omega +\beta )\big )\mathbf{u}^H}\nonumber \\&\quad \times d\mathbf{u} ~\frac{e^{\epsilon y(j\omega +\beta )}}{(j\omega +\beta )}~d\omega ,\nonumber \\ \end{aligned}$$

(76)

In the above equation, the inner integral is nothing but the Gaussian integral (details can be found in [20, 27] for a formal proof) whose solution can be shown to be

$$\begin{aligned}&\frac{1}{\pi ^{2M}}\int e^{-\mathbf{u}\big (\varvec{\varLambda }^{-1}-(y\mathbf{D}-\mathbf{D}_{k{\bar{k}}})(j\omega +\beta )\big )\mathbf{u}^H} d\mathbf{u}\nonumber \\&\quad =\frac{1}{\Big |\varvec{\varLambda }^{-1}-(y\mathbf{D}-\mathbf{D}_{k{\bar{k}}})(j\omega +\beta )\Big |}, \end{aligned}$$

(77)

As a result, the CDF of $Y_{k{\bar{k}}}$ is reduced to the following one-dimensional complex integral

$$\begin{aligned}&F_{Y_{k{\bar{k}}}}(y)=\frac{1}{2\pi |\varvec{\varLambda }|}\int _{-\infty }^{\infty }\nonumber \\&\quad \frac{e^{\epsilon y(j\omega +\beta )}}{\Big |\varvec{\varLambda }^{-1}-(y\mathbf{D}-\mathbf{D}_{k{\bar{k}}})(j\omega +\beta )\Big |(j\omega +\beta )}~d\omega . \end{aligned}$$

(78)

The above integral can be solved using the approach of [20] which will finally result in (60).

Appendix B: Evaluation of integral (50)

In this appendix, we provide the evaluation of integral (50). First, we employ the variable transformation $t=\frac{y}{(1-y)}$ in (50) which results in

$$\begin{aligned} E[Y_k]=\int _0^{\infty }\frac{\lambda _k^{2M}e^{\frac{-\epsilon t}{\lambda _k(1-\alpha )}}}{|\varvec{\varLambda }|(1+t)^2 \Pi _{m=1,m\ne k}^{2M}\left( \rho _{km}+\frac{td_m}{(1-\alpha )}\right) }dy \end{aligned}$$

(79)

Now, expanding the above fraction using partial fraction to get

$$\begin{aligned}&\frac{\lambda _k^{2M}}{|\varvec{\varLambda }|(1+t)^2 \Pi _{m=1,m\ne k}^{2M}\left( \rho _{km}+\frac{td_m}{(1-\alpha )}\right) }= \end{aligned}$$

(80)

$$\begin{aligned}&\frac{a_1}{(1+t)}+\frac{a_2}{(1+t)^2}+\sum _{m=1,m\ne k}^{2M}\frac{{\tilde{a}}_m}{\left( \rho _{km}+\frac{td_m}{(1-\alpha )}\right) }. \end{aligned}$$

(81)

where the constants $a_1$, $a_2$, and ${\tilde{a}}_m$ are found to be

$$\begin{aligned} a_1= & {} -\frac{a_2\left[ \frac{d}{dv}\prod _{m=1,m \ne k}^{2M}\left( \frac{\rho _{km}(1-\alpha )}{d_m}+v\right) \right] _{v=-1}\lambda _k^{2M}(1-\alpha )^{2M-1}}{|\varvec{\varLambda }| \prod _{l=1, l \ne k}^{2M}(d_l)\prod _{m=1,m \ne k}^{2M}\left( \frac{\rho _{km}(1-\alpha )}{d_m}-1\right) }\end{aligned}$$

(82)

$$\begin{aligned} a_2= & {} \frac{\lambda _k^{2M}(1-\alpha )^{2M-1}}{|\varvec{\varLambda }| \prod _{l=1, l \ne k}^{2M}(d_l)\prod _{m=1,m \ne k}^{2M}\left( \frac{\rho _{km}(1-\alpha )}{d_m}-1\right) }\end{aligned}$$

(83)

$$\begin{aligned} {\tilde{a}}_m= & {} \frac{\lambda _k^{2M}(1-\alpha )^{2M-1}}{|\varvec{\varLambda }| \prod _{l=1, l \ne k}^{2M}(d_l)(1-\frac{\rho _{km}(1-\alpha )}{d_m})^2\prod _{l=1,l \ne k, m}^{2M}\left( \frac{\rho _{kl}(1-\alpha )}{d_l}-\frac{\rho _{km}(1-\alpha )}{d_m}\right) } \end{aligned}$$

(84)

Appendix C: Evaluation of $E[\alpha _{\infty }]$

The aim of this appendix is to evaluate the moment $E[\alpha _i]$ at steady-state (i.e., $i \rightarrow \infty$). By looking the relation (69), it can be easily deduced that the conditional moment

$$\begin{aligned}&E[\alpha _{\infty }|\psi _{\infty }]\nonumber \\&\quad =\text{ erf }(\psi _{\infty })\text{ Pr }(\psi _{\infty }>1)+\psi _{\infty }(1-\text{ Pr }(\psi _{\infty }>1)) \nonumber \\ \end{aligned}$$

(85)

where $\text{ Pr }(\psi _{\infty }>1)$ is the probability that the random variable $\psi _{\infty }$ exceeds unity. To evaluate this probability, consider the recursion (68) at $i \rightarrow \infty$ which gives

$$\begin{aligned} \psi _{\infty }= \frac{\gamma |e_{\infty }|^2}{(1-\nu )}=\frac{\gamma |e_a(\infty )+v_{\infty }|^2}{(1-\nu )} \end{aligned}$$

(86)

The assumption A4 allows us to assume the error quantity $e_a(\infty )$ as complex zero mean Gaussian with variance $\zeta$ which results in $|e_{\infty }|^2$ to be central Chi-square random variable with 2 degree of freedom. Thus, we can evaluate the required probability $\text{ Pr }(\psi _{\infty }>1)$ as follows

$$\begin{aligned} \text{ Pr }(\psi _{\infty }>1)= & {} 1-\text{ Pr }\left( \frac{\gamma |e_{\infty }|^2}{(1-\nu )}<1\right) \nonumber \\= & {} 1-\text{ Pr }\left( |e_{\infty }|^2<\frac{(1-\nu )}{\gamma }\right) \nonumber \\= & {} e^{-\frac{(1-\nu )}{\gamma (\zeta +\sigma _v^2)}} \end{aligned}$$

(87)

Next, to remove the condition on $\psi _{\infty }$ in (85), we evaluate the moments $E[\psi _{\infty }]$ as follows

$$\begin{aligned} E[\psi _{\infty }]= & {} \frac{\gamma }{(1-\nu )}E[|e_{\infty }|^2]=\frac{\gamma }{(1-\nu )}(\zeta +\sigma _v^2) \end{aligned}$$

(88)

Now, the moment $E[\text{ erf }(\psi _{\infty })]$ can be evaluated as

$$\begin{aligned} E[\text{ erf }(\psi _{\infty })]= & {} \int _0^{\infty }\text{ erf }(\psi _{\infty }) f_{\psi }(\psi _{\infty })d\psi \end{aligned}$$

(89)

where $f_{\psi }(\psi _{\infty })$ is the pdf of $\psi$ which can be obtained from the variable transformation using the relation (86) and it is given by

$$\begin{aligned} f_{\psi }(\psi _{\infty })= \frac{(1-\nu )}{\gamma (\zeta +\sigma _v^2)}e^{\frac{\psi _{\infty }(1-\nu )}{\gamma (\zeta +\sigma _v^2)}} \end{aligned}$$

(90)

Hence, the moment $E[\text{ erf }(\psi _{\infty })]$ results in

$$\begin{aligned} E[\text{ erf }(\psi _{\infty })]= & {} \frac{(1-\nu )}{\gamma (\zeta +\sigma _v^2)} \int _0^{\infty }\text{ erf }(\psi _{\infty }) e^{\frac{\psi _{\infty }(1-\nu )}{\gamma (\zeta +\sigma _v^2)}}d\psi \nonumber \\= & {} \left[ 1-\text{ erf }\left( \frac{(1-\nu )}{2\gamma (\zeta +\sigma _v^2)}\right) \right] e^{\frac{(1-\nu )^2}{4\gamma ^2(\zeta +\sigma _v^2)^2}} \nonumber \\ \end{aligned}$$

(91)

where we have used the integral solution 6.282-1 in [22]. Finally, by using $E[\psi _{\infty }]$ and $E[\text{ erf }(\psi _{\infty })]$, the unconditional moment $E[\alpha _{\infty }]$ is given by

$$\begin{aligned} E[\alpha _{\infty }]= & {} \left[ 1-\text{ erf }\left( \frac{(1-\nu )}{2\gamma (\zeta +\sigma _v^2)}\right) \right] \nonumber \\&e^{-\frac{(1-\nu )}{\gamma (\zeta +\sigma _v^2)}\left[ 1-\frac{(1-\nu )}{4\gamma (\zeta +\sigma _v^2)}\right] }\nonumber \\&+\frac{\gamma }{(1-\nu )}(\zeta +\sigma _v^2)\left[ 1-e^{-\frac{(1-\nu )}{\gamma (\zeta +\sigma _v^2)}}\right] \end{aligned}$$

(92)

Appendix D: Evaluation of $E[\alpha ^2_{\infty }]$

In this appendix, we derive the moment $E[\alpha ^2_{\infty }]$. To do so, we start with the following conditional moment

$$\begin{aligned}&E[\alpha _{\infty }^2|\psi _{\infty }, \psi ^2_{\infty }]\nonumber \\&\quad =\text{ erf}^2(\psi _{\infty })\text{ Pr }(\psi _{\infty }>1)+\psi _{\infty }^2(1-\text{ Pr }(\psi _{\infty }>1)) \nonumber \\ \end{aligned}$$

(93)

The probability $\text{ Pr }(\psi _{\infty }>1)$ is derived in (87). Next, to remove the conditions on $\psi _{\infty }$ and $\psi ^2_{\infty }$, we evaluate the moments $E[\text{ erf}^2(\psi _{\infty })]$ and $E[\psi ^2_{\infty }]$. To do so, we used the relation (68) and (86) to obtain

$$\begin{aligned} \psi ^2_{\infty }=\frac{\gamma ^2(1+\nu )}{(1-\nu )(1-\gamma ^2)}|e_{\infty }|^4 \end{aligned}$$

(94)

Thus, it can be shown that

$$\begin{aligned} E[\psi ^2_{\infty }]=\frac{\gamma ^2(1+\nu )}{(1-\nu )(1-\gamma ^2)}(\zeta +\sigma _v^2)^2\Gamma (3) \end{aligned}$$

(95)

where $\Gamma ()$ represents the Gamma function. Next, to evaluate the moment $E[\text{ erf}^2(\psi _{\infty })]$ we use approximation $E[\text{ erf}^2(\psi _{\infty })]\approx (E[\text{ erf }(\psi _{\infty })])^2$. Hence, finally, the unconditional moment

$$\begin{aligned} E[\alpha _{\infty }^2]= & {} \left[ 1-\text{ erf }\left( \frac{(1-\nu )}{2\gamma (\zeta +\sigma _v^2)}\right) \right] ^2e^{-\frac{(1-\nu )}{\gamma (\zeta +\sigma _v^2)}\left[ 1-\frac{(1-\nu )}{2\gamma (\zeta +\sigma _v^2)}\right] }\nonumber \\&+\frac{\gamma ^2(1+\nu )}{(1-\nu )(1-\gamma ^2)}(\zeta +\sigma _v^2)^2\Gamma (3)\left[ 1-e^{-\frac{(1-\nu )}{\gamma (\zeta +\sigma _v^2)}}\right] \nonumber \\ \end{aligned}$$

(96)