Novel Sparse Algorithms Based on Lyapunov Stability for Adaptive System Identification

Adaptive filters are extensively used in the identification of an unknown system. Unlike several gradientsearch based adaptive filtering techniques, the Lyapunov Theory-based Adaptive Filter offers improved convergence and stability. When the system is described by a sparse model, the performance of Lyapunov Adaptive (LA) filter is degraded since it fails to exploit the system sparsity. In this paper, the Zero-Attracting Lyapunov Adaptation algorithm (ZA-LA), the Reweighted Zero-Attracting Lyapunov Adaptation algorithm (RZA-LA) and an affine combination scheme of the LA and proposed ZA-LA filters are proposed. The ZA-LA algorithm is based on l1-norm relaxation while the RZA-LA algorithm uses a log-sum penalty to accelerate convergence when identifying sparse systems. It is shown by simulations that the proposed algorithms can achieve better convergence than the existing LMS/LA filter for a sparse system, while the affine combination scheme is robust in identifying systems with variable sparsity.


Introduction
Adaptive filtering algorithms play a significant role in system identification applications, e.g., channel estimation and echo cancellation [1].Ideally, the adaptive filter with high convergence rate, stability, good tracking capability and robustness to random noise is desirable for many applications.The widely used optimization technique for optimal filter design is the gradient descent method.Least Mean Square (LMS) algorithm developed by Widrow and Hoff [2] is the most widely used adaptive algorithm due to its computational simplicity and proven robustness.Lyapunov Adaptive Filtering (LA) algorithms in the sense of the Lyapunov stability theory were proposed in [3][4][5][6] to overcome the problems faced by gradient descent-based techniques such as slow rate of convergence, sensitivity to variations in the eigenvalue spread and local minima problem.Moreover, the LA algorithm is independent of the stochastic properties of the input signal and additive noise.However, the LA algorithm suffers from poor convergence performance when the underlying system to be identified is sparse such as network and acoustic echo path [7], digital TV transmission channel [8], and underwater channel [9].In general, the sparse FIR system is characterized by its impulse response which consists of very few active coefficients among many inactive ones [10][11][12].
Conventional adaptive algorithms neglect the sparse information which is present in the system that leads to degrade their performance when estimating the sparse channels.Recent studies on system identification specify that by utilizing the a priori knowledge about the system sparsity, the estimation performance can be improved substantially.This motivated the researchers towards developing sparse adaptive filtering algorithms in the last few years.Inspired by the theory of Compressed Sensing (CS) [13], [14] and LASSO [15], ℓ 1 -norm relaxation is utilized to exploit system sparsity.In [16] sparse adaptive filtering approach based on LMS algorithm is developed by incorporating ℓ 1 -norm penalty into its cost function which is termed as Zero-Attracting LMS (ZA-LMS).This approach is easy to implement and performs well when the system is highly sparse, but it fails as the system sparsity decreases.Based on reweighted ℓ 1 -minimization sparse recovery algorithm [17], an improved version of ZA-LMS i.e.Reweighted ZA-LMS (RZA-LMS) is proposed in [16].The RZA-LMS algorithm performs better in less sparse conditions but at the cost of increased complexity.Following these ideas, in this paper, we propose two sparse Lyapunov Adaptation algorithms namely, Zero-Attracting Lyapunov Adaptation algorithm (ZA-LA) and Reweighted Zero-Attracting Lyapunov Adaptation algorithm (RZA-LA).
Over the past decade or so, a combination of adaptive filters has proven to be an efficient way to handle systems with variable sparsity.In [18], [19], an adaptive convex combination of two LMS filters with different parameter setting is proposed to alleviate the speed of convergence vs the residual error trade-off.In [20], a mixture approach of adaptively combining LMS and ZA-LMS algorithm using a convex combination has been proposed to achieve robustness against time-varying system sparsity.This approach is extended [21] to colored (correlated) input signal using a convex combination of the Affine Projection Algorithm (APA) and Zero Attracting Affine Projection Algorithm (ZA-APA), and in [22] steady state mean square analysis of convex combination in the context of acoustic echo cancellation is performed.In [23], a convex combination approach using another variant of sparse adaptive filtering, i.e.Improved Proportionate Normalized LMS (IPNLMS) algorithm is proposed.In all these papers, the authors have used the convex combination to combine effectively the outputs of the individual adaptive filters.The affine combination as a generalization of the convex combination is studied in [24], and in [25] affine combination analysis was extended for colored inputs and nonstationary environments.In [26], transient analysis for the affine combination of two NLMS adaptive filters is studied.In [27], [28], it is demonstrated that affine combination results in faster convergence than the convex combination of two adaptive filters.So, in this paper, we use affine combination scheme to combine LA and ZA-LA filters to handle systems with variable sparsity.To the best of our knowledge, no paper has reported on system identification using Lyapunov adaptive filter algorithm with sparsity constraints to identify an unknown sparse system.
The rest of the paper is organized as follows.Section 2 describes the system model and reviews the LA algorithms.In Sec. 3, we propose two adaptive sparse LA algorithms.In Sec. 4 we compare the algorithms in terms of computational complexity.Section 5 highlights the affine combination scheme and its stability is analyzed in Sec. 6. Simulation results of the proposed LA algorithms are presented in Sec. 7. Finally, we conclude the paper in Sec. 8.

Review of LA Algorithms
The basic idea of sparse system identification is to improve the filtering/estimation performance by utilizing the inherent sparse structure information.The block diagram of the sparse system identification is shown in Fig. 1.We consider an N length FIR filter coefficient vector w 0 = [w 0 , w 1 , …, w N -1 ] T and the input signal vector x ̅ (n) = [x(n), x(n -1), …, x(n -N + 1)] T .The input x ̅ (n) is applied to both the adaptive filter and the unknown sparse system.The desired signal d(n) that is generally corrupted by the observation noise v(n) is given by d The output estimate y(n) of the adaptive filter W ̅ (n) is subtracted from the reference signal d(n) to produce an error signal e(n) [29].The error signal e(n) is then used by the adaptive algorithm iteratively to manipulate the filter coefficients such that the error is minimized.
The cost function V(n) of the LA filter is defined as the square of error between the desired reference input and filtered output which is considered as the Lyapunov function [3], [4], where The Lyapunov adaptive filter weight update law is properly chosen such that ΔV(k) = V(k) -V(k -1) is negative definite at each iteration.According to Lyapunov stability theory [30], when ΔV(k) < 0, the output of the adaptive filter can asymptotically converge to the desired signal which means that the error e(n)0.
For the given desired signal d(n) and filter output y(n) = W ̅ T (n) x ̅ (n) the LA weight update rule is as follows: and where g(n) is the adaptation gain and α(n) is a priori estimation error defined as 0  < 1 and  1 ,  2 are small positive constants to prevent the singularities in the adaptation gain.The parameter  controls the convergence rate of the algorithm.
To improve the LA filter tracking performance and robustness, the Lyapunov function is redefined as  n e 2 (n) and the adaptation gain g(n) of the Lyapunov adaptive algorithm is modified [5], [6] as where  > 1 is a constant parameter.
To achieve faster convergence rate and lower steady state error performance in the case of noisy environments, a step size parameter, μ is included in the adaptation gain which is given by [31] The adaptation gain g(n) for the Lyapunov adaptive algorithm can also be defined as where the adaptive adaptation gain rate is adjustable in order to improve the tracking capability of the algorithm [32][33][34] ) ( 3. Proposed Sparse LA Algorithms

Zero-Attracting Lyapunov Adaptation Algorithm (ZA-LA)
The cost function V 1 (n) of ZA-LA is defined by combining the Lyapunov function of LA filter with the ℓ 1 -norm penalty of the weight vector where γ ZA > 0 denotes a regularization parameter which balances the error term and system sparsity.
Using the adaptation gain g(n) given in (6), we obtain The ZA-LA filter update rule is given by ZA where sgn(•) is the sign function.The ZA-LA algorithm complexity is slightly higher than that of LA algorithm due to the third term of (13).

Reweighted Zero-Attracting Lyapunov Adaptation Algorithm (RZA-LA)
The RZA-LA cost function is represented by where γ RZA > 0 is a regularization parameter which balances the estimation error term and sparsity of W ̅ (n).
According to the log-det heuristic approach used in [35], [36], the zero attractor term in (14) yields to a convex optimization problem.To establish this connection, consider the problem Iterative linearization of this objective function gives If w i (n) is small, its weighting factor in the next minimization step, (1 + ε│w i (n)│) -1 , is large.So the small entries in W ̅ (n) are pushed towards zero.Thus, the log-sum penalty function has the potential to be much more sparsity encouraging than the ℓ 1 -norm.
The RZA-LA filter update is defined as The RZA-LA algorithm complexity is slightly higher than that of LA algorithm due to the third term of (17).

Computational Complexity
The numerical complexity in terms of additions, multiplications, and divisions of the proposed sparse adaptive algorithms and those of the LMS algorithm and its sparse variants are shown in Tab. 1.

Algorithms Addition Multiplication Division
Tab. 1.Comparison of computational complexity of the investigated algorithms.
It can be seen from Tab. 1, that the proposed sparse algorithms have a moderate computational complexity increase when compared with the original algorithms.

Affine Combination of LA and ZA-LA Algorithms
In order to handle the system with varying level of sparseness, we have proposed to combine LA and ZA-LA algorithm using an affine combination approach.The configuration of the proposed affine combination scheme is shown in Fig. 2 in which Filter 1 is updated using LA algorithm (3), and Filter 2 is updated using ZA-LA algorithm as given in (13), respectively.We will hereafter call this filter as the Affine Combined Lyapunov Adaptation (ACLA) filter.
The output signal of ACLA filter is given by where (n) is the mixing parameter and can be any real number, and y 1 (n) and y 2 (n) denotes the output of the individual filters, i.e.
The a priori error signal e a (n) is obtained by subtracting the output signal of ACLA filter from the output signal of the unknown system, where To find the mixing parameter λ(n), the derivative of mean square of the a priori error with respect to λ(n) is evaluated and equated to zero.
where the unknown system output, y 0 (n), is replaced by d(n), which is a valid assumption.
The mathematical expectation in the numerator and the denominator of ( 22) are replaced by exponential averaging where p u (n) is the averaged quantity, u(n) is the signal to be averaged, and γ = 0.01.These results obtained for both the numerator and denominator are substituted in ( 22) to obtain λ(n).

Stability Analysis
Let us first provide the convergence analysis of LA filter.When the adaptive filter coefficient vector W ̅ (n) is updated by (3) and ( 5), the Lyapunov function is chosen as According to Lyapunov stability theory [30], the tracking error e(n) will asymptotically converge to zero.

T ( ) ( ) ( ) ( ) e n d n W n x n
and the overall system error is given by where and y(n) is the ACLA filter output as given in (18).
Equation ( 18) can be written as where The equivalent weight vector W ̅ c of the combined filter can be expressed as c 1 2 Using ( 30) and ( 31) in ( 29), we get where The Mean Square Deviation (MSD) of the ACLA filter at time n is Substituting ( 22) in (36) yields The first term of (37) corresponds to the MSD of the second adaptive filter, MSD 2 (n) and since the MSD is a positive quantity, it indicates that MSD c (n) is smaller than MSD 2 (n).
Equation (33) can also be expressed as Now by inserting (38) in the first line of (36), it can be shown that where The first term of (39) corresponds to the MSD of the first adaptive filter, MSD 1 (n) and since the MSD is a positive quantity, it indicates that MSD c (n) is smaller than MSD 1 (n).Thus, the combined filter performs at least as well as the best component filter or better than any of them, for every n and the stability is guaranteed.

Simulation Results
This section shows the simulations that are carried out to evaluate the performance of our proposed sparse algorithms.The length of unknown system w 0 is set to N = 16, and its impulse response is assumed to have only one nonzero value at the tap index 5 and zeroes elsewhere making the system highly sparse as shown in Fig. 3.
The input signal x ̅ (n) is considered in two ways: Case 1: Gaussian random signal with zero mean and unit variance, N(0,1) and Case 2: Correlated/Colored signal generated by passing a white Gaussian, u(n) through a first-order autoregressive process, AR(1) with a pole 0.8 that is represented as The output of the system is corrupted by an independent white Gaussian noise with variance 0.001.
The performance metrics used to evaluate the proposed algorithms are Mean Square Deviation (MSD), which is defined as where E{•} denotes expectation operator, and w ̅ 0 and W ̅ (n) are the true FIR filter vector and its adaptive estimator, respectively, and the Mean Square Error (MSE) which is given as The average of 200 trials is used in evaluating the results.
From the simulation results shown in Fig. 4(a), for the Gaussian input and when the system is highly sparse, it is observed that our proposed Sparse LA filters (ZA-LA & RZA-LA) converge faster than the existing LMS and LA algorithms which cannot exploit the sparseness information present in the system.In Fig. 4(b), for the case of colored input, it is observed that the proposed LA algorithms have converged while the LMS algorithms fail to converge.
The mean square error (MSE) performance of the proposed LA algorithms is depicted in Fig. 5.It is observed that the MSE of proposed LA algorithms is lower than the LMS algorithm and their sparse counterpart.Next, we have considered that the unknown system has all its tap coefficients set to non-zero values, i.e. a nonsparse system.Figure 6 shows the impulse response of the used non-sparse system.
With the Gaussian input, the proposed ZA-LA algorithm converges faster than ZA-LMS algorithm, but it exhibits high steady state error when the system is nonsparse as shown in Fig. 7(a).The convergence of RZA-LA algorithm is the same as that of the LA algorithm and much better than the LMS algorithm.In the case of colored input, we observe from Fig. 7(b) that the performance of our proposed algorithms is superior to that of LMS algorithm and their sparse counterpart.
The mean square error (MSE) of proposed LA algorithms is shown in Fig. 8.The RZA-LA algorithm achieves lower MSE value than that of the ZA-LA algorithm which is lower than that of the LMS algorithm and its sparse counterpart.
The performance of the ACLA filter is analyzed for identifying the system of length N = 16 with variable sparsity.Initially, the system is assumed to be a highly sparse system with impulse response as shown in Fig. 3.At the 600 th sample, the system is abruptly changed to a nonsparse system with impulse response as shown in Fig. 6.The learning curves of ACLA filter are shown in Fig. 9 and Fig. 10 for white and colored input cases, respectively.From Fig. 9, it can be seen clearly that for white input case, the ACLA filter achieves faster convergence and better steady state behavior.When the system is highly sparse (before the 600 th sample), the ACLA filter attains the lower steady state value of ZA-LA filter, and when the system is changed to a non-sparse system (at the 600 th sample), it achieves the steady state value of LA filter.When the input is considered as colored input, the steady state value of the ACLA filter is slightly better than that of the independent filters for a highly sparse system and when the system is changed to a non-sparse system, the ACLA filter behaves like LA filter that achieves the lower steady state value as shown in Fig. 10.Thus, the proposed ACLA filter is robust in identifying the systems with variable sparsity.Figure 11 shows the evolution of mixing parameter, λ(n) for the ACLA filter.
The behavior of the proposed algorithms with input as a real speech signal sampled at 8 kHz is evaluated in the next simulation.The system is considered to have a varying degree of sparsity.Initially, the system is assumed to be sparse with impulse response shown in Fig. 3, and at time t = 1.75 sec, it is changed to the non-sparse system as shown in Fig. 6.The simulations are averaged over 100 trials.The Echo Return Loss Enhancement (ERLE) is used as the performance metric and is defined as [23], [37]   It is observed from Fig. 12 that the proposed RZA-LA filter achieves higher ERLE values and the ERLE of ZA-LA filter is better than that of the LA filter when the sys-tem is sparse.At t = 1.75 sec, when the system impulse response is switched to non-sparse, the ERLE performance of ZA-LA is reduced while the RZA-LA filter still attains the higher ERLE value closer to the LA filter which performs well under non-sparse conditions.Hence, the robustness of the proposed algorithms is verified.

Conclusion
The standard Lyapunov Adaptation algorithm does not exploit sparsity present in an unknown sparse system.In this paper, we proposed two novel algorithms, namely, Zero-Attracting Lyapunov Adaptation algorithm (ZA-LA) and Reweighted Zero-Attracting Lyapunov Adaptation algorithm (RZA-LA) to improve adaptive sparse system identification performance.From the simulation results, the effectiveness of the proposed algorithms is verified for both white input and colored input case in terms of MSD and MSE.
Also, an Affine Combined Lyapunov Adaptation (ACLA) filter is presented to identify the systems with variable sparsity.The proposed combination filter exhibits robustness and achieves lower steady state value irrespective of the level of sparseness for the unknown system.The added complexity of the proposed algorithms is worth considering due to the increased performance.

Fig. 4 .Fig. 5 .Fig. 6 .
Fig. 4. MSD comparison of the proposed sparse LA algorithms with existing adaptive algorithms for highly sparse system with (a) white input, (b) colored input.