On an Indirect Method of Exponential Estimation for a Neural Network Model with Discretely Distributed Delays

The purpose of this research is to develop a method of calculation of exponential decay rate for a neural network model based on differential equations with discretely distributed delays. The method results in a quasipolynomial inequality allowing us to investigate the qualitative behavior of the model depending on parameters. In such way we showed inverse dependency in changes of exponential decay rate and time delay. An example of a two-neuron network with four delays is given and numerical simulations are performed to illustrate the obtained results.


Introduction
One of the most modern applications of differential equations with delay is dealing with modeling artificial neural networks.Such models allow us to investigate convergence of recognition algorithms.This is the most significant feature of such models enabling constant interest to the analysis of their qualitative behavior.
Hopfield [8] constructed a simplified neural network model, in which each neuron is represented by a linear circuit consisting of a resistor and a capacitor, and is connected to other neurons via nonlinear sigmoidal activation functions, called transfer functions.A survey of first works in area of neural network models based on differential equations with delay is presented in [25].
When analyzing publications in field of models of artificial neural networks based on differential equations with delay nowadays we can differ two general approaches.
The first one studies local behavior of such systems with help of comparison with linearized system.Here we would like to mention work [25] applying general technique presented in [19,20] for a two-neuron model including a method based on Rouché's theorem.
Linear stability of the model is investigated by analyzing the associated characteristic transcendental equation.The same method was implemented in [27] for a four-neuron model.
The next very important problem included in qualitative behavior of these models is Hopf bifurcation.Here we again consider work [25] where for the case without self-connection, it was found that the Hopf bifurcation occurs when the sum of the two delays varies and passes a sequence of critical values.Similar results were obtained in [27] and [9].The stability and direction of the Hopf bifurcation were determined by applying the normal form theory and the center manifold theorem.
The second approach is dealt with Lyapunov-Krasovkii functionals.The main advantage of this method is the ability to obtain constructive stability conditions.As a rule these conditions are very flexible because they include parameters of Lyapunov-Krasovskii functionals.That is they admit optimization also.
The general objective of papers of such type is not only stability investigation but development methods for calculation of numerical values of exponential decay rates.Stability conditions and corresponding calculations of decay rates result in solution of linear-matrix inequalities (LMI).It should be mentioned that in the past decade, the LMI approach has gained much attention for its computational tractability and usefulness in many areas, especially in the stability analysis for neural networks [6].
The possibility appeared to consider more complex models such as with discretely distributed time-varying delays [10,26], continuously distributed delays [6], of neutral type [14], with impulsive disturbances [21], without boundedness assumption of the activation function [22].In the work [5] they study delay-dependent exponential passivity for uncertain cellular neural networks with discrete and distributed time-varying delays.
The disadvantage of this direct approach is that the stability criterion even in the simplest case presented in Section 5.1 is performed in terms of LMI, which can be efficiently verified via solving the LMI numerically using, e.g. the Matlab LMI Control Toolbox or Scilab LMITool, but does not allow any analytical results needed for qualitative analysis.So, in spite of its universal character the approach based on Lyapunov-Krasovskii functionals resulting in LMIs does not offer clear answer in theoretical reasoning if we would like to get clear evidences for dependencies of decay rates and model parameters.
The next very important shortcoming of methods resulting in LMIs is that in such a way we get sufficient exponential stability only.It holds even in partial case of system (5.2) for which there were obtained necessary and sufficient conditions for global exponential stability using decomposition method offered in [4].It should be noted that, in contrary to Lyapunov-Krasovskii functionals, condition obtained with indirect method corresponds to the result of [4].Nevertheless, in contrary to decomposition method, the method presented in this paper provides in addition to necessary conditions in this case an ability to construct exponential stability conditions for models with multiple time-varying delays.
One of the most important stability problems dealing with neural networks models is the construction of estimates for exponential convergence in a clear form.
That is why the purpose of this work is to offer a method of obtaining estimates for exponential decays resulting in solution of scalar nonlinear inequality.The offered approach is primarily based on results of [18] concerning exponential estimation of solutions of linear systems, applied for compartmental systems in [16].
Earlier many attempts have been made in order to get exponential estimates for linear systems with delay.A number of studies have found exponential estimates using derivative of Lyapunov-Krasovskii functionals.
Another approach of exponential estimate construction for linear systems and leading to nonlinear equation solution was offered in [11].Although it assumes construction of Lyapunov-Krasovskii functional satisfying to some difference equation.
In the works [12,15,17] such clear estimates are obtained for Lyapunov-Krasovskii functionals satisfying to some difference-differential inequalities.
New results on exponential stability of non-autonomous linear delay differential system using the Bohl-Perron theorem were obtained in [1].
In Section 2 we describe model of neural network with discretely-distributed delays studied in the paper.In Section 4 we present method of exponential estimate construction and demonstrate its application when analysing dependence of exponential decay rate and time delay.In Section 5 we apply Theorem 4.1 for two-neuron model with four delays.Also we compare application of the method offered in the work with Lyapunov-Krasovskii functional method.
Within this paper we use the following notation: • the symbol i = m, n for some integer i, m, n , m < n means i = m, m + 1, . . ., n; • λ min (M), λ max (M) and tr(M) for minimal, maximal eigenvalues and trace of matrix M, respectively; • Euclidean norm x for vector x ∈ R n ; • the norm of a vector-function |φ( • an arbitrary matrix norm M and spectrum σ(M) for matrix M ∈ R n×n ; • let the space C[−τ, 0] = C([−τ, 0], R n ) be the Banach space of continuous functions mapping the interval [−τ, 0] into R n with the topology of uniform convergence;

Problem statement
In this paper we will study the exponential stability for the following neural networks where r are the synaptic connection weight matrices.The entries of W m , m = 1, r may be positive (excitatory synapses) or negative (inhibitory synapses).
We denote by f (x(t)) = [ f 1 (x(t)), f 2 (x(t)), . . ., f n (x(t))] ∈ R n the neuron activation functions which are bounded monotonically nondecreasing with f j (0) = 0 and satisfy the following condition ) Note that activation functions satisfying (2.2) imply Lipschitz condition and are special case of activations presented in [23].In [23, Subsection 3.2.1],there is survey of all activation functions applied in neural network models that have evolved from bounded cases to unbounded cases, from continuous to discontinuous, and from strictly monotonic case to nonmonotonic case.The conditions like (2.2) are essential when obtaining results on the existence, uniqueness, and global asymptotic or exponential stability of equilibrium point of neural network model.
Note that typical Lipschitz condition | can be used for activation functions only if they are not necessarily monotonic [24].
The bounded functions τ m (t) represent axonal signal transmission discrete delays of system with m = 1, r.Note that the condition (2.3) for derivative τm (t) is used for discretely distributed time-varying delays in method of Lyapunov-Krasovskii functionals when estimating derivative of the functional (see, for example, [6]).In contrary, the method that is offered in this paper does not directly require it.Suppose u ∈ R n is an equilibrium point of system (2.1), let x(t) = u(t) − u , and system (2.1) is transformed into the system with discretely distributed time-varying delays where x(t) ∈ R n is the state vector, g(x) = f (x + u ) − f (u ).Clearly, g belongs to sector non-linear function class defined by ) The initial conditions associated with system (2.4) are assumed to be where , under the assumption (2.5), there exists a unique trajectory of (2.4) starting from φ [7].Henceforth, we will focus our attention on system (2.4).

The basic steps of indirect method
We consider the system x(t) ∈ R n is the state vector, The method offered to find exponential estimate X(t, k, λ) = k|φ(θ)| τ e −λt includes the following steps.Remark 3.1.For the sake of simplicity we will use hereinafter the notations for exponential estimates Step 1. Write Cauchy formula for (3.1) Step 2. We choose Y(s, λ) as exponential estimate for y(s) satisfying to Cauchy-like formula Step 3. Consider distances where 1 for some δ > 0 is some monotonically increasing functional * .We get Step 4. Combining condition (3.4) and and taking into account (3.5) we can find parameter λ > 0 for exponential estimate X(t, λ).

Main result for neural network model
Theorem 4.1.Let system (2.4) be such that • matrix −A has all its eigenvalues with negative real parts.Pick α > 0 so that where λ i ∈ σ(−A).Let k = sup t≥0 e αt e −At < ∞, • there exists a solution λ > 0 of the quasipolynomial inequality Then the estimate x(t) ≤ k|φ(θ)| τ M e −λt is true for the solution of system (2.4) for any t ≥ 0, where λ > 0 is a number satisfying inequality (4.2).
Remark 4.2.The value of k is bounded provided that matrix −A has all its eigenvalues with negative real parts Proof.
Step 1.For the solution x(t) of the system (2.4) by virtue of the Cauchy formula the equality holds.Denote Then due to (4.1) the inequality holds.It is necessary to estimate x(t) , i.e., to find λ > 0 such that and let Y(t) be an unknown function such that for all [−τ M , ∞).
Step 2. Select function Y(t) so that Equality (4.7) does not guarantee that the equality Let us show that the function is a solution of (4.7).Indeed, we have Further, it is necessary to find λ > 0 such that x(t) ≤ X(t), y(t) ≤ Y(t), t ∈ [−τ M , ∞).

Let us first consider an interval
On this interval, let us derive a similar inequality for y(t) .Since we should have the value of x(t) on the interval [−2τ M , −τ M ].
For the sake of determinacy, let Then, taking into account that g j (•), j = 1, n are nondecreasing and denoting we obtain The last inequality holds for t ∈ [−τ M , 0] and for all λ > 0. Therefore, to derive the inequality y(t) ≤ Y(t), it is necessary to choose λ > 0 such that Then Step 3.For the further reasoning, let us introduce the notation It was shown that on the interval t ∈ [−τ M , 0] we have ρ 1 (t) ≤ 0 and ρ 2 (t) ≤ 0. Let us now find λ > 0 such that x(t) ≤ X(t) or ρ 1 (t) ≤ 0 for t ≥ 0. Let us estimate ρ 1 (t) by subtracting (4.7) from (4.5) (4.9) Considering (4.9), we can estimate ρ 2 (s): Some identical transformations yield )), their difference only increases if we assume that λ > 0 satisfies (4.2).
We obtain Since X(t) is monotonically decreasing, Therefore, taking into account (2.5), i.e., we have Since the integral is monotonic, substituting estimate (4.10) into (4.9)yields and calculating its derivative Corollary 4.8.For arbitrary m ∈ 1, r exponential decay rate estimate λ calculated based on the Theorem 4.1 is symmetric with respect to W m , i.e.
Moreover, the estimate depends exceptially on matrix norm W m .
Proof.It follows immediately from inequality (4.2) including matrix norms W m .

Illustrative examples
For the numerical experiment, simple examples are presented here to illustrate the usefulness of our main result.
Example 5.1.The model comes from [9, p. 808], where they considered the following simple two-neuron network with four delays (n = 2, r = 4) for some constant rate b: ( Considering initial conditions x 1 (t) ≡ 0.001, x 2 (t) ≡ 0.004, t ∈ [−τ M , 0] and applying Theorem 4.1 we can calculate the value of exponential decay λ.It can be readily solved by using the numerically efficient R package.Table 5.As a supplement, Fig. 5.1a shows the time response of state variables x 1 (t), x 2 (t) in this example with b = −0.1 and initial vector (0.001, 0.004) .Fig. 5.1b shows exponential estimate constructed in this model at b = −0.1.
Here from Table 5.1 we can see that positive estimate of exponential decay rate based on Theorem 4.1 can be calculated for b ∈ [−0.25, 0.25].That is in this case the equilibrium (0,0) of system (5.1) is delay-dependently locally exponentially stable

Comparing with the Lyapunov-Krasovskii functional method
The results of the application of the indirect method can be compared with the method of Lyapunov-Krasovskii functionals.Consider system (2.4) at r = 1, i.e. ẋ(t) = −Ax(t) + W 1 g(x(t − τ 1 (t))).
(5.2) In the work [6] there was offered investigation of neural network models with discretely and continuously distributed time-varying delays using Lyapunov-Krasovskii functional method resulting in LMI.Applying mentioned approach to the system (5.2) we apply the following positive-definite functional e 2λs g (x(s))Qg(x(s))ds (5.3) for unknown positive-definite matrices P, Q ∈ R n×n and positive constant λ > 0. Using the same technique as offered in [6] for exponential estimation of the solution of (5.2) we get the following LMI to search λ, P and Q In such a case we have exponential convergence for the solution of (5.2) of the form where γ(λ) = , l max = max{l 1 , . . ., l n }. (5.6) In this case we can find due to Theorem 4.1 the value of exponential decay rate λ = 0.3829018.The following Fig. 5.2 depicts the time responses of state variables of x 1 (t) and x 2 (t), which (5.8) In this case we can find due to Theorem 4.1 the value of exponential decay rate λ = 0.4877121.The following Fig.Then we have analyzed the system (5.8)applying the Lyapunov-Krasovskii functional method with help of LMI (5.4).When solving (5.4) with help of Scilab lmisolver it appeared that there does not exist solution λ ≥ 0 admitting (5.4) at P, Q > 0. That is we are not able to construct exponential convergence estimate for (5.8) with help of the Lyapunov-Krasovskii functional (5.3).

Conclusions
The indirect method offered here can be applied to other neural network models with delay.According to whether neuron states (the external states of neurons) or local fields states (the internal states of neurons) are taken as basic variables, neural networks can be classified as static neural networks or local field neural networks [13].For example, the recurrent backpropagation neural networks given below are static neural networks ẋ where x i is the state of neuron i with ∑ r m=1 ∑ n j=1 w m ij x j (t − τ m (t)) as its local field state.Systems (6.1) and (2.4) typically represent two fundamental modelling approaches in the present neural network research.Under the assumption that r = 1, W 1 A = AW 1 holds and W 1 is invertible, (6.1) can be easily transformed to network (2.4) by introducing the variable v(t) = W 1 x(t).However, in many applications, it may not be reasonable to assume that the matrix W 1 is invertible.Many neural systems exhibiting short-term memory are modelled by non-invertible networks.Moreover in case of multiple delays, i.e. r > 1, we are also not able to transform local field neural network to static one.
In order to apply the indirect method to get conditions for exponential convergence for (6.1) we let y(s) := ∑ r m=1 g(W m x(s − τ m (s))) at Step 1.Hence, as a result of obtained inequality (3.4) we are able to get condition for exponential stability.
It is worthwhile noting that in contrary to results obtained for static neural network model [23] this approach allows to consider multiple delays, i.e., r > 1.
The term 'indirect method' in the title of this work is used in order to contrast with methods of obtaining exponential estimates based on application of Lyapunov functions (or 'direct' method).
As compared with the Lyapunov-Krasovskii functional approach method offered here does not have such flexible possibilities for optimization of estimates and estimates obtained with help of developed approach are likely more rough and less accurate.
The "price" of this inaccuracy and roughness is comparatively clear form of expression for decay rate (as compared with multidimensional LMIs).This expression is quasipolynomial inequality which is well-known in stability analysis of delay differential equations.
Such simplicity of expressions is of importance in practical application like neural networks for obtaining analytical results.Namely, it allows to study dependencies of neural network exponential stability and changes in model parameters.
Moreover, as it was shown in the last example even with help of optimization and comparatively "flexible" functional there are cases if we are not able to construct exponential estimates in situations when "indirect" method gives us exponential convergence rates.

Remark 4 . 3 .Remark 4 . 4 . 1 . 4 . 5 .
Note that such values of α and k imply the following norm evaluation of matrix exponential e −At ≤ ke −αt for t ≥ 0. Note that in case of diagonal matrix A with positive entries, α can be chosen as α := min 1≤i≤n {a i } and k = Remark Assumption (4.2) for positive λ implies λ < α obviously.

1 :
1 shows the dependence of λ on the value of b.Dependence of value of b and λ > 0 calculated from (4.2) for Example 5.1

4877121 Figure 5 . 3
Figure 5.3 M , 2τ M ].Whence we may conclude that ρ 1 ≤ 0, t ∈ [0, ∞).The value of τ M admitting local exponential stability can be estimated from inequality At assumption of Theorem 4.1 there exists inverse dependency between τ M and λ.That is, when increasing in model (2.4) the value of τ M we decrease the estimate of exponential decay rate λ and vice versa.