Discrete-Time Zhang Neural Networks for Time-Varying Nonlinear Optimization

As a special kind of recurrent neural networks, Zhang neural network (ZNN) has been successfully applied to various time-variant problems solving. In this paper, we present three Zhang et al. discretization (ZeaD) formulas, including a special two-step ZeaD formula, a general two-step ZeaD formula, and a general five-step ZeaD formula, and prove that the special and general two-step ZeaD formulas are convergent while the general five-step ZeaD formula is not zero-stable and thus is divergent. Then, to solve the time-varying nonlinear optimization (TVNO) in real time, based on the Taylor series expansion and the above two convergent twostep ZeaD formulas, we discrete the continuous-time ZNN (CTZNN)model of TVNO and thus get a special two-step discrete-time ZNN (DTZNN) model and a general two-step DTZNN model. Theoretical analyses indicate that the sequence generated by the first DTZNN model is divergent, while the sequence generated by the second DTZNN model is convergent. Furthermore, for the step-size of the second DTZNNmodel, its tight upper bound and the optimal step-size are also discussed. Finally, some numerical results and comparisons are provided and analyzed to substantiate the efficacy of the proposed DTZNNmodels.


Introduction
As a subcase of nonlinear programming, nonlinear optimization has been widely encountered in a variety of scientific and engineering applications, and many applications can be modeled or reformulated as nonlinear optimization, e.g., the Markowitz mean-variance model in finance, the transportation problem in management, the shortest path in network model, the (non)convex separable optimization in image denoising, etc. [1,2].Due to its fundamental roles, nonlinear optimization has been extensively studied by many researchers during the last several centuries, and many efficient algorithms have been developed and investigated in the literature [3][4][5][6][7], which can be classified into two categories.The first category includes the first-order iteration methods, which only use the first derivative information of the objective function, such as the steepest descent method, the conjugate gradient method, and the memory gradient method.These methods are suitable to solve large-scale nonlinear optimization due to their simple structure and low storage.Numerically the conjugate gradient method and the memory gradient method usually perform better than the steepest descent method [6].The second category includes the Newton method and its variants, e.g., the quasi-Newton BFGS and DFP methods, which need to compute the second derivative of the objective function or an approximation of it.Therefore they are not suitable to solve large scale nonlinear optimization though they possess locally fast convergent rate.Then, to overcome the drawback of the quasi-Newton method, Nocedal [7] designed a limited memory BFGS method (L-BFGS) for nonlinear optimization, and numerical results indicate that the L-BFGS method is very efficient due to its low storage.In addition, during the last decades neural network has drawn extensive attention of researchers and practitioners due to its nice properties, e.g., distributed storage, high-speed parallel-processing, hardware applications, and superior performance in large-scale online applications [8].Some neural networks have been developed to solve nonlinear optimization during the last decades [9][10][11].Most of the above algorithms are designed intrinsically for solving static nonlinear optimization (SNO); therefore they might not be effective enough for solving time-varying nonlinear optimization (TVNO), whose objective function, denoted by ((), ), is a multivariate function with respect to the decision variable  and the time variable .The connection between SNO and TVNO is obvious: (1) SNO is a special case of TVNO, and when () =  for any  ≥ 0, TVNO reduces to SNO; (2) the discrete-time form of TVNO, whose objective function is (  ,   ) (  = ,  = 0, 1, . ..) and  > 0 denoting the sampling gap, can be viewed as a sequence of SNO.At each single time instant  =   , TVNO can be viewed as a static nonlinear optimization, and consequently it can be solved by the above mentioned algorithms.However this treatment is not advisable due to the following three reasons: (1) it is usually inefficient and lowly precise due to it need to solve a sequence of SNO [12]; (2) in the online solution process of discrete-time TVNO, the present and/or previous data with respect to   ( ≤ ) should be used sufficiently to generate the unknown decision variable  +1 ; (3) most importantly, at time instant   , we do not know the future information such as the function values (( +1 ),  +1 ), and only the current and past information for time instances   with  ≤  can be used.Therefore, the conventional static methods and the neural networks in [9][10][11], which are based on the future information, cannot solve TVNO.
As a special recurrent neural network, Zhang neural network (ZNN), named after Chinese scholar Zhang Yunong, serves as a unified approach to solve various online timevarying problems, such as time-varying quadratic function minimization [13], future minimization [14], time-varying matrix pseudoinversion [8], and TVNO [12,15,16].For example, based on ZNN, Jin et al. [15] presented a one-step discrete-time ZNN (DTZNN) model for TVNO, whose maximal residual error is theoretically O( 2 ).Subsequently, Guo et al. [12] proposed two DTZNN models for TVNO, which belong to three-step DTZNN with steady-state residual error (SSRE) changing in an O( 3 ) manner.Then, quite recently Zhang et al. [16] presented a general four-step discrete-time derivative dynamics model and a general four-step DTZNN model for TVNO; both models contain a free parameter which can take any values of the interval (1/12, 1/6) and is convergent with truncation error of O( 4 ).
In this paper we are going to further study the ZeaD formula, and three ZeaD formulas are presented, including a special two-step ZeaD formula with truncation error ( 3 ), a general two-step ZeaD formula with truncation error ( 2 ), and a general five-step ZeaD formula with truncation error ( 5 ).We prove that the first two ZeaD formulas is convergent while the third ZeaD formula is not zerostable and thus is divergent.Then, based on the Taylor series expansion and the above two convergent ZeaD formulas, we discrete the continuous-time ZNN (CTZNN) model for TVNO and thus get a special two-step DTZNN model and a general two-step DTZNN model for TVNO.Theoretical analyses indicate that the first DTZNN model is divergent, while the second DTZNN model is convergent for any  1 ∈ (−1/2, +∞) and the step-size ℎ ∈ (0, (2 + 4 1 )/(1 +  1 )).In addition, the tight upper bound of the step-size ℎ and the optimal step-size are also discussed.
The rest of the paper is organized as follows.We first recall some basic definitions and results in Section 2, including problem formulation of TVNO, continuous-time ZNN (CTZNN) model for TVNO, and the general -step ZeaD formula.In Section 3, a special two-step ZeaD formula with truncation error of ( 3 ) and a general two-step ZeaD formula with truncation error of ( 2 ) are presented and analyzed, and we also prove that the five-step ZeaD formula with truncation error of ( 5 ) or ( 6 ) is divergent in this section.Furthermore, based on the two convergent ZeaD formulas, two discrete-time ZNN (DTZNN) models for TVNO are presented, and we prove that the first DTZNN model is not convergent and the second DTZNN is convergent.Later, in Section 4, some numerical experiments are presented to illustrate and compare the performances of the convergent two-step DTZNN model with other variants.Finally, a concluding remark with future research direction is given in Section 5.The main contributions of this paper are summarized as follows.
(1) Three ZeaD formulas are presented, including a special two-step ZeaD formula, a general two-step ZeaD formula, and a general five-step ZeaD formula, whose convergence and stability are discussed in detail.
(2) Two DTZNN models are given to solve TVNO based on two convergent two-step ZeaD formulas, whose convergence is also studied in detail.
(3) The feasible region of the step-size ℎ in the convergent DTZNN model is studied, and its tight upper bound and the optimal step-size are also discussed.
(4) The high precision of the convergent DTZNN model is substantiated in numerical tests.

Preliminaries
In this section, the results in [12,16] are summarized for the foundation of further discussion, including the problem formulation of TVNO, the CTZNN model for TVNO, and the general ZeaD formula.
Firstly, the problem formulation of the TVNO is as follows [16]: where the time-varying nonlinear function (⋅, ⋅) : R  × [0,   ] → R is second-order differentiable and bounded.Problem (1) aims to find () ∈ R  such that the function ((), ) achieves its minimum at any time  ≥ 0. Thus in the sequent analyse, we assume that the solution of problem (1) exists at any time  ≥ 0.
It is well known that it is often hard to find the global optimum solution of time-invariant nonlinear optimization by traditional numerical algorithms [3,6].Therefore, researchers have resorted to find the stationary point of timeinvariant nonlinear optimization.We also transform problem (1) into finding its stationary point (), which satisfies the following nonlinear equations: where symbol ≐ denotes the computational assignment operation.In the following, we aim to find the solutions of problem (2) at any time  ≥ 0. Generally speaking, the solutions of problem (1) are the solutions of problem ( 2), but the inverse may be not, and if ((), ) is convex with respect to (), the inverse also holds [3].
Setting () = ((), ) in the following Zhang neural network (ZNN) design formula [18] we get the following continuous-time ZNN (CTZNN) model of problem (1) [15] where   ((), ) is the partial derivative of the mapping ((), ) with respect to its second variable , i.e., and ((), ) is the Hessian matrix of problem (1), i.e., which is assumed to be positive definite throughout the paper to ensure that the stationary point () of problem ( 1) is also its solution.
Remark .The main difference of the CTZNN model ( 4) and the neural network models in [9][10][11] lies in the former including the information of the time derivative   ((), ) to get fast and accurate solution of TVNO, while the motion equation in the latter neural network models for SNO and other optimization problems, such as variational inequalities and complementarity problems, can be expressed by and () : R  → R  , which is a mapping with respect to the decision variable  and is an implicit function with respect to .Furthermore, it generally satisfies ( * ) = 0, where  * is a solution of problem solving.When () =  for any  ≥ 0, that is to say that TVNO only contains the decision variable  and thus reduces to SNO, the CTZNN model ( 4) reduces to Obviously −( * ) −1 ( * ) = 0 for any  * being a solution of SNO; thus the CTZNN model ( 4) becomes a special neural network for SNO when () =  for any  ≥ 0.

Multistep ZeaD Formulas and Discrete-Time Models
In this section, we first propose two two-step ZeaD formulas with truncation error of ( 3 ) and ( 2 ) and prove that the five-step ZeaD formula with truncation error of ( 5 ) or ( 6 ) is not convergent.Then, two DTZNN models for TVNO are presented and analyzed subsequently.

. . Concepts of Convergence of Discrete-Time Models.
The following concepts about zero-stability and consistency are used to analyze the theoretical results of our proposed discrete-time models [20].
Concept .The zero-stability of an -step discrete-time method can be checked by determining the roots of the characteristic polynomial () =   + ∑  =1    − .If the roots of () = 0 are such that (i) all roots lie in the unit disk, i.e., || ≤ 1, (ii) any roots on the unit circle (i.e., || = 1) are simple (i.e., not multiple), then, the -step discrete-time method is zero-stable.
Concept .An -step discrete-time method is said to be consistent with order , if its truncation error is O(  ) with  > 0 for the smooth exact solution.
Concept .For an -step discrete-time method, it is convergent, i.e.,  [(− 0 )/] →  * () for all  ∈ [ 0 ,   ], as  → 0, if and only if such an algorithm is zero-stable and consistent (see Concepts 1 and 2).That is, zero-stability and consistency result in convergence.In particular, a zerostable and consistent method converges with the order of its truncation error.

. . Multistep ZeaD Formulas. Based on the Taylor series expansion
where  is a nonnegative integer, we can derive the two-step ZeaD formula, which is presented in the following lemma.
Proof.The proof is presented in Appendix A.
After we received the reviewers' comments on the paper, we were brought to the attention of the references [21,22], which have presented the rigorous proofs of Lemma 2 and the following Lemma 5.However, for the completeness of the paper, we have decided to give the proofs in the Appendix.

Corollary 4.
For any fixed and sufficiently small sampling gap  > 0, the truncation error of the general two-step ZeaD formula ( ) is decreasing as the parameter Proof.For any fixed and sufficiently small sampling gap  > 0, from the proof of Lemma 2, the truncation error is dominated by the term which obviously becomes smaller as The following theorem reveals that five-step ZeaD formula with truncation error of ( 5 ) or ( 6 ) is not convergent.
Proof.The proof is presented in Appendix B.
Figure 1 shows the graph of the function ( 6 ), from which we observe that ( 6 ) is always bigger than 1 except  6 = 0.However, from the definition of the general -step ZeaD formula, we have that  6 ̸ = 0.
Proof.Let Δ  =  +1 −   and Δ −1 =   −  −1 .Then, the proposed two-step DTZNN model ( 17) can be reformulated as On the other hand, by the Taylor series expansion, we have and where O(Δ 2  ) and O(Δ 2 −1 ) are absorbed into O( 2 ) as they are assumed to be of the same order of magnitude [20].By the algebraic manipulation "(23)/2 − (24)/2," the following results can be obtained: which together with ( 22) implies i.e., Setting   = ( +1 ,  +1 ) − O( 2 ), (27) can be written as The characteristic equation of the difference equation ( 28) is which has two different real roots from the discriminant Δ = ℎ 2 + 4 > 0. By [23], at least one root of the real quadratic equation ( 29) is greater than or equal to one in modulus.Thus, the sequence {  } is divergent, so is the sequence {(  ,   )}.
Proof.If ℎ ≥ (2+4 1 )/(1+ 1 ), then the characteristic equation of the difference equation (34) reduces to which has two different real roots Thus the general solution of the difference equation ( 34) is where  1 ,  2 two arbitrary constant which are determined by two initial states  0 ,  1 .So, the limit of the sequence {  } general does not exist except  1 = 0, which indicates that the sequence {‖(  ,   )‖ 2 } generally does not converge to zero.This completes the proof.
In the remainder of this subsection, let us investigate the optimal step-size for given  1 ∈ (−1/2, +∞).The discriminant of ( 35) is , which is a quadratic function with respect to ℎ, and its discriminant is The following analyses are divided into two cases according to the sign of Δ.
Overall, we get the following theorem.
Figure 4 shows the trajectories of (  ,   ) generated by the two tested models and their differences when  = 0.01, from which we can find that (  ,   ) generated by the DTZNN-I model is generally smaller than that generated by the DTZNN-II model, which means that the former is more accurate than the latter.Now, let us verify Theorem 11, and we compare the numerical results generated by the DTZNN model (18) with  = 0.01,  1 = −1/2, ℎ = 0.1 and those generated by the DTZNN model (18) with  = 0.01,  1 = −1/3, ℎ = (2 + 4 1 )/(1 +  1 ) = 1.The numerical results are depicted in Figure 5, from which we find that the generated sequence {‖(  ,   )‖ 2 } does not converge to zero when  1 = −1/2 or ℎ is equal to the upper bound (2 + 4 1 )/(1 +  1 ), and these are consistent to Theorem 11.
In the remainder of this section, let us verify Theorem 12 with  = 0.01,  1 = −1/3, ℎ = 0.1 and the optimal stepsize ℎ = 0.5 for fixed  1 = −1/3.The numerical results are depicted in Figure 6, which shows the performance of the DTZNN model with ℎ = 0.5 is better than that of the DTZNN model with ℎ = 0.1, and this is consistent to Theorem 12.

Conclusion
In this paper, we have investigated a convergent two-step ZeaD formulas with truncation error of ( 3 ), a convergent general two-step ZeaD formula with truncation error of ( 2 ) and a general five-step ZeaD formula with truncation error of ( 5 ), which is not convergent.Then, based on the two convergent ZeaD formulas, we presented two DTZNN models for TVNO and proved that one is divergent and the other with the free parameter  1 ∈ (−1/2, +∞) and step-size 0 < ℎ < (2 + 4 1 )/(1 +  1 ) is convergent.We also proved that (2+4 1 )/(1+ 1 ) is tight upper bound of ℎ and (1+2 1 )/(1+ 1 ) is the optimal step-size.Numerical results illustrate that the proposed DTZNN model is efficient for solving TVNO.
In the future the following two issues related to this paper deserve further studying: (I) Theorem 12 only considers the optimal step-size for any given  1 ∈ (−1/2, 0), and therefore we need to study the optimal step-size for any given  1 ∈ (0, +∞); (II) the general three-step DTZNN model and the general four-step DTZNN model proposed in [16,17] both do not give the relationship of the free parameter  1 and step-size ℎ, and therefore we are going to extend the technique used in Theorems 8 and 9 to study the two general multistep DTZNN models and explore the relationship of the parameter  1 and the step-size ℎ.