Embedding recurrent neural networks into predator–prey models

doi:10.1016/S0893-6080(98)00120-8

Neural Networks

Volume 12, Issue 2, March 1999, Pages 237-245

https://doi.org/10.1016/S0893-6080(98)00120-8 Get rights and content

Abstract

We study changes of coordinates that allow the embedding of ordinary differential equations describing continuous-time recurrent neural networks into differential equations describing predator–prey models—also called Lotka–Volterra systems. We transform the equations for the neural network first into quasi-monomial form (Brenig, L. (1988). Complete factorization and analytic solutions of generalized Lotka–Volterra equations. Physics Letters A, 133(7–8), 378–382), where we express the vector field of the dynamical system as a linear combination of products of powers of the variables. In practice, this transformation is possible only if the activation function is the hyperbolic tangent or the logistic sigmoid. From this quasi-monomial form, we can directly transform the system further into Lotka–Volterra equations. The resulting Lotka–Volterra system is of higher dimension than the original system, but the behavior of its first variables is equivalent to the behavior of the original neural network. We expect that this transformation will permit the application of existing techniques for the analysis of Lotka–Volterra systems to recurrent neural networks. Furthermore, our results show that Lotka–Volterra systems are universal approximators of dynamical systems, just as are continuous-time neural networks.

Introduction

Although major advances have been made, the field of application of dynamical system theory to recurrent neural networks still has much uncharted territory. Cohen and Grossberg studied a large class of competitive systems (which include many types of neural networks) (Cohen and Grossberg, 1983) with a symmetric interaction matrix for which they were able to construct a Lyapunov function guaranteeing global convergence. Hirsch considered cascades of networks and proved a cascade decomposition theorem (Hirsch, 1989) giving conditions under which convergence of the individual networks in isolation guarantees the convergence of the whole cascade. For systems with inputs, Sontag, Sussman and Albertini have studied their property from the point of view of nonlinear control system theory (Albertini and Sontag, 1993, Sontag and Sussman, 1997). But for the analysis of many properties of recurrent networks, simple computational tools are still missing.

Recently, we have started to investigate how changes of coordinates allow us to find equivalences between different neural networks. We have shown (Moreau and Vandewalle, 1997) how to use linear changes of coordinates to compute a transformation that maps a continuous-time recurrent neural network with a hidden layer onto a neural network without a hidden layer and with an extra output map. In the present paper, we will show how to embed different types of neural networks (having the hyperbolic tangent or the logistic sigmoid as activation function) into predator–prey models (also called Lotka–Volterra systems) of the form: $z ̇_{i} =λ_{i} z_{i} +z_{i} ∑ j=1 m M_{ij} z_{j}, with j=1,…,m.$ The Lotka–Volterra system will be of higher dimension than the original system but its n first variables will have a behavior equivalent or identical to that of the original system (if n was the number of variables of the original system).

Such a Lotka–Volterra representation of neural networks is of interest for several reasons. Lotka–Volterra systems are a central tool in mathematical biology for the modeling of competition between species and a classical subject of dynamical system theory (MacArthur, 1969, May and Leonard, 1975). They have therefore been the object of intense scrutiny. These systems have simple quadratic nonlinearities (not all quadratic terms are possible in each equation), which might be easier to analyze than the sigmoidal saturations of the neural networks. We thus expect that this representation will allow us to apply methods and results from the study of Lotka–Volterra systems to neural networks. In particular, we think about applying conditions for the global asymptotic stability of an equilibrium developed for quasi-monomial systems and Lotka–Volterra systems (Gouzé, 1990, Redheffer and Walter, 1984) to neural networks. Also, early work by Grossberg (1978) exploited one of the changes of variables that we present here to study convergence and oscillations in generalized competitive networks and generalized Lotka–Volterra systems. Furthermore, since we know that dynamical neural networks can approximate arbitrary dynamical systems (Sontag, 1992, Funahashi and Nakamura, 1993) for any finite time because of their equivalence with dynamical single hidden layer perceptrons (Moreau and Vandewalle, 1997), our result also serves as a simple proof that Lotka–Volterra systems enjoy the same approximation property; and we might want to investigate them further from that point of view.

We have structured this work as follows. First, we will consider the simple case of a dynamical perceptron. We will transform it into what we call a quasi-monomial form in a first step. This form will allow us to further transform the system into a Lotka–Volterra system. Second, we will consider the case of the dynamical perceptron with self-connections—that is, where each neuron has a linear connection to itself. We will perform the same transformations, first into a quasi-monomial form, and then into a Lotka–Volterra system. Finally, we will show that the same transformations apply to much more general types of continuous-time recurrent neural networks with hyperbolic tangent or logistic activation functions. In Appendix A Transformation of the dynamical perceptron into a quasi-monomial form, Appendix B Transformation of the dynamical perceptron with self-connections into a quasi-monomial form, we describe in detail the structure of the quasi-monomial forms for the dynamical perceptron and the dynamical perceptron with self-connections. In Appendix C, we clarify what the exact constraints on the activation function are and we show that both the hyperbolic tangent and the logistic sigmoid satisfy these.

Section snippets

The dynamical perceptron

We consider first an n-dimensional neural network of the following form, which we call a dynamical perceptron: $x ̇ = σ → (Ax+a_{0}),$ where x∈ $R$ ⁿ, $A∈ R^{n×n}$ , a₀∈ $R$ ⁿ, and $σ →$ is defined as $σ → (x)=(σ(x_{1}),σ(x_{2}),…,σ(x_{n}))^{T}$ with σ a continuous sigmoidal function from $R$ to $R$ such that $lim_{x→−∞} σ(x)=a$ and $lim_{x→∞} σ(x)=b (a,b∈ R$ and a<b).

We consider this type of system because its simplicity makes our calculations easier to follow, yet it permits us to illustrate the essential points of our method. In this paper, we do all

Transformation into a quasi-monomial form

By quasi-monomial form, we mean a dynamical system that we can write as $z ̇_{i} =z_{i} (∑ j=1 r A ̄_{ij} ∏ k=1 n z^{B_{jk}}_{k}), with i=1,…,n.$ The number of different products of powers of the variables is r and we define $A ̄ = A ̄_{ij} ∈ R^{n×r}$ and $B= B_{jk} ∈ R^{r×n}$ . It is important to note that the exponents B_jk are not restricted to integer values, hence the name quasi-monomial form. Now, to reach this form, we write Eq. (5) as $z ̇_{i} =z_{i} [∑ j=1 n A_{ij} (z^{−1}_{i} z_{j} −z_{i} z_{j})].$ We then need to collect all possible monomials and use their exponents to construct

Transformation into a Lotka–Volterra system

The whole transformation relies now on a simple trick: we look at each product of powers of the variables that appear in the quasi-monomial form as a new variable of our dynamical system. With r the number of different monomials in the quasi-monomial form, we define the supplementary variables z_n+1,…,z_n+r as $z_{n+k} = ∏ l=1 n z^{B_{kl}}_{l}, with k=1,…,r.$ If we now look at the system of n+r variables $z ̇_{i}$ , taking into account these new definitions, we have for the first n variables $z ̇_{i} =z_{i} ∑ j=1 r A ̄_{ij} z_{n+j}, with i=1,…,n.$

The dynamical perceptron with self-connections

One criticism that we can formulate about the dynamical perceptron is that its global behavior is often similar to the behavior around its fixed point. Although we can obtain complex dynamics by a careful choice of parameters, this is a delicate operation. The dynamics of such systems appear to be essentially limited. One way to obtain complex oscillatory behaviors more easily is to add a proportional self-connection to each variable: $x ̇_{i} =−λ_{i} x_{i} +σ(∑ j=1 n A_{ij} x_{j} +a_{0i}).$ The reason we can obtain

Embedding a dynamical multi-layer perceptron into a Lotka–Volterra system

The transformations we have shown for the dynamical perceptron and the dynamical perceptron with self-connections apply also to the dynamical Multi-Layer Perceptron (MLP). To avoid an unnecessary clutter of notation, we will take the specific example of a two-hidden layer continuous-time recurrent MLP: $x ̇ =σ(Cσ(Bσ(Ax+a_{0})+b_{0})+c_{0})$ where we have n states, p hidden units in the first layer, q hidden units in the second layer, and the activation function σ is the hyperbolic tangent. Thus, we find that

Discussion and conclusion

An important corollary of this embedding of continuous-time recurrent multi-layer perceptrons into higher-dimensional Lotka–Volterra systems is that we can use Lotka–Volterra systems to approximate arbitrarily closely the dynamics of any finite-dimensional dynamical system for any finite time. The argument for this universal approximation property is exactly the same as that given by Funahashi and Nakamura (1993) which says that, for any given finite time, the trajectories of two sufficiently

Acknowledgements

The work of Yves Moreau and Joos Vandewalle has been supported by several institutions: the Flemish Government, GOA-MIPS (Model-based Information Processing Systems); the FWO Research Community, ICCoS (Identification and Control of Complex Systems); the Belgian State, DWTC—Interuniversity Poles of Attraction Programme (IUAP P4-02 (1997–2001): Modeling, Identification, Simulation and Control of Complex Systems. Léon Brenig and Stéphane Louies are grateful to the CATHODE ESPRIT Working Group (WG

Yves Moreau is a Research Assistant with the F.W.O.-Vlaanderen

References (14)

F. Albertini et al.
For neural networks function determines form
Neural Networks
(1993)
L. Brenig
Complete factorization and analytic solutions of generalized Lotka–Volterra equations
Physics Letters A
(1988)
K.I. Funahashi et al.
Approximation of dynamical systems by continuous-time recurrent neural networks
Neural Networks
(1993)
S. Grossberg
Decisions, patterns, and oscillations in nonlinear competitive systems with applications to Volterra–Lotka systems
Journal of Theoretical Biology
(1978)
S. Grossberg
Nonlinear neural networks: principles, mechanisms, and architectures
Neural Networks
(1988)
M.W. Hirsch
Convergent activation dynamics in continuous-time networks
Neural Networks
(1989)
R. Redheffer et al.
Solution of the stability problem for a class of generalized Volterra prey–predator systems
Journal of Differential Equations
(1984)

There are more references available in the full text version of this article.

Cited by (15)

The almost periodic solution of Lotka-Volterra recurrent neural networks with delays
2011, Neurocomputing
By the fixed-point theorem subject to different polyhedrons and some inequalities (e.g., the inequality resulted from quadratic programming), we obtain three theorems for the Lotka–Volterra recurrent neural networks having almost periodic coefficients and delays. One of the three theorems can only ensure the existence of an almost periodic solution, whose existence and uniqueness the other two theorems are about. By using Lyapunov function, the sufficient condition guaranteeing the global stability of the solution is presented. Furthermore, two numerical examples are employed to illustrate the feasibility and validity of the obtained criteria. Compared with known results, the networks model is novel, and the results are extended and improved.
Almost periodic solutions for Lotka-Volterra systems with delays
2009, Communications in Nonlinear Science and Numerical Simulation
This paper studies a general class of delayed almost periodic Lotka–Volterra system with time-varying delays and distributed delays. By using the definition of almost periodic function, the sufficient conditions for the existence and uniqueness of globally exponentially stable almost periodic solution are obtained. The conditions can be easily reduced to special cases of cooperative systems and competitive systems.
Positive periodic solutions of delayed periodic Lotka-Volterra systems
2005, Physics Letters, Section A: General, Atomic and Solid State Physics
Citation Excerpt :
This type of neural networks have been implemented by subthreshold MOS integrated circuits [1]. In [14], it was shown that the continuous-time recurrent neural networks can be embedded into Lotka–Volterra models by changing coordinates, which suggests that the existing techniques in the analysis of Lotka–Volterra systems can also be applied to recurrent neural networks. Due to its importance in real applications, the analysis of dynamic behaviors, including stability, periodic oscillation, chaos and bifurcation, plays a key role in the studies on these dynamical models.
In this Letter, for a general class of delayed periodic Lotka–Volterra systems, we prove some new results on the existence of positive periodic solutions by Schauder's fixed point theorem. The global asymptotical stability of positive periodic solutions is discussed further, and conditions for exponential convergence are given. The conditions we obtained are weaker than the previously known ones and can be easily reduced to several special cases.
Quasipolynomial generalization of Lotka-Volterra mappings
2002, Journal of Physics A: Mathematical and General
Stability properties of a general class of nonlinear dynamical systems
2001, Journal of Physics A: Mathematical and General
Universal embedding of autonomous dynamical systems into a Lotka-Volterra-like format
2024, Physica Scripta

View all citing articles on Scopus

Yves Moreau is a Research Assistant with the F.W.O.-Vlaanderen

Joos Vandewalle is Full Professor at the K.U. Leuven.

View full text

Embedding recurrent neural networks into predator–prey models

Abstract

Introduction

Section snippets

The dynamical perceptron

Transformation into a quasi-monomial form

Transformation into a Lotka–Volterra system

The dynamical perceptron with self-connections

Embedding a dynamical multi-layer perceptron into a Lotka–Volterra system

Discussion and conclusion

Acknowledgements

Neural Networks

Physics Letters A

Neural Networks

Journal of Theoretical Biology

Neural Networks

Neural Networks

Journal of Differential Equations