A SIMPLE DELAYED NEURAL NETWORK WITH LARGE CAPACITY FOR ASSOCIATIVE MEMORY

. We consider periodic solutions of a system of diﬀerence equations with delay arising from a discrete neural network. We show that such a small network possesses a huge amount of stable periodic orbits with large domains of attraction if the delay is large, and thus the network has the potential large capacity for associative memory and for temporally periodic pattern recogni- tion.


1.
Introduction. Multistability in a dynamical system referees to the coexistence of multiple stable patterns such as equilibria and periodic orbits. It has been shown that the coexistence of multiple equilibria/fixed points in a neural network lies at the basis of the mechanism for associative content-addressable memory storage and retrieval [15,21,22,28,32]. It has also been known that stable periodic orbits and limiting cycle attractors are important for memory storage and other neural activities as some form of memories are encoded as temporally patterned spike trains [8,15,16]( see also limiting cycle attractors in excitable cells [24] and in neural circuits constructed from invertebrate neurons [23]). According to Milton and Black [30], there are over 30 diseases of the nervous system in which recurrence of symptoms or the appearance of oscillatory signs are a defining feature. It was also noted in [29] that more than 25 years of experimental and theoretical work indicates that the onset of oscillations in neurons and in neuron populations is characterized by multistability.
Time delays are intrinsic properties of the nervous systems and unavoidable in electronic implementations due to axonal conduction times, distances of interneurons and the finite switching speeds of amplifiers. See, for example, [2,3,4,5,15,16,17,18,19,20,26,27,42]. Periodic orbits for delay differential equations and systems have been extensively studied in the literature. In particular, for a system of two coupled delay differential equations describing the information processing of two identical neurons with delayed feedback, the series of papers [9,10,11,12,13,14] established the coexistence of multiple periodic orbits and gave detailed description of their domains of attraction and the structure of the global attractor 852 JIANHONG WU AND RUYUAN ZNAG as the Morse decomposition of these periodic orbits and their connecting orbits. Most of these periodic orbits, except one, are unstable but with large domains of attractions in high dimensional submanifolds. These should be useful for secured encoding and communication, but less useful for associative memory due to the lack of their stability. See also [34,35,36,37,38,39] for work on the transient behaviors and on domains of attraction for a network of two neurons with delayed feedback.
Our purpose here is to show that a very simple and small network of neurons with delayed excitation/inhibition exhibits multistability and possesses a very large number of stable attractive periodic orbits with prescribed periods if the delay is sufficiently large and the updating is discrete. This is in sharp contrast with the aforementioned dynamical behaviors of a continuously updating network of neurons with delay for which at most one periodic solution can be stable. This phenomenon of multistability due to the coupling of discretely updating and time delay was observed in the series of papers [15,16,17] where it was shown that time delayed recurrent loops have a potentially large capacity to encode information in the form of temporally patterned spike trains. The model used in the aforementioned papers is a hybrid delay differential equation where the membrane potential of one neuron increases linearly until it reaches the firing threshold, when the neuron fires and is then reset to its resting membrane potential, the firing of the neuron excites another inhibitory interneuron such that at a time τ > 0 later the membrane potential of the excitatory neuron is decreased by a fixed amount ∆. Under general conditions on ∆ and τ , it was shown numerically that multiple periodic orbits exist and the presence of noise can cause switches between basins of attraction. Similar results were obtained for some simple scalar second-order delay differential equations with negative feedback, see [1,6,7,25].
The model considered in this paper is simple in its mathematical formulation and also for its hardware implementation. This is a system of two difference equations coupled through an excitatory feedback with an integer delay given below where n ∈ N (the set of all nonnegative integers), α > 0 (the case where α < 0 can be transformed into the case of α > 0 via a simple change of variables), β ∈ (0, 1), k ≥ 1 is a fixed integer, f : R → R is a nonlinear function satisfying standard conditions of the usual signal function such as McCulloch-Pitts step-function and sigmoid function with large gain f (0). Such a system describes the dynamics, updating discretely, of a network of two identical neurons where the information processing of a neuron involves the internal decay and feedback from another with a delay. It will be shown that for each positive prime integer p|2k such a system has (2 p − 2)/p distinct periodic orbits with the minimal period p and some prescribed signs (a similar but a little more complicated formula for the number of the distinct periodic orbits will also be given when p is not a prime number). Our approach is based on the elegant method recently developed by Walther in [40,41], that allows us to construct a closed bounded convex subset in a 2k-dimensional Euclidean space (based on some simple analysis of the periodic orbits of (1.1) with the simple step function f ) and a contractive self-mapping defined on this subset such that a periodic point of the mapping gives a stable and attractive p-periodic orbit.
The fact that in a continuous system of two coupled delay differential equations most of periodic orbits, except one, are unstable but with large domains of attractions in high dimensional submanifolds and the fact that a discrete analogue is capable of generating a large number of stable periodic orbits seem to suggest that a combination of discrete and continuous signal processing maybe the most effective way for neural information processing in order to achieve the optimal quality, large capacity, secured encoding and easy retrieval. We must emphasize that while the network of two neurons, when updated discretely, allows the coexistence of an amazingly large number of stable periodic orbits, it is still a remaining open problem how this can be used as a device for memory storage. The problem is that the model involves very limited number of parameters (β: the internal decay rate; α: the synaptic weight; and k the delay) and thus it is difficult to train such a network so that a large number of memories can be stored as stable periodic orbits. How our results can be extended to large networks with more complicated connection topology remains to be an interesting open problem.
The main results will be described and proved in Section 2, followed by short discussions in Section 3 related to the issue of training and distributed delays.

2.
Introduction. We consider the following nonlinear discrete-time system where n ∈ N (the set of all nonnegative integers), α > 0, k ≥ 1 is a fixed integer, f : R → R is a nonlinear function satisfying the following conditions: Note that if = 0, r = 0 and R = ∞, then f must be the widely used McCulloch-Pitts nonlinearity given by and in this case, L = 0. System (2.1) describes the evolution of a discrete-time network of two identical neurons with excitatory interactions, where β ∈ (0, 1) is the internal decay rate, f is the signal transmission function, and k is the signal transmission delay. See [42 for general backgrounds on delayed neural networks, and [43,44] for results about the existence of k-periodic orbits and 2k-periodic orbits. In this section, we consider the existence, stability, multiplicity and domain of attraction of p-periodic solutions for every positive integer p with p|2k.
Recall that by a solution of (2.1), we mean a sequence {(x(n), y(n))} of points in R 2 that is defined for every integer n ≥ −k and satisfying (2.1) for n ∈ N. In what follows, in a statement involving a " p-periodic solution", we always mean the p is the minimum period of the solution.
The following simple observation is useful for our existence result.
1 We wish to thank Cornelius Greither and a referee for bringing our attention to the connection of (2.10) with the well-known Möbius inversion formula described below: (iii): . The number of elements of the set Σ p is given by elsewhere. (2.10) (2.11) Therefore, s n+2k = s n for n ∈ N. So 2k is a period of {s n } n∈N . By (2.8) and π p σ = σ, we have This, together with (2.11) and (2.12), yields , · · · , 2k. For n ≥ 2k + 1, we have n = l(2k) + q with some l ≥ 1 and q ∈ {1, · · · , 2k − 1}. Therefore, s n+p = s l(2k)+q+p = s q+p = s q = s n . Thus, p is the minimum period of {s n } n∈N . As p < 2k and as 2k is a period of {s n } n∈N , we conclude that p|2k. This proves (i). (ii) follows easily as Σ = ∪ 2k p=1 Σ p . (iii) is simple.
We can now state our existence result.
x > 0, y < 0: In this case, we have This proves the claim.
We now describe the coexistence and the domains of attraction of periodic solutions.
Proof. We will divide the long proof into five steps.
Let Ω(σ, c) be given in (2.14). We first show that if c ∈ [0, r * ), σ ∈ Σ p and p|2k, then for any integer l wit 1 ≤ l ≤ k and for any w , w ∈ Ω(σ, c) (2.19) l Using the same argument as that in Step 1 of the proof for Theorem 2.2, we have w j w j > 0, F l j (w )F l j (w ) > 0 for j = 1, · · · , 2k and 1 ≤ l ≤ k. (2.19) l holds when l = 1, because by (H2) we have Assuming now that (2.19) l holds for some l with 1 ≤ l ≤ k − 1, then Similarly, we have holds. This proves the claim, and from (2.19) k we get Step 2. We now show that if {w(n, w 0 )} n∈N is a p-periodic solution of (2.3) with w(j, w 0 ) ∈ Ω for j = 1, · · · , p then |w(j, w 0 )| ≤ b * for j = 1, · · · , p.
To prove this, we first obtain a p-periodic solution {x(n), y(n)} of (2.1) from {w(n, w 0 )} n∈N . Assume, by way of contradiction, that there exists n 0 > p such that |x(n 0 )| > b * . We first consider the case where x(n 0 ) > b * . Then b * < x(n 0 ) < R, and we can write x(n 0 ) = b * + δ 0 with some δ 0 > 0. We have from (2.1), (H1) and (2.17) that Repeating the above argument, we get In particular, a contradiction to the p-periodicity. Similarly, we exclude the case x(n 0 ) < −b * .
(iv). We have from the result in Step 4 that the period p of any given periodic solution of (2.3) must satisfy that p|2k. This, together with (i) and the definition of N (p), completes the proof.
Note that two different periodic solutions may give rise to a single periodic orbit if they are the time-translation from each other. To determine exactly the number of periodic orbits, we introduce the following Definition 2.4. Two periodic solutions w(n, w ) and w(n, w ) of (2.3) are equivalent to each other, if there exists q ∈ N such that w(n, w ) = w(n + q, w ) for n = 0, 1, · · · . (2.21) Clearly, two equivalent periodic solutions w(·, w ) and w(·, w ) of (2.3) give the same orbit Lemma 2.5. For any fixed p ∈ N with p|2k, and any given σ and σ ∈ Σ p with σ = σ, w σ and w σ are equivalent to each other if and only if there exists q ∈ {1, · · · , p − 1} such that σ = π q σ.
Theorem 2.4. Let N * (p) be the number of p-periodic solutions of (2.1) which are not equivalent to each other. Then for each p ∈ N with p|2k, we have Proof. We have shown that N (p) is exactly the number of the elements of Σ p . For each p ∈ Σ p , the p-periodic solution w σ is equivalent to each of the following p-periodic solutions w πσ , · · · , w π p−1 σ . As π q σ = σ for any q ∈ {1, · · · , p − 1}, we conclude that w π i σ and w π j σ are not equivalent to each other when i = j and i, j ∈ {1, · · · , p − 1}. Therefore, for each w σ , there are exactly (p − 1) p-periodic solutions that are equivalent to each other. This completes the proof.
We conclude this section by listing values of N (p) and N * (p) for p = 1, · · · , 20.   3. Discussions. We show that a very simple and small network of neurons with delayed coupling exhibits multistability with a very large number of stable attractive periodic orbits with prescribed periods if the delay is sufficiently large. The model is a system of two coupled difference equations with delayed feedback, it is simple in its mathematical formulation and perhaps for hardware implementation.
The existence of multiple equilibria/fixed points and stable periodic orbits and limiting cycle attractors in a neural network is important for associative contentaddressable memory, and many neural activities are encoded as temporally patterned spike trains. Periodic orbits for delay differential equations and systems have been extensively studied in the literature. Though a system of two coupled delay differential equations describing the information processing of two identical neurons with delayed feedback exhibits the coexistence of multiple periodic orbits, all of these periodic orbits, except possibly one, are unstable but with large domains of attractions in high dimensional submanifolds. These are useful for secured encoding and communication, but less useful for associative memory due to the lack of their stability.
Our work shows that the discrete model is capable of generating a large number of stable periodic orbits. This raises the issue how a network utilizes a combination of discrete and continuous signal processing for neural information processing in order to achieve the optimal quality, large capacity, secured encoding and easy retrieval.
While the network of two neurons, when updated discretely, allows the coexistence of an amazingly large number of stable periodic orbits, it is still a remaining open problem how this can be used as a device for memory storage. The problem is that the model involves very limited number of parameters (β: the internal decay rate; α: the synaptic weight; and k the delay) and thus it is difficult to train such a network so that a large number of memories can be stored as stable periodic orbits. One way seems to replace the coupling term αf (y(n − k)) by a weighted sum involving multiple delays ∞ j=1 α j f (y(t − jk)) and then to determine the coefficients α j from the training data sets. This is equivalent to the training the density of the distribution of delay in a model of difference equations with distributed delays, and the problem will be addressed in a future work.