Sequential associative memory with nonuniformity of the layer sizes

Jun-nosuke Teramae and Tomoki Fukai

Phys. Rev. E 75, 011910 – Published 11 January 2007

Abstract

Sequence retrieval has a fundamental importance in information processing by the brain, and has extensively been studied in neural network models. Most of the previous sequential associative memory embedded sequences of memory patterns have nearly equal sizes. It was recently shown that local cortical networks display many diverse yet repeatable precise temporal sequences of neuronal activities, termed “neuronal avalanches.” Interestingly, these avalanches displayed size and lifetime distributions that obey power laws. Inspired by these experimental findings, here we consider an associative memory model of binary neurons that stores sequences of memory patterns with highly variable sizes. Our analysis includes the case where the statistics of these size variations obey the above-mentioned power laws. We study the retrieval dynamics of such memory systems by analytically deriving the equations that govern the time evolution of macroscopic order parameters. We calculate the critical sequence length beyond which the network cannot retrieve memory sequences correctly. As an application of the analysis, we show how the present variability in sequential memory patterns degrades the power-law lifetime distribution of retrieved neural activities.

Received 17 May 2006

DOI:https://doi.org/10.1103/PhysRevE.75.011910

Authors & Affiliations

Jun-nosuke Teramae^* and Tomoki Fukai^†

Laboratory for Neural Circuit Theory, RIKEN Brain Science Institute, Hirosawa 2-1, Wako, Saitama 351-0198, Japan

^*Electronic address: teramae@brain.riken.jp
^†Electronic address: tfukai@brain.riken.jp

Article Text (Subscription Required)

Click to Expand

References (Subscription Required)

Click to Expand

Issue

Vol. 75, Iss. 1 — January 2007

Reuse & Permissions

Access Options

Author publication services for translation and copyediting assistance advertisement

Images

Figure 1
Time evolution of the pattern overlap during memory retrieval. A network model embeds an infinitely long sequence of memory patterns with a constant memory size, $a = 0.1$ . The pattern overlap was obtained from (a) numerical simulations of a 4000-neuron network or (b) numerical solutions to the order-parameter equations. The memory load, or equivalently the sequence length, was set as $α = 0.6$ (left), 0.7 (middle), and 0.8 (right).Reuse & Permissions
Figure 2
Storage capacity of a network model embedding a sequence of memory patterns. Their normalized sizes ${a_{μ}}$ were chosen from a binomial distribution at $b_{1} = 0.1$ (with Prob $1 - p$ ) and $b_{2} = 0.04$ (with Prob $p$ ). The capacity was calculated by numerically solving the order-parameter equations. The scaling factor can be calculated as $c = {(B_{1} - B_{2})}^{2} p (1 - p) ∕ (B_{1} B_{2}) + 1$ , where $B_{1} = b_{1} (1 - b_{1})$ and $B_{2} = b_{2} (1 - b_{2})$ , and changes nonmonotonically with $p$ (dashed curve). If the scaling factor is neglected in the expressions, the storage capacity increases monotonically with increasing $p$ (gray curve). The scaling factor, however, turns the storage capacity into a nonmonotonic function that takes a minimum value at a finite value of $p$ (black curve).Reuse & Permissions
Figure 3
Time evolution of the pattern overlap during memory retrieval. A network model embeds an infinitely long sequence of memory patterns with highly variable memory sizes. The sizes of memory patterns were chosen from a uniform distribution [0.01:0.1]. The pattern overlap was obtained from (a) numerical simulations of a 4000-neuron network or (b) numerical solutions to the order-parameter equations. The memory load, or equivalently the sequence length, was set as $α = 0.6$ (left), 0.7 (middle), and 0.8 (right). Note that the range of the horizontal axes is five times longer than that of Fig. 1 to show the stochastic behavior clearly.Reuse & Permissions
Figure 4
Time evolution of the average pattern overlap during memory retrieval. The dynamical trajectories were calculated from (a) numerical simulations of a 1000-neuron network similar to that shown in Fig. 3 or (b) numerical solutions to the order-parameter equations. Other parameters were the same as in Fig. 3. The trajectories were averaged over 1000 different realizations of memory patterns with variable sizes.Reuse & Permissions
Figure 5
Estimation of the mean lifetime of the sequence memory retrieval in Fig. 4. The average of the pattern overlap was taken over 1000 different realizations of the memory size fluctuations. (a) From top to bottom, the value of $α$ was increased from 0.6 to 0.7 with a step size of 0.01. We fixed the values of other parameters at the same values as in Fig. 2b. (b) Exponential functions of $α$ could well fit the curves shown above to obtain the decay constant.Reuse & Permissions
Figure 6
Memory retrieval in a neural network storing infinitely many sequences of finite size. Power-law distributions of the length of embedded sequences (gray) and the lifetime of retrieved sequences (black) were shown. The dotted line indicates a power law with exponent $- 2$ . The normalized sizes of sequential memory patterns $a_{μ, k}$ were chosen recursively from a Gaussian distribution with $σ = 0.05$ according to the rules shown in Eq. (22). The range of the pattern size was set as $a_{\min} = 10^{- 4}$ and $a_{\max} = 0.5$ . As we increase the number of memory patterns ( $α = 0.1$ , 0.2, 0.4, 0.8, and 1.6), the position at which an exponential cutoff appears in the power-law distributions shifts toward shorter lifetime. The inset shows the cutoff position as a function of $α$ .Reuse & Permissions

Physical Review E

covering statistical, nonlinear, biological, and soft matter physics