Pattern recognition with simple oscillating circuits

Neural network devices that inherently possess parallel computing capabilities are generally difficult to construct because of the large number of neuron–neuron connections. However, there exists a theoretical approach (Hoppensteadt and Izhikevich 1999 Phys. Rev. Lett. 82 2983) that forgoes the individual connections and uses only a global coupling: systems of weakly coupled oscillators with a time-dependent global coupling are capable of performing pattern recognition in an associative manner similar to Hopfield networks. The information is stored in the phase shifts of the individual oscillators. However, to date, even the feasibility of controlling phase shifts with this kind of coupling has not yet been established experimentally. We present an experimental realization of this neural network device. It consists of eight sinusoidal electrical van der Pol oscillators that are globally coupled through a variable resistor with the electric potential as the coupling variable. We estimate an effective value of the phase coupling strength in our experiment. For that, we derive a general approach that allows one to compare different experimental realizations with each other as well as with phase equation models. We demonstrate that individual phase shifts of oscillators can be experimentally controlled by a weak global coupling. Furthermore, supplied with a distorted input image, the oscillating network can indeed recognize the correct image out of a set of predefined patterns. It can therefore be used as the processing unit of an associative memory device.

Abstract. Neural network devices that inherently possess parallel computing capabilities are generally difficult to construct because of the large number of neuron-neuron connections. However, there exists a theoretical approach (Hoppensteadt and Izhikevich 1999 Phys. Rev. Lett. 82 2983) that forgoes the individual connections and uses only a global coupling: systems of weakly coupled oscillators with a time-dependent global coupling are capable of performing pattern recognition in an associative manner similar to Hopfield networks. The information is stored in the phase shifts of the individual oscillators. However, to date, even the feasibility of controlling phase shifts with this kind of coupling has not yet been established experimentally. We present an experimental realization of this neural network device. It consists of eight sinusoidal electrical van der Pol oscillators that are globally coupled through a variable resistor with the electric potential as the coupling variable. We estimate an effective value of the phase coupling strength in our experiment. For that, we derive a general approach that allows one to compare different experimental realizations with each other as well as with phase equation models. We demonstrate that individual phase shifts of oscillators can be experimentally controlled by a weak global coupling. Furthermore, supplied with a distorted input image, the oscillating network can indeed recognize the correct image out of a set of predefined patterns. It can therefore be used as the processing unit of an associative memory device.

Introduction
Information processing in biological systems is accomplished in a fundamentally different way than in conventional computers. Instead of having one (or a few) central processing units that perform a given task by executing commands step by step in series, biological neural networks consist of a large number of highly interconnected information processing units (the neurons) that all act in parallel. Furthermore, the rules are not predefined; rather, the network is capable of learning. Being able to mimic such a biological neural network in a man-made neurocomputer might allow for conceptually novel computer architectures with unforeseen areas of application. Key challenges are the large number of connections needed to couple all neurons mutually and the flexible adjustment of the connection strengths during the learning phase of the network.
In this respect, a charming mathematical model for an oscillatory associative memory was suggested by Hoppensteadt and Izhikevich [1]. It couples the oscillatory neurons dynamically through a time-dependent input rather than through direct connections between them. A physical network whose dynamics is governed by these rules would not encounter the abovementioned difficulties. The model comprises a number N of sinusoidal oscillators with different frequencies ω i whose phases ϑ i evolve according to ϑ i (t) = ω i t + ϕ i (t), where ϕ i are the phase shifts. All oscillators are subject to a weak time-dependent global couplinġ For a suitable choice of the coupling function a(t) and a small coupling strength, the phase shift dynamics reduces tȯ 3 where w i j are the elements of a coupling matrix defined below. System (2) is capable of pattern recognition similar to a Hopfield network [2][3][4][5][6][7]. The system is stationary if all phase shift differences ϕ j − ϕ i are either 0 or π . The coupling coefficients w i j determine the stability of the stationary solutions, with w i j > 0 stabilizing the solution ϕ j − ϕ i = 0 and with w i j < 0 stabilizing the solution ϕ j − ϕ i = π . The system is initialized to a binary pattern ξ with ξ i ∈ {−1, +1} by choosing w i j = ξ i ξ j . As a result all phase shifts corresponding to a '−1' in the pattern will converge to the same value ϕ − , just like all phase shifts that correspond to a '+1' will converge to ϕ + . The relative phase shift between the two groups will be ϕ + − ϕ − = π. In the recognition process, an imperfect pattern ξ is compared to a set of memorized patterns ξ k , k = 1, . . . , M. First the system is initialized to ξ . Then the coupling is altered according to the Hebbian learning rule With the changed coupling, ξ is no longer a preferred state of the network [5] and the phase shifts tend to evolve to a pattern near the ξ k that is closest to ξ . Note that the synaptic weights w i j are set only once for a given set of memorized patterns instead of gradual training. This means that the set of memorized patterns can be changed flexibly. In turn, other than in the Hopfield model, the set of memorized patterns is not kept in the system itself, so the system is a processor rather than a storage.
While there are some general ideas for experimental realizations of this network [8,9], none have been elaborated so far. One difficulty in doing so is that in general a coupling of physical variables does not lead to the sinusoidal coupling in ϑ implied in (1). Also, it is experimentally challenging to design a coupling mechanism that allows for arbitrary variations of the coupling function in time, including synchronizing and desynchronizing coupling. Below, we first propose a practicable coupling strategy for electrical oscillators with which the evolution equation of the phase shifts take the form (2) in good approximation. We then show experimental data for both pattern initialization and pattern recognition. In the last section, we describe the experimental procedures in more detail.

Coupled van der Pol oscillators
All experiments were performed with the circuit of N = 8 globally coupled van der Pol oscillators [10] depicted in figure 1(a). The frequencies lay in the range 28-73 kHz. Each oscillator had a resistance R in series allowing for independent oscillations. All oscillators were connected to the ground through a common external impedance. This impedance was a parallel connection of a constant negative impedance Z p = −R/N and a variable impedance Z c (t) = R c · a(t). Here, a(t) is the time-dependent coupling function we gave as an input to the system. Implementation details of Z p and Z c are given in the methods section ( figure 9). Applying Kirchhoff's laws to the circuit yields the time evolution of the voltage drop across an individual oscillator i: Here, I TD (U ) is the nonlinear characteristic of the tunnel diode and C i the capacitance of the oscillator. The last term globally couples a single oscillator to all other oscillators. The value for Z p was chosen such that the coupling term is proportional to a(t), which results in a coupling governed by the rate constant ε i : The order of magnitude is the same for all ε i in our experiment, where R = 500 and R c = 30 . A typical value for C i is 100 nF that results in ε i = 1200 s −1 . The specifications of all circuit elements are given in section 4.

Derivation of the phase equation and the coupling strength
Since ε i is much smaller than the frequency of an individual oscillator (for the considered capacitance ω i ≈ 4.5 × 10 5 s −1 ) the coupling is weak and the dynamics of a single oscillator can be reduced to a phase equation [11], which in our case reads: is the infinitesimal phase response function, i.e. the oscillator's change in phase per unit perturbation of U i if the perturbation is small. Expressing all ϑ i through ω i and ϕ i gives the time evolution of ϕ i :φ We determined the response functions Z i experimentally using a method similar to that used by Kiss et al [12]. The waveforms of the oscillation and the phase response are almost sinusoidal 5 in a wide frequency range, including the frequencies we used, with a phase shift of π/2 between them (see figure 1(b)). Approximating U i and Z i by a sine and a cosine function, respectively, yieldsφ Owing to the small value of ε i /ω i , ϕ i is a slow variable and any fast oscillating term on the right-hand side will have no average contribution toφ i . In particular, if a(t) = const, φ i = 0 for averaging times longer than a few oscillation periods. An average effect only arises if a(t) has resonant Fourier components with frequencies ω j − ω i . Provided that each resonance occurs only for one pair of oscillators (ω k − ω l = ω j − ω i for (k, l) = ( j, i), i.e. the frequencies form a so-called Golomb ruler [13]), the coupling function The analysis presented in this section provides a general way to map all kinds of systems of weakly coupled oscillators to (2). In particular, it relates the effective coupling strength ε to physical parameters.

Experimental pattern initialization and recognition
In the experiments, we recorded the voltage signal U i for each oscillator over a time period of 1 s. During this time, we changed the coupling impedance Z c (t) by applying a voltage U c (t) to a voltage-controlled resistor (VCR) that, in series with a constant negative impedance, provides Z c . The relation between U c and Z c was fairly linear (see figure 10 to obtain the coupling function (4). In the experiments we always kept U c ∈ [0 V, 3 V], which results in Z c ∈ [−30 , 30 ]. L i and C i were chosen such that frequency differences are pairwise different (numerical values are given in section 4).
The choice of the coupling coefficients w i j depended on the patterns we chose for initialization and recognition, as already described. From the measured voltages U i (t), we obtained the phase shifts by comparing the time of the actual zero crossings with the time of the zero crossings expected for an oscillation of frequency ω i : ϕ i (t cross ) = ω i (t cross, expected − t cross ). Figure 2 shows the initialization of the network with initially randomly distributed phase shifts to a pattern (the first 0.2 s). After each 0.2 s, the coupling coefficients were changed to represent a new pattern that differed from the old one in one pixel. Obviously, we can individually select oscillators and tune their phase shifts with a weak global coupling. Moreover, the phase shifts do indeed represent the patterns selected by the coupling. Switching between patterns took at most 0.1 s, after which time the difference between phase shifts was again close to integral multiples of π .
For recognition, we set w i j = ξ i ξ j for 0.5 s, to initialize the system to the pattern ξ . Then we changed the coupling matrix to w i j = 3 k=1 ξ k i ξ k j , as discussed above, giving the system three patterns to 'choose from' for recognition. With our network of only eight oscillators we used pairwise orthogonal (ξ l × ξ m = 0) patterns because otherwise the memorized patterns would have been too similar for successful recognition [16]. This does not mean that pattern recognition is in general restricted to orthogonal patterns for a larger number of oscillators [1]. Figure 3 shows a measurement with pattern recognition from three memorized patterns. The initial pattern had a defect in 1 bit. At 0.5 s, when recognition sets in, the phase shift of the respective oscillator switched from 0 to π . The switching took about 0.05 s. Now, the pattern represented by the system was the memorized pattern closest to ξ (in the case of figure 3, the pattern ξ 2 ), which means that the network recognized ξ as a defective version of ξ 2 . The common drifts in the phase shifts for t > 0.5 s do not derogate recognition. However, at long times the phase shift differences moved away from 0 and π , which means that the recognized pattern was lost after some time. This is not a substantial problem for pattern readout, which takes at most one period of the slowest oscillation until a zero crossing for each oscillator has been determined.  we define recognition as successful if the deviation of the phase shift from the expected value is smaller than π/2. In one case, however, the desired bit did not change during recognition. In the following section, we investigate reasons for this failure with simulations.

Influence of frequency deviations on pattern recognition
In experiments, the frequencies of the oscillators can only be determined up to a certain accuracy. Moreover, the frequencies might drift slightly, as was the case in our simple network. Thus, some deviations ω of the oscillators are intrinsically present in a measurement. Therefore, we quantified the effect of the frequency mismatch in the coupling on pattern recognition with the following simulations. First, we simulated a circuit of eight van der Pol oscillators with a slightly inaccurate coupling function where we chose the frequencies ω i,coup randomly from the interval [ω i (1 − ω/ω), ω i (1 + ω/ω)]. We used three randomly selected memorized patterns that were pairwise orthogonal for each single run. The initialized pattern had one randomly selected defective bit. The system of coupled differential equations we integrated numerically is given by (3)   In figure 5, the changes in all of the oscillators' phase shifts ϕ during the recognition process are displayed. For the oscillators representing the defective bits, the changes are shown in red for the rest of the oscillators in black. For each value of ω/ω, the average change in ϕ taken over ten simulations is plotted with a filled circle for the two groups of oscillators. Simulation results indicate that for ω/ω < 10 −5 , pattern recognition works, as all the oscillators that are supposed to exhibit a jump in phase shift change their phase shift by a margin considerably larger than π/2, while the changes in phase shift of the other oscillators stay considerably smaller than π/2. At ω/ω ≈ 10 −5 recognition fails in a few cases but still works most of the time. This was also the behavior we observed in the experiment ( figure 4). However, at frequency deviations ω/ω 10 −4 , both groups of oscillators are not distinguishable by their phase shift jumps during the recognition process any more. Therefore, the system cannot be used for pattern recognition in this case.
As can be seen in figures 6 and 7, the effect of inaccurate frequencies seems to be strongest for a small number of oscillators. Figure 6 shows a series of simulations with a system of 16 oscillators in the same frequency range and with the same coupling strength as the simulations of figure 5. Again, we used three memorized patterns with one defective bit in the initial pattern for each run. For this configuration, pattern recognition works fine up to ω/ω = 10 −4 , which is more than an order of magnitude better than that in the case of eight oscillators. Of course, the initial pattern deviates less from the restored pattern than in the eight-oscillator case, since N defective bits /N correct bits = 1/16 instead of 1/8. To investigate recognition with a comparable deviation of the initial pattern from the correct pattern, we also performed a series of simulations with two defective bits in the initial pattern, while leaving the coupling strength and oscillator frequencies unchanged (see figure 7). Also in this case, pattern recognition works up to ω/ω = 10 −4 , but compared to figure 6, the respective average changes are further away from 0 and π , respectively. These simulations indicate that pattern recognition is more robust to frequency inaccuracies with a larger number of oscillators under our specific experimental conditions. However, the condition ω < min i = j |ω i − ω j | must be fulfilled for any number N of oscillators. Otherwise, the single oscillators cannot be identified reliably by their frequency. As of now, the exact relation between N and the value All parameters are the same as in figure 6, only the initialized pattern had two defective bits.
of min i = j |ω i − ω j | is unknown, since all known algorithms for the computation of optimal Golomb rulers take exponential time [14], but there exists a numerical study [15] proving that for N 65 000 it is possible to find near-optimal Golomb rulers with where ω 1 is the minimal frequency and ω N is the maximal frequency. Therefore, we obtain the following constraint for the accuracy: It is reasonable to choose frequencies such that ω 1 , ω N and ω N − ω 1 have the same order of magnitude. If not, either a great deal of the available frequency space is wasted (if ω N − ω 1 ω 1 ) or the higher-frequency oscillators must have a much higher relative accuracy compared to the lower-frequency oscillators (if ω 1 ω N ). Therefore, we arrive at the following approximate condition for the accuracy of a single oscillator, where we denote the order of magnitude of the frequencies with ω: While this condition is a substantial constraint for realizations of larger networks, it does not explain why recognition fails at ω/ω = 10 −4 in a system with eight oscillators (where 1/N 2 ≈ 0.016), much less why recognition improves when we switch to 16 oscillators. The improved pattern recognition ability for larger networks can also be seen in figure 8, where a simulation of the recognition experiment with 60 oscillators suggested in [1] is shown for our van der Pol oscillator network. Thus, the simulations suggest that the experimental difficulties we experienced are due to a small size effect rather than due to the principal scalability issues of the network.

Conclusions
The experiments document that with a weak global coupling the phase shifts of oscillators can be manipulated individually and information can be stored in the relative phase shifts. In addition, we employed a network of eight such oscillators for pattern recognition tasks. We thus gave an experimental proof of principle that a network of weakly coupled electrical oscillators with a time-dependent global coupling via the electric potential is capable of pattern recognition. However, the phase shift dynamics of the realized circuit exhibited deviations from ideal behavior. Firstly, we observed a loss of coherence of the phase shift differences in the recognition mode. In most cases, the coherence was lost only on a time scale that was much longer than the recognition time. Therefore, pattern recognition worked on a short time scale. In some cases, however, recognition failed altogether. One possible reason for this is the accuracy of the oscillator frequencies. In simulations with a small number of oscillators (eight, 16 and 60), we observed that reliable pattern recognition requires a higher accuracy for a small number of oscillators than for a moderately large number of oscillators. Increasing the number of oscillators gave reliable pattern recognition at larger frequency deviations. This small size effect does not yet have a theoretical foundation. We also discuss the fact that the intrinsically finite accuracy limits the number of oscillators in the network and thus restricts the principal scalability (equation (6)), but in our experiment with eight oscillators this was not the crucial factor. Secondly, after switching to recognition coupling, we observed a common drift in all phase shifts. This so-called acceleration effect was discussed in the framework of frustrated coupled oscillator systems [5,17]. It occurs if the effective coupling between two oscillators is not an exactly sinusoidal function of the phase shift difference. However, since we are interested only in phase shift differences, the acceleration effect does not hinder successful pattern recognition. Thus, the origins of the deviations from the ideal phase shift dynamics observed in our network seem to be technical in nature and will not appear with optimized oscillators and larger networks.
One should also be aware that the network is a reusable processing unit rather than a classical neural network that learns and stores a limited set of patterns in the mainly fixed connection strengths. The trade-off is that additional peripheral devices, namely a memory to store the patterns and a function generator that provides the required coupling function for a set of memorized patterns, are necessary. In our experiments, a desktop computer with a DA card was used for this purpose. However, if need be, the peripheral functionality can be implemented on the same circuit board as the network itself to form an integrated device.
In conclusion, our experiments open the route to a qualitatively new hardware architecture. However, clever design changes that only partially use the elegant method for globally coupling the oscillators, but gain scalability instead, are needed. We are currently exploring approaches along this line.

Experimental setup
The measurement setup consisted of the circuit depicted in figure 9, a PC equipped with a Spectrum M2i.4032 AD card and a Spectrum M2i.6021 DA card, and a LeCroy WaveRunner 44MXi oscilloscope connected to the PC via ethernet. The oscilloscope and the AD card were each connected to four oscillator signals U i . We used the card with a time resolution of 1 MS s −1 and the oscilloscope with a time resolution of 10 MS s −1 . To obtain the times of the zero crossings of the signal, we linearly interpolated the intersection between two consecutive measured values with the first being smaller than zero and the second being greater than or equal to zero. We had to filter the oscilloscope's signal with the ERES algorithm included in the oscilloscope's software in order to avoid spurious zero crossings due to noise.
The van der Pol oscillators in the circuit are basically LC circuits with the addition of a tunnel diode. The negative nonlinear resistance in the characteristic of the tunnel diode (see figure 10(a)) destabilizes the stationary state of the LC circuit. The results are sustained oscillations in the range from 28 kHz (C = 660 nF) up to 73 kHz (C = 100 nF). The constant negative impedance Z p = −62.5 was implemented using a standard design [18] (see also figure 9). Z c is a series connection of another constant negative impedance with the value −50 and a VCR2N JFET transistor acting as a VCR. The output U c (t) of the DA card, for which we set the update rate to 1 MS s −1 , was connected to the VCR. Figure 10(b) shows the source drain resistance R JFET of the VCR against the control voltage U c applied to the gate. Since the VCR provides resistances in the interval [20 , 80 ], the series impedance of −50 allows for variation between equally large positive and negative values of Z c .
Note that, prior to each experiment, we had to determine the frequencies ω i . Therefore, we set U c = 0 (and thus a(t) = const). Then, we counted the number of zero crossings with positive slope N i for each oscillator during 1 s and assumed that ω i = 2π · N i Hz. This preliminary step was necessary because the frequencies are subject to slight changes over time due to thermal effects and fluctuations in the offset voltage U 0 of the tunnel diodes. Also, frequencies changed in the order of ω i ε between the coupled and the uncoupled system. In figures 2, 3 and 5, we corrected for these changes, which, however, do not interfere with the initialization and recognition capabilities of the network. U c applied to the VCR used in the experiment. Bottom: measured source drain resistance R JFET of the VCR for the control voltage given above. We put the VCR in series with a 12 resistance and measured the voltage drops across both resistances. From those measurements we computed the green curve for R JFET . The red curve was obtained using the WaveRunner 44MXi oscilloscope's ERES filtering algorithm on the green curve.

Measuring the phase response function
To obtain the phase response curves (see figure 1(b) for an example), we connected the van der Pol subcircuits to a variable voltage source U ex via their R = 500 series resistance. The time evolution equation for the oscillator signal U then readṡ A short high-amplitude pulse in U ex (ideally a delta function) will result in a jump in U . In the experiment, we used a pulse with the shape U ex (t) = 1.5 V · e −[t/(3×10 −7 s)] 2 , resulting in a voltage jump U with O( U ) ≈ 0.01 V. The excitation in U causes a shift in the phase ϑ. As phase variable we simply use ϑ(t) = 2π t/T , where T is the oscillation period.
To compute ϑ, we measure the time T ex between two zero crossings of the U signal directly before and after a pulse in U ex . By comparing this time to the period of the oscillator without perturbations, we obtain ϑ: from which the phase response curve Z ϑ (ϑ) = ϑ(ϑ)/ U was computed. The dispersion of the measured points is due to electronic noise. Noise effects could be reduced by using a larger U at the cost of moving further away from the undisturbed oscillation and therefore distorting the phase response curve.
This means that there are no contributions to g i j (ϕ i , ϕ j ) from the last two terms of the sum. We also made sure that which implies that there are only contributions from the first two terms of the sum if k = j and i = l: Evaluating this expression further yields g i j (ϕ i , ϕ j ) = 1 2 w i j sin(ϕ j − ϕ i ). Inserting the last term for g i j (ϕ i , ϕ j ) in (A.1), one obtainṡ Since the second term only takes effect on a much slower time scale than the first one, it may be neglected when considering the effective dynamics: With (A.2), we have arrived at (5) of this paper.

Appendix B. Golomb rulers used for determining the frequencies
For our simulations, we used ideal frequency distributions which maximize the minimal difference between the two frequencies ω i and ω j while still maintaining the condition ω i − ω j = ω k − ω l for (k, l) = (i, j). Such a distribution is called a Golomb ruler. We used the From these marks, we obtained the actual frequencies as within the frequency range [ω 1 , ω N ].