Keywords
neural coding, synaptic weights, Hebbian learning, intrinsic excitability, rate coding, spike frequency, neural circuits, neural networks, lognormal distributions.
This article is included in the INCF gateway.
neural coding, synaptic weights, Hebbian learning, intrinsic excitability, rate coding, spike frequency, neural circuits, neural networks, lognormal distributions.
Thanks to the reviewers for the thorough review and many thoughtful comments. I made two major revisions:
To read any peer review reports and author responses for this article, follow the "read" links in the Open Peer Review table.
Individual neurons have very different, but mostly stable, mean spike rates under a variety of conditions1,2. To report on behavioral results, spike counts are often normalized with respect to the mean for each neuron. But this obscures an important question: Why do neurons within a tissue operate at radically different levels of output frequency? In order to answer this question our approach is twofold: (a) we try to document this phenomenon for different neural tissues and behavioral conditions. We also examine neural properties for their distribution, namely intrinsic gains and synaptic weights; (b) we build a very generic neural model to explore the conditions for generating and maintaining these distributions. First, we give examples for the distribution of mean spike rates for principal neurons under spontaneous conditions, as well as in response to stimuli. We furthermore document distributions for intrinsic excitability3–5 for cortical and striatal neurons, as well as synaptic weight distributions6–11.
With the current data, we show that the distribution of spike rates within any neural tissue follows a power-law distribution, i.e. a distribution with a ‘heavy tail’. There is also a small number of very low-frequency neurons, so that we have a lognormal distribution2. This lognormal distribution is present in spontaneous spike rates as well as under behavioral stimulation. For each neuron, the deviation from the mean rate attributable to a stimulus is small (CV = 0.3–1, standard deviation = 1–4 spikes/s), when compared to the variability in mean spike rate over the whole population (5-7-fold), cf. Table 1 and Table 3.
Tissue | Mean μ | Median μ* | Variance σ2 | Width σ* | Mode eμ−σ2 | n |
---|---|---|---|---|---|---|
IT cortex39 | 1.5 | 4.50 | 0.71 | 2.32 | 2.2 | 100 |
A1 cortex2 | 1.6 | 4.95 | 0.47 | 1.98 | 3.1 | 145 |
A1 cortex40 | 3.3 | 27.11 | 0.69 | 2.29 | 13.6 | 263 |
Purkinje in vitro42 | 3.46 | 31.82 | 0.14 | 1.46 | 27.6 | 106 |
Purkinje in vitro41 | 3.44 | 31.19 | 0.198 | 1.56 | 30.0 | 34 |
Purkinje in vivo43 | 3.5 | 33.12 | 0.47 | 1.98 | 20.7 | 319 |
Inferior colliculus45 | 3.31 | 27.39 | 0.90 | 2.58 | 11.13 | 30 |
This work refers back to data initially reported in 1. At the time, we only had data on spike rates of cortical neurons available, plus independent evidence on intrinsic properties of striatal neurons. The observation on cortical data was taken up by2,12, and led to a number of papers13,14 focusing on the power-law distribution of spike rates as a cortical phenomenon, seeking explanations in the recurrent excitatory connectivity of cortical tissue2,13,15,16. However, we find the same spike rate distributions for midbrain nuclei, medium striatal neurons and cerebellar Purkinje cells, which do not have this kind of connectivity. It has even been found in the spinal motor networks of turtles17. We then extended the data search for intrinsic excitability and found that lognormal distributions are ubiquitous there as well, at least in cortical as well as in striatal tissues. Finally, lognormal distributions have also been found for synaptic weights6–11,18. The explanation for this universal phenomenon must lie elsewhere.
For this purpose we constructed a generic model for neuronal populations with adaptable weights and gains. We initialized both weights and gains with uniform, Gaussian or lognormal distributions. We then employed either Hebbian or homeostatic adaptation rules on both. Under a variety of conditions we could show that lognormal distributions develop from any initial distribution only with Hebbian (positive) adaptivity. Additional homeostatic adaptation stabilized learning but erased the lognormal distributions if it was stronger than Hebbian adaptation. We could even show that the widths of the distributions from the model match with the experimental data for rates, weights, and gains (Table 1 and Table 3) that we have available. Lognormal distributions can only be maintained by positive, Hebbian-type learning rules15, while homeostatic plasticity alone destroys lognormal distributions19. There are a number of different learning rules and variants which all follow the ’Hebbian’ principle: strong activation leads to strengthening, weak activation leads to weakening20. STDP rules are a variant of Hebbian learning for spiking neurons, which emphasize temporal sequence, but have the same positive learning effect21–23. It has been noticed that positive learning rules lead to run-away activation and unstable network behavior, and that they need to be counteracted by homeostatic processes24,25. We present a generalized model of synaptic learning which consists of both Hebbian (positive) and homeostatic (negative) adaptation rules24, and show that positive (Hebbian) learning is necessary to establish a lognormal synaptic weight distribution.
For intrinsic learning it has often been assumed that it may implement purely homeostatic adaptation26–30, but see also31. Experimental results are often inconsistent4,32–38. We will present results for a lognormal gain distribution in a number of tissues. It will be shown by simulation that the same principle holds: only a Hebbian, positive learning rule is capable of maintaining lognormal distributions, while homeostatic adaptation serves to establish stability. This finally answers the question that experimental researchers have investigated for some time: Is intrinsic plasticity mostly homeostatic, i.e. adjusts values inversely to use, or is there Hebbian, positive learning involved: when a neuron fires, does its gain increase? The answer is that the attested distribution of intrinsic gain can only derive from a Hebbian style adjustment rule, even though additional homeostatic adaptation is possible. Intrinsic plasticity is Hebbian.
In this section we report on data collection for spike rates, intrinsic properties and synaptic weights. Secondly, we explain the simulation model we constructed to explore the generation and persistence of the attested distributions.
We analyze five data sets for spike rates from principal neurons under behavioral activation:
In monkey IT, single unit activity was recorded over 200ms for passive viewing of 77 different natural stimuli for 100 neurons, each stimulus shown 10 times39. This yielded 770 spike rate response data points per neuron. For these data, we show mean spike rate, standard deviation, max-min values, coefficient of variation (CV) and Fano factor (FF) (Figure 1), (cf. 1). What is remarkable is that the dispersion for each neuron (variance related to mean, FF) is fairly constant, and not related to the rank of a neuron as high or low-frequency. In other words, neurons have roughly similar behavioral responses relative to their average spike rate. For this reason, many behavioral experiments have reported percentage of increase/decrease of spiking as the relevant parameter.
Additionally, we show data from primary auditory cortex (A1) from awake monkeys, which were recorded for spike responses to a 50ms, 100ms, or 200ms pure tone (40, Figure 2A and B) and data for spike rates from the primary auditory cortex of rats for four different conditions, which were recorded as cell-attached in vivo recordings (2, Figure 2C).
For midbrain nuclei neurons (IC), we re-analyzed spike rates in response to tones (for 200ms after stimulus onset) under variations of binaural correlation45. The frequency ranking of neurons by mean spike rate, standard deviation, min-max values, CV and FF are shown in (Figure 3A–C). CV and FF are similar to the cortical data. Data from GABAergic cerebellar Purkinje cells offer some difficulty for this analysis since they have regular single spikes at high frequencies, and in addition, calcium-based complex spikes43. Complex spike rates however are low (< 1Hz). This can therefore be regarded as a form of multiplexing, with two separate codes, where single spike rates can be separately assessed in their distribution. Here we report data for single spikes from in vivo recordings in anesthetized rats43 (Figure 4A) and data from spontaneous spiking (in the absence of synaptic stimulation) under in vitro conditions (41, 42, Figure 4B and C).
In order to show values for standard deviation and variance, data for two Purkinje cells from a behavioral experiment,44, i.e. single spike rates during arm movements from monkeys, have been added to the ranking of spontaneously firing neurons by mean spike rate from 41 (Figure 4D).
The logarithmic (heavy-tailed) distribution of spike rates is evident under all conditions.
The distribution of spike rates for neurons spiking in the absence of synaptic input shows that there are differences in the intrinsic excitability of neurons. To further explore this we looked at three additional datasets, which report the action potential firing of a cell due to injected current, such as by a constant pulse. This can be defined as the neuronal gain parameter (spike rate divided by current, [Hz/nA]).
In 1, we already presented the data from striatum, which show that the spike response to a constant current follows a heavy-tailed distribution3. Figure 5A shows the spike rate in response to current pulses of different magnitude in two different areas, nucleus accumbens (NAcb) shell and dorsal striatum. Figure 5B and C show the distribution of rheobase (current-to-threshold) for dorsal striatal and NAcb shell neurons. Distributions appear mostly lognormal, with the exception of the 200pA current pulse response and the data in Figure 5C, which appear normally distributed.
We extend this dataset by recordings from different types of cortical neurons in cat area 17 in vivo (46, Figure 6A and B) and from GP in awake rats (5, Figure 6C).
A lognormal distribution of intrinsic gain is clearly apparent, except for fast-spiking interneurons, which, however, may be a sampling error (n=33).
Synaptic weight distributions have been investigated starting with10 in hippocampus by measuring EPSC magnitude6,47–50 (Figure 7). There is also a review paper available51 to summarize the findings. Recently, the expression of AMPA receptor subunit GluA1, which is correlated with spine size, has also been measured52, Figure 8. We used five datasets from cortex, hippocampus and cerebellum:
1. EPSPs for deep-layer pyramidal-pyramidal cell connections in rat visual cortex6,47
2. EPSP amplitudes for deep-layer excitatory neuron connections in somatosensory cortical slices of juvenile rats49
3. EPSP amplitudes for CA1 to CA3 connections in guinea pig hippocampal slices10
4. EPSCs for granule cells to Purkinje cells in adult rat cerebellar slices50
5. labeled GluA1 AMPA receptor subunit in mouse somatosensory barrel cortex52
In 6, EPSP magnitude was measured for L5 pyramidal neurons in slices from rat visual cortex, averaged over 45–60 responses, and peak amplitude recorded, (Figure 7A). Similar data were used in 49 for slices from a single barrel column in rats (Figure 7B). A 30-fold variation of coupling strength was noted. In 10, EPSPs between CA3 and CA1 in hippocampal slices were recorded, by detecting somatic membrane potential changes in response to presynaptic neuron stimulation (Figure 7C). There are also synaptic weight data on granule cell to Purkinje cell connections11,50,51, which show a similar distribution, but have an order of magnitude weaker connections than cortical connections (Figure 7D). Finally, a different type of evidence was obtained in 52, namely labeling for a subunit of AMPA receptors in layer 2/3 mouse barrel cortex in vivo both before and after whisker stimulation. The AMPA intensity is distributed lognormally over the spines, corresponding to the observations on the strengths of EPSPs. It is noticeable that stimulation leads to increase of on average 200% (two-fold) in about 30% of spines52. Yet as we know, over time the overall distribution of synaptic strengths remains stable. For synaptic weights, just as for intrinsic gains and spike rates, lognormal distributions have been found for both EPSPs and AMPA receptor distribution in a highly consistent manner.
In many cases, the data were only available in the form of histograms. The parameters of the lognormal distribution were then obtained by fitting the data using a Nelder-Mead optimization method. A number of parameters were derived from these fits and reproduced in Table 1–Table 3, cf. 3.2.
Tissue | Peak [Hz/nA] | Mean µ[Hz/nA] | Median µ* | Variance σ2 | Width σ* | Mode eµ–σ2 | n |
---|---|---|---|---|---|---|---|
Dorsal striatum3 | 48 | 4.24 | 69.41 | 0.36 | 1.82 | 48.3 | 28 |
NAcb shell3 | 65 | 4.67 | 106.70 | 0.49 | 2.01 | 65.4 | 24 |
GP in vivo5 | 6.6 | 3.4 | 29.96 | 1.54 | 3.46 | 6.4 | 146 |
GP model5 | 37 | 4.0 | 54.60 | 0.40 | 1.88 | 36.6 | 10000 |
Cortical46 | 105 | 4.96 | 142.59 | 0.31 | 1.75 | 104.6 | 220 |
Tissue | Mean µ | Median µ* | Variance σ2 | Width σ* | Mode eµ−σ2 | n |
---|---|---|---|---|---|---|
Cortex L237 | -0.99 | 0.37 | 0.76 | 2.39 | 0.17 | 48 |
Cortex L238 | 0.25 | 1.28 | 1.41 | 3.28 | 0.31 | 35 |
Cortex L239 | -0.94 | 0.39 | 1.54 | 3.46 | 0.08 | 61 |
Cortex L547 | -0.56 | 0.57 | 1.47 | 3.36 | 0.13 | 1004 |
Cortex L549 | -0.31 | 0.73 | 0.58 | 2.14 | 0.41 | 26 |
Hippocampus10 | -2.61 | 0.07 | 0.43 | 1.93 | 0.05 | 71 |
Cerebellum GC-PC11 | -2.70 | 0.07 | 1.82 | 3.85 | 0.01 | 104 |
Cortex in vivo52 | -1.14 | 0.32 | 0.90 | 2.59 | 0.13 | 560 |
Given are two neuron populations I, J each with n = 1000 neurons and variable random connectivity C between I and J. C determines the density between I and J. The input population I always has excitatory output onto J. Inhibitory input to J is modeled by a population H with n = 200. The output neuron population J may also have recurrent excitatory connectivity. Figure 9 shows the architecture for the generic neural network used. The model (GNN) was programmed in Matlab, and is available in the public repository github (https://github.com/gscheler/GNN, DOI: https://doi.org/10.5281/zenodo.829949).
The input population I is modeled according to2 for pyramidal cortical neurons with a spike rate distribution of µ* = 4.95 and σ* = 1.98 (Table 1). The goal is to generate a spike rate distribution RJ for J, given a gain distribution G for the target neurons, and the weight distribution WIJ such that RJ is similar to RI.
For each neuron j in J the spike rate rj is calculated by applying its gain gj to the weighted sum of its connected excitatory inputs and its inhibition. Cj is the set of neurons from I that have excitatory connections to neuron j.
where the rate ri is taken from the distribution RI, from RH, wij from WIJ, and gj from G. gj is modeled as a factor for a linear gain function. It is possible to use a sigmoidal gain function instead, but this makes no difference for the conclusions from the model (Section 3.3). The output RJ may be used as input to I with a matrix WI,J for tests of the adaptation rules.
For the adaptation of weights WIJ and gains G, we use Hebbian or homeostatic rules, as described in Section 3.3. The system described in this way is sufficient for all the calculations on the shape of distributions used in this paper.
We have documented the distribution of spike rates, gains, and weights for different types of neurons (Figure 1–Figure 8). The distribution in all cases follows a lognormal shape. In some cases, we had data on the variability of spike rates and analyzed them for dispersion (CV, FF) under behavioral stimulation. While the fold-change from low spiking neurons to high spiking neurons is high, 5- to 7-fold, the variability for each neuron is comparatively low. It also seems to be adequately described by a percentage change over the whole population. This means that a low spiking neuron never reaches the same rate as a high spiking neuron, even when fully activated.
The similarities across neural systems are striking. For instance, in a midbrain nucleus (inferior colliculus) which is essentially an ’output’ site for auditory and somatosensory cortex, spike rates are high overall45, nonetheless the distribution of mean spike rate, and the variability are comparable to cortical data (Figure 3). Hippocampus, cerebellum and cortex vary in degree of bursting and spike irregularity53, but the rate distribution is constant. The distribution of mean spike rate is also essentially the same under spontaneous and under behavioral conditions.
Lognormal distributions were obtained by fit to the histograms obtained from data (goodness of linear fit, mean ≈ 0.92, s. Figure 1–Figure 8). The lognormal distribution is a very simple statistical distribution54, almost as simple and as universal in the description of natural processes as a Gaussian distribution (with which it is identical for small σ*). Even though the datasets were occasionally fairly small, and more data could be added to obtain greater precision, the conclusion seems warranted that the underlying natural process is as simple and general as the multiplication of independent variables55, rather than assuming more complex processes which may lead to other exponential-family distributions.
Lognormal rate distributions appear to be an essential property of neural tissues that occur in areas with very different neuron types and connectivity, and different absolute spike frequencies. They are present during spontaneous activity, and under activation of a network, in vivo as well as in vitro. They have a counterpart in a lognormal distribution of intrinsic excitability, and lognormal synaptic connectivity. This type of distribution seems to be an essential component of the functional structure of a mature network, which is not altered by learning, plasticity, or processing of information.
A lognormal distribution is characterized by parameters µ* and σ*. µ* = eµ is the median, a scale parameter, which determines the height of the distribution. σ* = eσ is the multiplicative standard deviation, a shape parameter which determines the width of the distribution. For distributions with small σ*(approximately σ* < 1.2 or σ < 0.182) a lognormal distribution is essentially identical to a normal distribution. (The coefficient of variation CV ~ σ* − 1, so that for CV < 0.18, a lognormal equals normal distribution.) We collected data on spike rate, gain and synaptic weight distribution for a number of tissues in different experimental conditions (Table 1–Table 3). For the height of the spike rate distribution, there are known differences, e.g. with lower values for cortex (µ* ≈ 4.5) and higher values for Purkinje cell (µ* ≈ 30) and midbrain nuclei (cf. Table 1). In other words, spike rates differ between brain areas such as cortex and cerebellum by a factor of 10.
In contrast to that, the width of spike rate distributions is more similar, with an average at σ* ≈ 2.2, with one outlier. The gain has a smaller σ*, i.e. a more normal, less heavy-tailed distribution than the spike rate. Minus the outlier (3.46), the mean for σ* is only 1.86, considerably lower than the width of the spike rate distribution (Table 2). For weight distributions (Table 3), the width σ* is consistently larger, with an average of almost 3 (2.91). The synaptic strength (µ*) varies over at least one order of magnitude between cortex and cerebellum.
It turns out that σ* values are significantly different for rates, gains and weights, lowest for gains (σ* ≈ 1.8), higher for rates (σ* ≈ 2.2) and highest,(σ* ≈ 3), for weights. The data that we have are not precise enough to draw quantitative conclusions, but no large distinctions are apparent between the tissues (Table 1 – Table 3). We use a generic neural network to recreate lognormal distributions by adaptation rules and we will also show that distribution widths are structural properties which follow from general network properties.
Since not only mean spike rates but also both components, intrinsic excitability and synaptic weights, have lognormal distribution, this raises the question of how the functional system that we observe is generated. It is obvious, if the data are accurate, that these are basic parameters of any simulation and need to be reproduced in any model to make it biologically realistic.
We set up a generic neural network model (cf. Section 2.2) to explore the mechanisms of generating and maintaining rate, weight and gain distributions. The model consists of a source neuron group I, a target group J, a population of inhibitory neurons H, which are connected with J, and potentially recurrent excitation in the target group J. The spike rate distribution RI acts through a weight distribution W onto a gain distribution G, where inhibition H is subtracted, and a spike rate output distribution RJ is produced (Figure 9).
In the simplest case, we look at two sets of neurons, the source and the target. The source sends excitatory connections to the target, and exhibits variable weights at outgoing synapses. The input that a target neuron receives is fed through a linear filter G to produce an output rate RJ according to Eq (1). The distribution for RJ depends on G and W as well as on RI. The system is sufficient for calculations on the shape of distributions, as well as the effects of Hebbian and homeostatic plasticity.
We have explored the dependencies between gain, weight and rate distributions in simulations. First, we found that the width of the output spike rate distribution RJ depends heavily on the gain distribution, but only slightly on the input weight distribution (Figure 10). It does depend on the overall connectivity C, where is wider for lower connectivity, but not very much (Figure 10). Secondly, the width of the output distribution RJ does not depend on RI or RH either (Figure 11). The most important factor for a spike rate distribution remains the gain
We may now ask, where do lognormal spike rate distributions come from? How is the system set up, i.e. what rules of adaptation generate lognormal distributions in weights and gains?
In the case of cortical networks, there are excitatory recurrent interactions that constitute a significant part of total input. In the case of cerebellar or striatal neurons, there are no recurrent excitatory interactions, only inhibitory interneurons and excitatory input. The generation of lognormal distributions must therefore be independent of recurrent excitation. It requires a system where continuous input shapes the weights and gains of a target network J. We start with the system that we described before, with random assignment of weights and gains. We employ adaptivity for weights, and also for gains, by positive Hebbian learning, or by negative homeostatic learning. The output of I is fed into J, and W and G are adaptive. Additionally, J may have excitatory recurrent connectivity, and learning takes place within the network J.
From any given initial spike rate distribution (Gaussian, uniform, lognormal) for I, we calculate W assuming a positive learning (Hebbian) adjustment rule, which is dependent on input and output frequencies. Each individual weight wij is updated by
We use parameters λ and µ, such that the generated spike rate output RJ is compatible in strength with the input rate RI.
Using Hebbian learning, we generate a weight distribution W that is lognormally distributed, independent of the initial configuration or the distribution of the gains in the system (Figure 12). The lognormal distribution also develops independently of the rate distribution of the inputs, it only develops faster with lognormal rather than normally distributed spike rate input (not shown). It makes no difference whether we use a recurrent system J, or a non-recurrent population J with input from a population I with a spike rate distribution, as long as we use a Hebbian weight adaptation rule. For the shape of the distribution, it also does not matter whether we route the output of J back to I, or whether we use local or no recurrence. To show the effect of the adaptation rule, we also used homeostatic synaptic plasticity to adjust the weights. This means that the weight is adjusted inversely to the spike rate of input and output neurons.
In this case, it is very clear that with any input or initial configuration and any gain distribution, only a normal distribution of weights results (Figure 13). Again, a lognormal input spike rate slows the process of adaptation, but the end result is the same, a normal distribution.
Since gain distributions are also lognormal, we may ask in the same way how they develop and are maintained by plasticity rules. We adapt the linear gain G by either Hebbian or homeostatic learning. Each gain can be adjusted by a Hebbian rule
or a homeostatic rule
with parameters λ and µ.
We start with uniform or normally distributed G in an environment where W is lognormal, normal or uniform, and RI is normal or lognormal. If we adapt only G for any initial configuration, using any distribution for RI, including the lognormal distribution, and a lognormal or normal weight distribution, we arrive at a normal distribution for G with homeostatic learning and a lognormal distribution with Hebbian learning (Figure 14).
Lognormal distributions develop from Hebbian plasticity, and homeostatic plasticity generates only normal distributions. The explanation lies in the nature of random statistical events, which generate normal distributions when the underlying mechanisms are sums of many small events, but lognormal distributions when the underlying mechanisms are multiplicative54. We also wanted to understand the observed widths of the distributions. We hypothesized that the differences for σ* between W, R and G result from the network structure. Accordingly we started a simulation with initial uniform values for G and W and Hebbian update rules using the same learning rate λ for both (Figure 15). We find that gain, rate and weight distributions match the experimental values, and that this is true for any tested constellation. We also found that Hebbian learning alone quickly escalates values, which develop exponentially, and that additional rounds of homeostatic adaptation are required to stabilize the system. Homeostatic learning pushes the system back towards a normal distribution.
Our data, in the most general way, allowing for various conditions and architectures, show that Hebbian learning is required both for intrinsic gain and for weights in order to generate the attested lognormal distributions. This is an interesting result, because it shows that we need prominent Hebbian intrinsic learning to explain the gain distributions that we find experimentally. Intrinsic learning is not just homeostatic adaptation, it follows the same rules as synaptic weight learning.
Spike rates of neurons seem to be universally distributed according to a lognormal distribution, with many neurons at low spike rates, and a small number at successively higher spike rates (heavy-tail)1. The same distributions are found for synaptic weights6, and intrinsic properties associated with excitability (gain)1. The neurons that we reported on are of very different types, and they are embedded in different kinds of connectivity. Medium spiny neurons and Purkinje cells are GABAergic (inhibitory), while cortical and IC neurons are glutamatergic (excitatory), but this is not reflected in a distinct spike rate distribution. They also fire with very different average spike rates. IC neurons operate at very high frequencies, and Purkinje neurons at much higher frequencies than cortical or striatal projection neurons. But they all have the same spike rate distribution. It has been suggested2 that lognormal spike distributions are a feature of cortical tissue and arise from strong excitatory recurrent connectivity, but this is experimentally not substantiated nor is it theoretically necessary. While cortical pyramidal neurons exist in a heavily excitatory recurrent environment, medium spiny neurons, cerebellar Purkinje cells and IC neurons act mostly in a feed-forward way, i.e. they don’t have significant recurrent excitatory (glutamatergic) connectivity.
Beyond spike rate distribution, we also gathered data on weight and gain distributions. Again the observation of lognormal distributions is ubiquitous. We find synaptic weight distributions for cortex6 and cerebellum that are lognormal, with characteristic width of distributions. For intrinsic properties, striatal projection neurons and cortical neurons46 show responses to constant current and current-to-threshold (gain) distributions, which again appear lognormally distributed, with smaller widths than spike rate distributions.
Our models show that lognormal distributions arise even in a purely input-output environment, and that they are a result of Hebbian learning of weights and gains, quite independent of the overall magnitude of the spike rates.
Mean spike rates, as well as intrinsic excitability and synaptic weights, have lognormal distributions.
It has often been assumed that variability in intrinsic excitability is a source of noise in neural computation56, even though others have argued that intrinsic variability contributes to neural coding57,58 and that intrinsic plasticity follows certain rules19. An excellent overview of the experimentally attested forms of intrinsic plasticity is contained in 35, cf. 59–61. Many other detailed observations are contained in 36–38,62.
Recently, Mahon and Charpier4 have shown that intrinsic excitability is stable in individual neurons under control conditions, while stimulation protocols (e.g. in barrel cortex of anesthetized rats) change intrinsic excitability by at least 50–100%. However, the conclusions drawn from the experimental research are often contradictory. Intrinsic plasticity is sometimes assumed to act in a negative, homeostatic way, i.e. opposite to synaptic plasticity4, but sometimes in a ’Hebbian’, positive way, i.e. cooperative with synaptic plasticity38,63. There is evidence for (short-term) negative or homeostatic plasticity, which has been previously investigated4.
Our work has now shown that any kind of neural system with linear gains requires positive, Hebbian intrinsic plasticity to produce and maintain a lognormal distribution of gains. We also could show that the observed widths of the distributions, i.e. the differences for σ* between W, R and G, naturally result from the network structure and are built into the system simply by Hebbian adaptation.
Lognormal distributions may arise as stable properties of the system during early development (the set-up of the system), i.e. before actual pattern storage or event memory develops, and they are maintained during processing by a Hebbian type of positive adaptation events. Homeostatic plasticity consists in downregulating gains or weights with increases in firing rates. Purely homeostatic learning results in normal distributions, and erases existing lognormal distributions. By combining homeostatic and Hebbian adaptation we can achieve and maintain stable lognormal distributions.
A lognormal distribution means that values are normally distributed on a logarithmic scale. From an engineering perspective, basic Hebbian plasticity for synapses and intrinsic properties is sufficient to generate stable logarithmic distributions. If there is random variation of multiplicative events, as in Hebbian plasticity, a lognormal distribution will be the result54.
This is related to principles of sensory coding, where logarithmic scale signal processing enhances perception of weak signals, while also being able to respond to large signals - effectively increasing the perceptual range compared to linear coding14. In an interconnected network logarithmic coding may turn into a property for the access of representations. Feature clusters, or event traces could be accessed by targeted connections to the top-level neurons, which then activate lower level neurons in their immediate vicinity. By accessing high frequency neurons preferentially, a whole feature area can be reached, and local diffusion will provide any additional computation. Similarly, the results of a local computation can be efficiently distributed by high frequency neurons to other areas. Fast point-to-point communication using only high frequency neurons may be sufficient for fast responses in many cases. Scale-free networks in general support synchronization, which is also a useful feature for rapid information transfer and access64.
Recently, publications65,66 have shown that there is indeed a difference between high-frequency and low-frequency neurons in their connectivity: high-frequency neurons have short delays, strong connections, and directed targets, while low-frequency neurons have long delays, weak connections and diffuse targets.
The lognormal distribution of spike rates has significant implications for neural coding. Logarithmic spike rates are coupled with linear variance for responses to behavioral stimulation. In other words, the greatest part of the coding results already from the frequency rank of the neuron itself, such that high frequency neurons have the largest impact. A fixed mean rate for each neuron allows stable expectation values for network computations.
Logarithmic, hierarchical coding does not need to be sparse. The low frequency neurons may matter the most in terms of input response. With lognormal synaptic weight distributions, if strong synapses are kept stable, they may transmit an input neuron’s mean firing rate to targets and in this way provide stability to the system. All other synapses could be arbitrary. This would allow for continued pattern learning to be implemented by the bulk of low weight synapses, while the framework of neuronal interactions, e.g., the ensemble structure, could be unchanged. Such a division of labor between strong synapses and weaker ones could have many advantages in a complex, modular network.
Experimental data have often shown that sampling of neuronal responses from a large population (105 or more neurons), which become activated at 30% or more, yields accuracy for a stimulus already for small samples (100–200 neurons or 1–2%) (e.g., 67). We may suggest that this happens when we sample from a highly modular structure, and we have been able to replicate the effect with lognormal networks68.
In our earlier work1, we found that intrinsic excitability manifested by spike response to current injection and rheobase in vitro for dorsal striatal and nucleus accumbens neurons seems to have the same distribution as the firing rate in cortex under in vivo conditions. Approximately at the same time6, had observed a heavy-tailed distribution of synaptic weights in cortical tissue.
In this article, we have done three things: (a) collected data to show that rate, weight and gain distributions in different brain areas all follow a heavy-tailed, specifically a lognormal distribution; (b) created a generic neural network model to show that these distributions arise from Hebbian learning, and specifically that intrinsic plasticity must be Hebbian as well; and (c) shown that the width of the distributions, as experimentally attested, arise naturally from the network structure and the role of its components, in a very robust way. We have also discussed what the lognormal distribution means for neural coding: a division of labor between fast transmission by high-frequency neurons and low-level computation by low-frequency neurons in a modular structure, and possibly a division of labor between stable components (strong synapses, high-frequency neurons) and more variable components (weak synapses, low-frequency neurons).
The GNN simulation software was programmed in Matlab, and is available in GitHub: https://github.com/gscheler/GNN/tree/v0.1
Archived source code as at time of publication: https://doi.org/10.5281/zenodo.82994949
OSS approved license: Apache 2.0.
All the data required for re-analysis of the study have been referenced throughout the manuscript.
Views | Downloads | |
---|---|---|
F1000Research | - | - |
PubMed Central
Data from PMC are received and updated monthly.
|
- | - |
Competing Interests: No competing interests were disclosed.
Is the work clearly and accurately presented and does it cite the current literature?
Yes
Is the study design appropriate and is the work technically sound?
Yes
Are sufficient details of methods and analysis provided to allow replication by others?
Yes
If applicable, is the statistical analysis and its interpretation appropriate?
Partly
Are all the source data underlying the results available to ensure full reproducibility?
Yes
Are the conclusions drawn adequately supported by the results?
Partly
Competing Interests: No competing interests were disclosed.
Is the work clearly and accurately presented and does it cite the current literature?
Partly
Is the study design appropriate and is the work technically sound?
Yes
Are sufficient details of methods and analysis provided to allow replication by others?
Yes
If applicable, is the statistical analysis and its interpretation appropriate?
Partly
Are all the source data underlying the results available to ensure full reproducibility?
Yes
Are the conclusions drawn adequately supported by the results?
Partly
Competing Interests: No competing interests were disclosed.
Alongside their report, reviewers assign a status to the article:
Invited Reviewers | ||
---|---|---|
1 | 2 | |
Version 2 (revision) 11 Oct 17 |
read | |
Version 1 25 Jul 17 |
read | read |
Provide sufficient details of any financial or non-financial competing interests to enable users to assess whether your comments might lead a reasonable person to question your impartiality. Consider the following examples, but note that this is not an exhaustive list:
Sign up for content alerts and receive a weekly or monthly email with all newly published articles
Already registered? Sign in
The email address should be the one you originally registered with F1000.
You registered with F1000 via Google, so we cannot reset your password.
To sign in, please click here.
If you still need help with your Google account password, please click here.
You registered with F1000 via Facebook, so we cannot reset your password.
To sign in, please click here.
If you still need help with your Facebook account password, please click here.
If your email address is registered with us, we will email you instructions to reset your password.
If you think you should have received this email but it has not arrived, please check your spam filters and/or contact for further assistance.
Comments on this article Comments (0)