Highly Heterogeneous Excitatory Connections Require Less Amount of Noise to Sustain Firing Activities in Cortical Networks

Cortical networks both in vivo and in vitro sustain asynchronous irregular firings with extremely low frequency. To realize such self-sustained activity in neural network models, balance between excitatory and inhibitory activities is known to be one of the keys. In addition, recent theoretical studies have revealed that another feature commonly observed in cortical networks, i.e., sparse but strong connections and dense weak connections, plays an essential role. The previous studies, however, have not thoroughly considered the cooperative dynamics between a network of such heterogeneous synaptic connections and intrinsic noise. The noise stimuli, representing inherent nature of the neuronal activities, e.g., variability of presynaptic discharges, should be also of significant importance for sustaining the irregular firings in cortical networks. Here, we numerically demonstrate that highly heterogeneous distribution, typically a lognormal type, of excitatory-to-excitatory connections, reduces the amount of noise required to sustain the network firing activities. In the sense that noise consumes an energy resource, the heterogeneous network receiving less amount of noise stimuli is considered to realize an efficient dynamics in cortex. A noise-driven network of bi-modally distributed synapses further shows that many weak and a few very strong synapses are the key feature of the synaptic heterogeneity, supporting the network firing activity.


INTRODUCTION
Intrinsic neuronal activities in the cortical network are inherently noisy (Tomko and Crapper, 1974;Softky and Koch, 1993;Shadlen and Newsome, 1994;Faisal et al., 2008;Stiefel et al., 2013). The so-called spontaneous irregular firings have been characterized by extremely low firing frequency (Hromádka et al., 2008;Mizuseki and Buzsáki, 2013), high-amplitude membrane potential fluctuations (Wilson and Kawaguchi, 1996;Destexhe et al., 2001), persistent UP (depolarized) state of membrane potential (Steriade et al., 2001;Destexhe et al., 2003;Shu et al., 2003a), and sensitivity to perturbations (London et al., 2010). They have been observed in cortical cell cultures (Gross et al., 1982;Plenz and Aertsen, 1996;Marom and Shahaf, 2002), brain slices (Mao et al., 2001;Shu et al., 2003b), and in vivo (Timofeev et al., 2000) even in the absence of external stimuli. Such randomness is considered to play a key role in various computations in the cortex, ranging from sensory perception (Arieli et al., 1996;Tsodyks et al., 1999), working memory (Fuster, 1995;Wang, 2002;Compte, 2006), dynamical switching of cortical states (Kenet et al., 2003) to information propagation (Destexhe and Contreras, 2006;Kumar et al., 2010). Although detailed cortical microcircuitry is yet to be explored, theoretical models have been developed to elucidate the electrophysiological mechanisms that underlie the spontaneous firing. The modeling approaches can be roughly classified into two types: externally driven network dynamics vs. self-sustained network dynamics. In the externally driven network, individual neurons as well as their network state are quiescent when no external input is provided. The network dynamics is activated by external stimuli such as background noise inputs, which may represent, e.g., variability of presynaptic discharges of individual cells (Destexhe and Contreras, 2006). Alternatively, existence of a certain portion of endogenously active cells within a network of quiescent cells may also act as external input to maintain the ongoing firing state (Latham et al., 2000a,b). In the modeling type of selfsustained network dynamics, on the other hand, interaction between a network of cortical neurons plays an essential role for maintaining the intrinsic firing activities. In random networks of leaky integrate-and-fire neurons, balance between excitatory and inhibitory activities is shown to be the key to sustain asynchronous irregular dynamics (van Vreeswijk and Sompolinsky, 1996;Brunel, 2000) even without external stimuli (Vogels and Abbott, 2005;Kumar et al., 2008). Modeling the statistical features of synaptic strengths of cortical neurons (Song et al., 2005;Lefort et al., 2009;Avermann et al., 2012;Buzsáki and Mizuseki, 2014) enabled asynchronous irregular firing in a biologically plausible parameter range. Teramae et al. (2012) studied a network of leaky integrate-and-fire neuron model with lognormally distributed excitatory postsynaptic potentials (EPSPs) and showed analytically that low-frequency firings of few Hz can be generated without external stimuli. Ikegaya et al. (2013) observed lognormal distribution in the amplitudes of unitary excitatory postsynaptic conductance in rat hippocampal CA3 pyramidal cells. Lognormally distributed excitatory-to-excitatory synapses, fitted from the experimental data, were implemented into the recurrent network model, which realized long-tailed distribution of firing rates and infrequent spontaneous firings (< 2 Hz) without external stimuli. Kriener et al. (2014) studied a network of strong excitatory-to-excitatory synapses and showed that bistability of quiescent state and moderate firing state is the key determinant for the onset and the lifetime of self-sustained activity states.
As a combined situation of the two types, cooperative dynamics between the network of long-tailed synaptic distribution and the external noise stimuli has not yet been thoroughly investigated. Kriener et al. (2014) simulated external Poisson processes to drive such network and reported an amplification of the input spike correlations and the spike irregularity, leading to fluctuations of a large population of neurons. They, however, focused on the noise input not as the main source of sustaining the network dynamics. Of particular interest in the present study is that the noise may enhance subthreshold state of the intrinsically active network, leading to the onset of self-sustained neuronal firings. Since the noise requires an energy resource (Laughlin et al., 1998;Attwell and Laughlin, 2001;Lennie, 2003), it is biologically more efficient to maintain the network firing activities with less amount of noise. To distinguish the present approach from the preceding studies (Vogels and Abbott, 2005;Kumar et al., 2008;Teramae et al., 2012;Ikegaya et al., 2013;Kriener et al., 2014), which discussed spontaneous firings without assuming any external stimulus, we refer to low-frequency irregular firings observed in our noise-driven system as "network firing activities." We numerically detect the minimal level of noise that gives rise to the network firing activities and compared such levels between networks of lognormally and normally distributed EPSPs. Our study shows that the noise required to generate the network firings is strongly reduced in the network of lognormally distributed EPSPs. Examination of the network of bi-modally distributed EPSPs further clarifies that essential mechanism of the network can be determined only by a pair of one weak and one strong EPSP values and their balance. Our results suggest a strong advantage of the heterogeneous excitatory-to-excitatory connections that realize robust and efficient operation of the cortical network.
The rest of the paper is organized as follows. In section 2, a network of integrate-and-fire neurons is introduced as a mathematical model for the cortical network. Quantity to measure the level of synchronized firings is also introduced. In section 3, the effect of dynamical noise on the network of lognormally distributed EPSPs is studied in comparison to a network of normally distributed EPSPs. By introducing Bernoulli distribution of binary variables, which represent weak and strong synapses, to the noise-driven neural network, the key statistical feature is highlighted for the heterogeneous distribution of the EPSPs. The final section is devoted to our results summary and discussions on future problems of the cortical networks.

Model of a Single Neuron
The following conductance-based leaky integrate-and-fire neuron was utilized to represent the dynamics of individual neuron where v stands for membrane potential, τ m is a membrane time constant, and V L , V E , and V I denote reversal potential of leak, excitatory, and inhibitory postsynaptic currents, respectively. The constant values were set to τ m = 20 ms for excitatory neurons, τ m = 10 ms for inhibitory neurons, V L = −70 mV, V E = 0 mV, and V I = −80 mV. The excitatory and inhibitory synaptic conductance normalized by the membrane capacitance, g E and g I , evolves as follows: where the indices X = E and X = I are for excitatory and inhibitory conductance, respectively. δ(t) represents the delta function, G X,j , d j , and s j are synaptic weight, delay, and spike timing of synaptic input from the j-th neuron, respectively. The decay time constant τ s was set to 2 ms for both excitatory and inhibitory conductance. The synaptic delays d j obey uniform distribution that ranges from d 0 − 1 to d 0 + 1 [ms]. The mean synaptic delay was set to d 0 = 2 ms for excitatory-to-excitatory connections and d 0 = 1 ms for other connections. The threshold value for spike generation was set to V thr = −50 mV, where v was reset to V r = −70 mV after the spiking. The refractory period was set to 1 ms. These parameter setting is based on the one use by Teramae et al. (2012).

Organization of Cortical Network Model
The network model consisted of 10, 000 excitatory neurons and 2, 000 inhibitory neurons ( Figure 1A). Based on the physiological measurements of Song et al. (2005), probability that a pair of excitatory neurons is bidirectionally connected was set to P bi = 0.0542, while probability that a pair is unidirectionally connected was set to P uni = 0.123. For excitatory-to-excitatory connections, their synaptic weights G E,j∈E were randomly generated such that the amplitudes of EPSPs x measured from the resting membrane potential obey the following lognormal distribution µ L and σ L represent mean and standard deviation of the variable's natural logarithm. As the parameter to control variability of the EPSPs, the value of σ L was varied under the condition that the mean EPSP remained the same, i.e., µ L + σ 2 L /2 = log(0.9). The case of σ 2 L = 1.0 reproduces the long-tailed EPSP distribution observed in the experiment of Song et al. (2005). When generating the synaptic weights, any unrealistic value G E,j∈E that gave EPSP amplitude larger than 20 mV was discarded and we selected another value from the distribution.
Following the former study of Teramae et al. (2012), a constant value of G E,j∈I = 0.018 was used for the excitatoryto-inhibitory connections. Since the inhibitory-to-inhibitory synaptic weight was set to be relatively small by Teramae et al. (2012), the weight value was modified as G I,j∈I = 0.018. The corresponding postsynaptic potentials, which were measured from the resting membrane potential of −70 mV (EPSP) and −55 mV (IPSP), were 1.66 and −0.55 mV, respectively. They are within the range of experimentally measured postsynaptic potentials (Tamás et al., 1998;Avermann et al., 2012;Jiang et al., 2015).
It has been shown that lognormal distribution of inhibitoryto-excitatory connections plays an important role of suppressing synchronized neuronal firings especially in a high-frequency firing state (Kada et al., 2016). This setting is consistent with the physiological experiments reporting that inhibitory postsynaptic potentials (IPSPs) are indeed highly heterogeneous in the cortex (Miles and Wong, 1984;Holmgren et al., 2003;Chapeton et al., 2012). Following these studies, the synaptic weights G I,j∈E were generated so that the corresponding IPSPs on excitatory neurons were lognormally distributed as p( The mean and standard deviation of the variable's natural logarithm were set as µ I = log(0.52) − σ 2 I /2 and σ I = 1.25, respectively. Any unrealistic value of G I,j∈E that gave IPSP amplitude larger than 30 mV was discarded and another value was selected from the distribution.
The connection probabilities of excitatory-to-inhibitory, inhibitory-to-excitatory and inhibitory-to-inhibitory neurons were P EI = 0.1157, P IE = 0.3, and P II = 0.32, respectively. Excitatory-to-excitatory synaptic transmissions failed with a rate p E = b/(b + EPSP) (b = 0.1). This formula has been developed by Teramae et al. (2012) to model the experiment of Lefort et al. (2009), who reported that trial-to-trial variability of largeamplitude EPSPs is very low compared with highly variable responses observed at small-amplitude synaptic connections.
It has been known that presynaptic discharges of individual cells are variable in time and can be represented as background noise inputs (Tomko and Crapper, 1974; Softky and Koch, 1993; Shadlen and Newsome, 1994;Destexhe and Contreras, 2006;Faisal et al., 2008). To take into account such noisy inputs, external Poisson spike trains were added to all excitatory neurons. The Euler's method was used to integrate the differential Equations (1),(1) with a time step of 0.01 ms.

Index of Synchronized Firings
We quantified the level of synchronization between spike trains based on the cross-correlogram (CCG). The cross-correlogram CCG has been a standard method for detecting spike synchrony (Logigian et al., 1988;Gray et al., 1989;Datta and Stephens, 1990;Bremner et al., 1991;Engel et al., 1991;Nordstrom et al., 1992). First, from 10, 000 excitatory neurons, 100 neurons were randomly selected. Then, the CCG was drawn as a histogram of spike-time differences (bin size: 1 ms, time lag: ±20 ms) for all pairs of the 100 neurons. In accordance with our previous study (Kada et al., 2016), we define the synchronization index as a normalized height of the peak, where M and A are the maximum and the average of CCG, respectively. The present index quantifies the level of synchrony, since peaks in the CCG indicate occurrences of synchronized firings.
As a quantity to measure distance as well as synchrony between spike trains, various methods have been developed (Victor and Purpura, 1996;van Rossum, 2001;Quian Quiroga et al., 2002;Aronov, 2003;Kreuz et al., 2007Kreuz et al., , 2009). To validate our approach, the CCG-based synchronization index was compared with the event synchrony (Quian Quiroga et al., 2002), which is one of the simple yet well-established methods to quantify synchronized firings.
As a test data-set, we generated 50 independent Poisson spike trains with a time-interval of 20 s. Then, another Poisson spike train was generated and included in the 50 spike trains as the common spikes. Frequency of the common spikes ranged from 0 to 9 Hz, while that of the individual spike trains was set to be 10 Hz. Figure 1B shows the results of applying our synchronization index and the event synchrony to these artificial data. As the frequency of the common spikes was increased, the level of synchrony monotonously increased for both CCG-based synchronization index and the event synchrony. The error-bars, obtained from 10 realizations of different data sets, were relatively small. Correlation coefficient between the CCG-based index and the event synchrony was r = 0.872 (p < 10 −4 ), indicating that the present index provides a reliable measure for detecting the synchronized firings.

Noise-Induced Network Firing in the Network of Lognormally Distributed EPSPs
We first examined the effect of external Poisson spike trains on the cortical network model. The EPSP amplitudes between excitatory neurons were lognormally distributed with σ L = 1.0.
To activate the network firing state, relatively strong Poisson spikes with a frequency of 1 Hz were applied during the initial duration of 100 ms. After 100 ms, frequency of the external spike inputs was lowered to 0.05 Hz. The results are shown in the raster plots of Figures 2A-E. First, the network firings were activated by the initial application of strong external inputs. After the initial transients, the firing activities were eventually weakened and the firing level was lowered to that of the external noise. The duration, for which the high-frequency network firings last, was variable, depending upon the random initial conditions (note that the time intervals are different in Figures 2A-E). For high-frequency network firing and low-frequency quiescent states, their average firing frequencies of the excitatory neurons were comparable among different initial conditions ( Figure 3A).
Next, frequency of the external spike inputs, which were given after the initial strong inputs, was increased to 0.09 Hz. The duration of the network firings was lengthened. Among 5 random initial conditions, 3 of them sustained network firing activities for more than 10 s and only 2 switched to low-frequency quiescent states before 10 s. In the case that the frequency of the external spikes was further increased to 0.2 Hz, the network firings became stable. For all 5 random initial conditions, the network firing activities lasted more than 10 s.
For the external spike inputs of 0.05 , 0.09 , and 0.2 Hz, average firing frequencies of the excitatory neurons during the quiescent states were slightly above those of the external inputs (see Figure 3). During the network firing activities, the average firing frequencies were around 1.5 Hz, which were independent of both initial conditions and frequency of the external inputs. The fact that these firing frequencies were much higher than those of the external inputs implies that the high-frequency activities are due to intrinsic network dynamics. The external noise merely supported the base-line level of the network activity.

Bistability of Low and High Frequency States
To further study the effect of external Poisson inputs on the network of lognormally distributed EPSPs, dependence of the firing frequency of the excitatory neurons on the frequency of noise inputs was computed in more detail. Two settings of the standard deviations, i.e., σ L = 0.90, 0.95, were examined. In each setting, frequency of the external noise was initially set to 0 Hz. During the network simulation, the noise frequency was increased by 0.02 Hz every one second until it reached to 0.28 Hz. The noise frequency was then decreased by 0.02 Hz every one second until it reached to 0 Hz. For both increasing and decreasing frequency of the external noise, average firing frequency of the excitatory neurons was plotted in Figures 4A,B. We see a clear transition between low-frequency quiescent state and high-frequency network firing. The low-frequency states coincide with the base-line frequencies of the noise input (dashed black line), whereas the high-frequency states are clearly distinguished from them. Bistability of the two states is indicated by the hysteresis region. This is consistent with Kriener et al. (2014), who showed that saddle-node bifurcation gives rise to bistability of quiescent state and moderate firing state in a network of heavy tailed distribution of excitatory-to-excitatory synapses.
To further study the existence domain of high-frequency network firing, its life-time was computed following the procedure of Kriener et al. (2014). For each setting of the input noise frequency, strong noise with a frequency of 0.3 Hz was injected during initial 10 ms to induce high-frequency network firings. Then, the noise frequency was lowered to the setting level and the system was ran for 2 s. For different initial conditions and different realizations of the noise processes, this free-run was repeated for 30 times. From the 30 time traces, their average firing rate r(t) was obtained with a sampling time interval of 1.5 s. The averaged time trace was fitted to an exponential decay function as r(t) = A exp(−t/λ), where the estimated life-time was given by λ. As shown in Figure 4C, the life-time grew rapidly as the frequency of the external noise was increased. The critical point of the noise frequency, above which the high-frequency network firings became stable, was lowered as the standard deviation of the lognormally distributed EPSPs was increased from σ L = 0.90 to 0.95.
To compare the lognormal EPSP distribution with Gaussian EPSP distribution, we simulated a network with normally distributed EPSPs N(µ G , σ 2 G ). Standard deviation of the Gaussian distribution was varied as σ G = 4.0, 6.0, 8.0, 10.0, while the averaged EPSP was set to be the same as that of the lognormally distributed EPSPs in the following manner. To generate realistic EPSP values, the EPSP amplitudes were restricted between 0 and 20 mV. For a given σ G , the mean parameter µ G was determined in such a way that the probability density function p G ( 0≤x≤20) yields an average EPSP equal to that of the lognormal EPSP distribution. To randomly generate EPSP values within a range between 0 and 20 mV, acceptance-rejection method (Neal, 2003) was utilized. Figure 4D shows the results. As the frequency of the noise inputs was increased, the average firing frequency of the excitatory neurons monotonously increased. In the four settings of the normally distributed EPSPs, the average firing frequency lay on the level of input noise up to 0.035 Hz. No significant frequency jump was observed. Even after the onset of network firing state, deviation of the average firing frequency from that of the input noise was rather minor. Between increasing and decreasing directions of the noise frequency, no hysteresis was observed.

Comparison of Onset Noise Frequency
Next, we studied onset frequency of the external noise, that is required to induce network firing state, and its dependence on the EPSP distributions. The onset point was determined as follows. First, distribution of the EPSPs and frequency of the input noise were fixed. Then, the network was simulated by supplying a strong external noise (frequency of 1 Hz) during initial duration of 100 ms. After 100 ms, frequency of the external noise was lowered to the setting level and the system was ran for 10 s. From time interval between 500 ms and 10, 100 ms, average firing frequency of the excitatory neurons was computed. This simulation was repeated for 5 different realizations of the random initial conditions and the input noise processes. Second, for each setting of the EPSP distribution, the noise frequency was increased from 0 to 0.3 Hz. The onset frequency was detected at the point, where, for all 5 realizations, the average firing frequency became larger than that of the input noise by 0.08 Hz and standard deviation of the average frequencies of the 5 realizations became less than 0.03 Hz. Figure 5A displays dependence of the onset noise frequency on the standard deviation σ L of the lognormally distributed EPSPs. As the standard deviation σ L of the lognormal distribution is increased, the onset noise frequency continues to decrease and it reaches to a value less than 0.1, that is about three times smaller than that of σ L = 0.8. Figure 5C, on the other hand, shows dependence of the onset noise frequency on the standard deviation σ G of the normally distributed EPSPs. Compared to the case of lognormal distribution, the onset noise frequency is much higher in the case of normal distribution. This indicates that much less amount of noise is needed to induce network firing in a network of lognormally distributed EPSPs.
As another quantity of the network dynamics, the synchronization indices corresponding to Figures 5A,B were shown in Figures 5C,D, respectively. In both cases, the synchronization index was kept within a low level (SI < 0.05), indicating that asynchronous firings have been realized in this region. It has been shown by Kada et al. (2016) that heterogeneity in the inhibitory-to-excitatory connections plays a key role in suppressing synchronized firing activities in a cortical network. The lognormal distribution, introduced to the inhibitory-to-excitatory connections, functioned efficiently to maintain the desynchronized firing dynamics.

Bernoulli Distribution
In the previous subsection, the network of lognormally distributed EPSPs has been shown to require much lower frequency of noise to induce network firing activities, compared to that of normally distributed EPSPs. This result implies that modest heterogeneity of EPSP amplitudes, i.e., the Gaussian distribution, is not efficient for giving rise to network firings, while the highly heterogeneous EPSP distribution, such as the lognormal distribution, works efficiently to support the firing state with less amount of noise. Which statistical properties of the lognormal EPSP distribution are essential for maintaining the network firing activities? One of the key feature of the lognormal distribution is that it has a long-tailed distribution with many weak and a few extremely strong synapses. As another type of such distribution, we introduce Bernoulli distribution of binary EPSPs. For two values of EPSP amplitudes a, b (a < b), their probabilities are defined as P a = p and P b = 1 − p, respectively. Mean and standard deviation of the binary EPSPs are given by m = ap + b(1 − p) and σ B = p(1 − p)(b − a) 2 , respectively.
First, the effect of external Poisson spike trains on the network of binary EPSPs was examined. By increasing and decreasing the external noise frequency, average firing frequency of the excitatory neurons was drawn in the hysteresis plots of Figures 6A,B. The value for large EPSP amplitude was set to b = 9 mV, whereas its probability was set to P b = 0.015 ( Figure 6A) and P b = 0.020 ( Figure 6B). For the two settings, the mean EPSP was set to be the same as m = 0.9 mV. In both settings of P b , we see a clear transition between low-frequency quiescent state and high-frequency network firings. The low-frequency states lie on the base-line frequencies of the noise input (dashed black line), whereas the high-frequency states are distinguished from them. The hysteresis region indicates bistability of the two states. These features agree quite well with those of the network having lognormally distributed EPSPs. Figure 6C displays dependence of the onset noise frequency on the probability P b of large EPSP. As the probability P b is increased, the onset noise frequency continues to decrease and reaches to a value close to 0.1 Hz. This indicates that a larger number of large EPSP amplitudes is quite helpful to give rise to network firing, when the external noise input is rather weak. On the other hand, the synchronization indices of Figure 6D show that the level of synchrony reaches to a high level (SI∼0.35) as the probability P b is increased. Too many strong synapses may create hubs in the complex network of neurons, where spike generation from such hubs leads to synchronized firings of the connected neurons. Thus, to realize asynchronous network firing, too many strong synapses may not be desired. Taking into account both the onset noise frequency and the synchronization indices, not too large, not too small, but intermediate number of large EPSPs is needed to maintain the asynchronous network firing state.
To see the generality of the present analysis, dependence of the optimal configuration of the Bernoulli (binary) distribution on the value of large EPSP amplitude b was further examined. For three settings of the large EPSP amplitudes, i.e., b = 9 mV (red), 10 mV (green), and 11 mV (blue), the probability P b was varied, while the mean EPSP was kept to be the same as m = 0.9 mV. As the probability P b was increased, the onset noise frequency decreased and reached to a value close to 0.1 Hz or even lower ( Figure 7A). The corresponding synchronization indices of Figure 7B increased to a high level as the probability P b was increased. The same tendency as Figure 6 was therefore confirmed. The optimal probability of large EPSPs, at which the onset noise frequency reaches the bottom level and at the same time the synchronization index SI is minimized, was P b = 0.025, 0.02, and 0.015 for b = 9, 10, and 11 mV, respectively. As the EPSP took larger values, smaller number of large EPSPs is required to maintain asynchronous firing state.
To further study the effect of the mean EPSP, three settings were considered as m = 0.8 mV (red), 0.9 mV (green), and 1 mV (blue). The probability P b was varied, while the value of large EPSP amplitude was fixed to b = 9 mV. Again, as the probability P b was increased, the onset noise frequency decreased ( Figure 7C) and the corresponding synchronization indices increased ( Figure 7D). The optimal number of large EPSPs to maintain asynchronous firing state was lowered as the mean EPSP was increased.
The present analysis based on Bernoulli distribution of the EPSP amplitudes is therefore general in the sense that it does not depend upon the detailed setting of the binary distribution (b and m). Neither too large nor too small numbers of large EPSPs works. There exist an optimal number of large EPSPs, which efficiently sustain the network firing activities and at the same time suppress their synchronous firings. Although there exist various types of long-tailed distributions that can represent many weak and a few extremely strong EPSPs, the Bernoulli model, which represents one of the simplest distribution types, suggests that the essential mechanism of sustaining the network firing activity can be elucidated by only a pair of one weak and one strong EPSP values and their balance.

DISCUSSIONS
In this paper, cooperative dynamics that emerge from a network of highly heterogeneous synaptic connections and external noise inputs have been investigated. Heterogeneous distribution, typically a lognormal distribution, of excitatory-to-excitatory connection strengths have been reported experimentally (Song et al., 2005;Ikegaya et al., 2013). External noise stimuli can be thought of as representing the variability of presynaptic discharges to single cells (Destexhe and Contreras, 2006) or as endogenously active cells that stimulate the neighboring quiescent cells (Latham et al., 2000a,b). As the combined effect, the noise works efficiently to activate intrinsic network dynamics to give rise to asynchronous firings. Comparative study with Gaussian distribution as well as Bernoulli distribution revealed that weak-dense and sparse-strong connections are the key feature of the synapses that support the stable firing activity. Our finding is consistent with the theoretical study of the autonomous network of lognormal EPSP distribution, which was shown to be capable of sustaining spontaneous firings without external stimulus (Teramae et al., 2012;Kriener et al., Figure 6C (b = 9 mV), whereas green and blue dotted lines correspond to the cases of b = 10 mV and b = 11 mV, respectively. As the probability P b was varied, average of the EPSPs was set to be constant (m = 0.9 mV). (B) Synchronization index of the network dynamics corresponding to (A). The error bars indicate standard deviation of 5 realizations started from different random initial conditions. (C) Dependence of the onset noise frequency on the probability (P b ) of large value. Large EPSP value was set to b = 9 mV, whereas average of the EPSPs was kept as m = 0.9 mV (green dotted line), m = 0.8 mV (red solid line), and m = 1.0 mV (blue dotted line). (D) Synchronization index of the network dynamics corresponding to (C). 2014). The present study extends such idea to the case of non-autonomous network having external stimuli, where the long-tailed EPSP distribution reduces the noise level needed for the onset of network firings. In the sense that stochastic noise is ubiquitous in neural systems, this extension is of essential importance to incorporate the lognormal network to more realistic neurophysiological situation. Since synaptic noise requires an energy resource (Laughlin et al., 1998;Attwell and Laughlin, 2001;Lennie, 2003), it is biologically more efficient to maintain the firing activities with a few support from such expensive resource. The fact that the network of lognormally distributed EPSPs effectively reduced the noise level compared to that of normally distributed EPSPs indicates that highly heterogeneous distribution of excitatoryto-excitatory connection strengths can be advantageous for maintaining network firing activity in an efficient and robust fashion.
It should be noted that energy expenditure in the cortical system provides an intriguing issue. Not only the energy spent for spike generations (Laughlin et al., 1998;Attwell and Laughlin, 2001;Lennie, 2003;, resting potentials , and other electrical activities, but also metabolic energy spent for development of synapses (Karbowski, 2012) and their maintenance (Bezprozvanny and Hiesinger, 2013;Tononi and Cirelli, 2014) have been recently discussed. The advantage of having few but very strong synapses should be considered carefully by taking into account the maintenance cost of such strong synapses.
In the present study, the external noise inputs to individual neurons have been assumed to be independent from each other. Considering the real neurophysiology, where synaptic projections are overlapped among neurons and the neurons may receive common stimuli from the same external sources, it would be more realistic to assume a certain level of correlation in the noise inputs. It is therefore an important future study to deal with the case that the external noise is spatially correlated. How to separate synaptic noise from irregular firings that originate from intrinsic network dynamics provides also an important open problem to quantitatively examine our modeling study in experimental data.
Interesting application of the present study is to model the transition of cortical neurons between Up (depolarized) and Down (hyperpolarized) states. Stimulus-dependent transitions between the Up and Down states have been reported in cat visual cortex (Anderson et al., 2000). Single neuron membrane potentials in slice preparations also show spontaneous transition between them (Cossart et al., 2003;Shu et al., 2003b), where intrinsic network dynamics is considered to play an important role (Cossart et al., 2003). Such transition has been modeled as bistable Up and Down states, embedded in a network of excitatory and inhibitory neurons (Wilson and Cowan, 1972), where the switching can be triggered, e.g., by synaptic depression (Bazhenov et al., 2002). Synaptic noise can lead to random transitions between the basins of Up-Down states (Holcman and Tsodyks, 2006). It should be of great interest to examine how the lognormal network combined with the noise stimuli may model such Up-Down transitions.