Elsevier

NeuroImage

Volume 42, Issue 1, 1 August 2008, Pages 147-157
NeuroImage

Technical Note
Population dynamics: Variance and the sigmoid activation function

https://doi.org/10.1016/j.neuroimage.2008.04.239Get rights and content

Abstract

This paper demonstrates how the sigmoid activation function of neural-mass models can be understood in terms of the variance or dispersion of neuronal states. We use this relationship to estimate the probability density on hidden neuronal states, using non-invasive electrophysiological (EEG) measures and dynamic casual modelling. The importance of implicit variance in neuronal states for neural-mass models of cortical dynamics is illustrated using both synthetic data and real EEG measurements of sensory evoked responses.

Introduction

The aim of this paper is to show how the sigmoid activation function in neural-mass models can be understood in terms of the dispersion of underlying neuronal states. Furthermore, we show how this relationship can be used to estimate the probability density of neuronal states using non-invasive electrophysiological measures such as the electroencephalogram (EEG).

There is growing interest in the use of mean-field and neural-mass models as observation models for empirical neurophysiological time-series (Wilson and Cowan, 1972, Nunez, 1974, Lopes da Silva et al., 1976, Freeman, 1978, Freeman, 1975, Jansen and Rit, 1995, Jirsa and Haken, 1996, Wright and Liley, 1996, Valdes et al., 1999, Steyn-Ross et al., 1999, Frank et al., 2001, David and Friston, 2003, Robinson et al., 1997, Robinson et al., 2001, Robinson, 2005, Rodrigues et al., 2006). Models of neuronal dynamics allow one to ask mechanistic questions about how observed data are generated. These questions or hypotheses can be addressed through model selection by comparing the evidence for different models, given the same data. This endeavour is referred to as dynamic causal modelling (DCM) (Friston, 2002, Friston, 2003, Penny et al., 2004, David et al., 2006a, David et al., 2006b, Kiebel et al., 2006). There has been considerable success in modelling fMRI, EEG, MEG and LFP data using DCM (David et al., 2006a, David et al., 2006b, Kiebel et al., 2006, Garrido et al., 2007, Moran et al., 2007). All these models embed key nonlinearities that characterise real neuronal interactions. The most prevalent models are called neural-mass models and are generally formulated as a convolution of inputs to a neuronal ensemble or population to produce an output. Critically, the outputs of one ensemble serve as input to another, after some static transformation. Usually, the convolution operator is linear, whereas the transformation of outputs (e.g., mean depolarisation of pyramidal cells) to inputs (firing rates in presynaptic inputs) is a nonlinear sigmoidal function. This function generates the nonlinear behaviours that are critical for modelling and understanding neuronal activity. We will refer to these functions as activation or input-firing curves.

The mechanisms that cause a neuron to fire are complex (Mainen and Sejnowski, 1995, Destexhe and Paré, 1999); they depend on the state (open, closed; active, inactive) of several kinds of ion channels in the postsynaptic membrane. The configuration of these channels depends on many factors, such as the history of presynaptic inputs and the presence of certain neuromodulators. As a result, neuronal firing is often treated as a stochastic process. Random fluctuations in neuronal firing function are an important aspect of neuronal dynamics and have been the subject of much study. For example, Miller and Wang (2006) look at the temporal fluctuations in firing patterns in working memory models with persistent states. One perspective on this variability is that it is caused by fluctuations in the threshold of the input-firing curve of individual neurons. This is one motivation for a sigmoid activation function at the level of population dynamics; which rests on the well-known result that the average of many different threshold functions is a nonlinear sigmoid. We will show the same sigmoid function can be motivated by assuming fluctuations in the neuronal states (Hodgkin and Huxley, 1952). This is a more plausible assumption because variations in postsynaptic depolarisation over a population are greater than variations in firing threshold (Fricker et al 1999): in active cells, membrane potential values fluctuate by up to about 20 mV, due largely to hyperpolarisations that follow activation. In contrast, firing thresholds vary up to only 8 mV. Furthermore, empirical studies show that voltage thresholds, determined from current injection or by elevating extracellular K+, vary little with the rate of membrane polarisation and that the “speed of transition into the inactivated states also appears to contribute to the invariance of threshold for all but the fastest depolarisations” (Fricker et al., 1999). In short, the same mean-field model can be interpreted in terms of random fluctuations on the firing thresholds of different neurons or fluctuations in their states. The latter interpretation is probably more plausible from neurobiological point of view and endows the sigmoid function parameters with an interesting interpretation, which we exploit in this paper. It should be noted that Wilson and Cowan (1972) anticipated that the sigmoid could arise from a fixed threshold and population variance in neural states; after Eq. (1) of their seminal paper they state: “Alternatively, assume that all cells within a subpopulation have the same threshold, … but let there be a distribution of the number of afferent synapses per cell.” This distribution induces variability in the afferent activity seen by any cell.

This is the first in a series of papers that addresses the importance of high-order statistics (i.e., variance) in neuronal dynamics, when trying to model and understand observed neurophysiological time-series. In this paper, we focus on the origin of the sigmoid activation function, which is a ubiquitous component of many neural-mass and cortical-field models. In brief, this treatment provides an interpretation of the sigmoid function as the cumulative density on postsynaptic depolarisation over an ensemble or population of neurons. Using real EEG data we will show that population variance, in the depolarisation of neurons in somatosensory sources generating sensory evoked potentials (SEP) (Litvak et al., 2007) can be quite substantial, especially in relation to evoked changes in the mean. In a subsequent paper, we will present a mean-field model of population dynamics that covers both the mean and variance of neuronal states. A special case of this model is the neural-mass model, which assumes that the variance is fixed (David et al., 2006a, David et al., 2006b, Kiebel et al., 2006). In a final paper, we will use these models as probabilistic generative models (i.e., dynamic causal models) to show that population variance can be an important quantity, when explaining observed EEG and MEG responses.

This paper comprises three sections. In the first, we present the background and motivation for using sigmoid activation functions. These functions map mean depolarisation, within a neuronal population, to expected firing rate. We will illustrate the origins of their sigmoid form using a simple conductance-based model of a single population. We rehearse the well-known fact that threshold or Heaviside operators in the equations of motion for a single neuron lead to sigmoid activation functions, when the model is formulated in terms of mean neuronal states. We will show that the sigmoid function can be interpreted as the cumulative density function on depolarisation, within a population.

In the second section we emphasise the importance of variance or dispersion by noting that a change in variance leads to a change in the form of the sigmoid function. This changes the transfer function of the system and its input–output properties. We will illustrate this by looking at the Volterra kernels of the model and computing the modulation transfer function to show how the frequency response of a neuronal ensemble depends on population variance.

In the final section, we estimate the form of the sigmoid function using the established dynamic causal modelling technique and SEPs, following medium nerve stimulation. In this analysis, we focus on a simple DCM of brainstem and somatosensory sources, each comprising three neuronal populations. Using standard variational techniques, we invert the model to estimate the density on various parameters, including the parameters controlling the shape of the sigmoid function. This enables us to estimate the implicit probability density function on depolarisation of neurons within each population. We conclude by discussing the implications of our results for neural-mass models, which ignore the effects of population variance on the evolution of mean activity. We use these conclusions to motivate a more general model of population dynamics that will be presented in a subsequent paper (Marreiros et al., manuscript in preparation).

Section snippets

Theory

In this section, we will show that the sigmoid activation function used in neural-mass models can be derived from straightforward considerations about single-neuron dynamics. To do this, we look at the relationship between variance introduced at the level of individual neurons and their population behaviour.

Saturating nonlinear activation functions can be motivated by considering neurons as binary units; i.e., as being in an active or inactive state. Wilson and Cowan (1972) showed that

Kernels, transfer functions and the sigmoid

In this section, we illustrate the effect of changing the slope-parameter (i.e., variance of the underlying neuronal states) on the input–output behaviour of neuronal populations. We will start with a time-domain characterisation, in terms of convolution kernels and conclude with a frequency-domain characterisation, in terms of transfer functions. We will see that the effects of changing the implicit variance are mediated largely by first-order effects and can be quite profound.

Estimating population variance with DCM

In this final section, we exploit the interpretation of the sigmoid as a cumulative density on the states, specifically the depolarisation. This interpretation renders the derivative of the sigmoid a probability density on the voltage: recall from the first sectionwp(xi)xi=S(μiw)p(xi)=S(xiμi).

This means we can use estimates of the slope-parameter, which specifies S′, to infer the underlying variance of depolarisation in neuronal populations (or an upper bound; see Appendix). In what

Epilogue

The preceding analysis assumes that the variance is fixed over peristimulus time. Indeed neural-mass models in general assume a fixed variance because they assume a fixed form for the sigmoid activation function. Neural-mass models are obliged to make this assumption because their state variables allow only changes in mean states, not changes in variance or higher-order statistics of neuronal activity. Is this assumption sensible?

In our next paper on population variance we will compare

Conclusion

In this paper, our focus was on how the sigmoid activation function, linking mean population depolarisation to expected firing rate can be understood in terms of the variance or dispersion of neuronal states. We showed that the slope-parameter ρ models formally the effects of variance (to a first approximation) on neuronal interactions. Specifically, we saw that the sigmoid function can be interpreted as a cumulative density function on depolarisation, within a population. Then, we looked at

Acknowledgments

The Portuguese Foundation for Science and Technology and the Wellcome Trust supported this work. We thank Marta Garrido and CC Chen for help with the DCM analyses and Vladimir Litvak for the MN-SSEP data.

Software note

Matlab demonstration and modelling routines referred to in this paper are available as academic freeware as part of the SPM software from http://www.fil.ion.ucl.ac.uk/spm (neural models toolbox).

References (51)

  • Lopes da SilvaF.H. et al.

    Models of neuronal populations: the basic mechanisms of rhythmicity

    Prog. Brain Res.

    (1976)
  • MalenkaR.C. et al.

    LTP and LTD: an embarrassment of riches

    Neuron

    (2004)
  • MoranR.J. et al.

    A Neural-mass Model of spectral responses in electrophysiology

    NeuroImage

    (2007)
  • NunezP.

    The brain wave equation: a model for the EEG

    Math. Biosci.

    (1974)
  • PennyW.D. et al.

    Comparing dynamic causal models

    NeuroImage

    (2004)
  • RodriguezR. et al.

    Noisy spiking neurons and networks: useful approximations for firing probabilities and global behavior

    Biosystems

    (1998)
  • RodriguesS. et al.

    On the genesis of spikewave activity in a mean-field model of human thalamic and corticothalamic dynamics

    Phys. Lett., A

    (2006)
  • WilsonH.R. et al.

    Excitatory and inhibitory interactions in localized populations of model neurons

    Biophys. J.

    (1972)
  • AllisonT. et al.

    Potentials evoked in human and monkey cerebral cortex by stimulation of the median nerve. A review of scalp and intracranial recordings

    Brain

    (1991)
  • BlissT.V. et al.

    Long-lasting potentiation of synaptic transmission in the dentate area of the anaesthetized rabbit following stimulation of the perforant path

    J. Physiol. (Lond.)

    (1973)
  • CookeS.F. et al.

    Plasticity in the human central nervous system

    Brain

    (2006)
  • DayanP. et al.

    Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems

    (2001)
  • DecoG. et al.

    Extended method of moments for deterministic analysis of stochastic multistable neurodynamical systems

    Phys. Rev., E

    (2007)
  • DestexheA. et al.

    Impact of network activity on the integrative properties of neocortical pyramidal neurons in vivo

    J. Neurophysiol.

    (1999)
  • FrankT.D. et al.

    Multivariate Ornstein–Uhlenbeck processes with mean-field dependent coefficients: application to postural sway

    Phys. Rev., E Stat. Nonlin. Soft. Matter Phys.

    (2001)
  • Cited by (136)

    • Complex regimes in electronic neuron-like oscillators with sigmoid coupling

      2022, Chaos, Solitons and Fractals
      Citation Excerpt :

      The sigmoid activation function means depolarisation within a neuronal population to expected firing rate. It can be interpreted as a cumulative density function on depolarization within a population [20]. Hyperbolic tangent function is classically used as a sigmoid function being a particular case of the more general Richard's curve [27]; in our case it was shifted and scaled to match the desired range [0, 1], as it is shown in the formula (2).

    View all citing articles on Scopus
    View full text