Input nonlinearities can shape beyond-pairwise correlations and improve information transmission by neural populations

Joel Zylberberg and Eric Shea-Brown

Phys. Rev. E 92, 062707 – Published 9 December 2015

Abstract

While recent recordings from neural populations show beyond-pairwise, or higher-order, correlations (HOC), we have little understanding of how HOC arise from network interactions and of how they impact encoded information. Here, we show that input nonlinearities imply HOC in spin-glass-type statistical models. We then discuss one such model with parametrized pairwise- and higher-order interactions, revealing conditions under which beyond-pairwise interactions increase the mutual information between a given stimulus type and the population responses. For jointly Gaussian stimuli, coding performance is improved by shaping output HOC only when neural firing rates are constrained to be low. For stimuli with skewed probability distributions (like natural image luminances), performance improves for all firing rates. Our work suggests surprising connections between nonlinear integration of neural inputs, stimulus statistics, and normative theories of population coding. Moreover, it suggests that the inclusion of beyond-pairwise interactions could improve the performance of Boltzmann machines for machine learning and signal processing applications.

Received 14 December 2012
Revised 19 November 2015

DOI:https://doi.org/10.1103/PhysRevE.92.062707

Physics Subject Headings (PhySH)

Neural information

Information theory

Physics of Living Systems

Authors & Affiliations

Joel Zylberberg

Department of Applied Mathematics, University of Washington, Seattle, Washington 98195, USA

Eric Shea-Brown^*

Department of Applied Mathematics, Program in Neuroscience, Department of Physiology and Biophysics, University of Washington, Seattle, Washington 98195, USA

^*etsb@washington.edu

Article Text (Subscription Required)

Click to Expand

References (Subscription Required)

Click to Expand

Issue

Vol. 92, Iss. 6 — December 2015

Reuse & Permissions

Access Options

Article Available via CHORUS

Download Accepted Manuscript

Author publication services for translation and copyediting assistance advertisement

Images

Figure 1
Encoding model. (a) Each model neuron has a bias $h_{i}$ (blue) determined by external stimuli. Recurrent pairwise ( $J_{i j}$ , red) and triplet ( $γ_{i j k}$ , green) interactions further modify the output statistics. Illustrative schematic shown for $N = 3$ neurons; models can have arbitrary $N$ . (b) The firing rate (spiking probability) of each neuron varies sigmoidally with the strength of its input $(x_{i} = h_{i} + \sum_{j \neq i} J_{i j} σ_{j} + \sum_{j < k; j, k \neq i} γ_{i j k} σ_{j} σ_{k})$ ; see text. The steepness of the sigmoid depends on $β$ .
Reuse & Permissions
Figure 2
Stimuli considered in this work. (a) Marginal histogram of one stimulus, $h_{1}^{s}$ , for the jointly Gaussian ensemble. (b) One of the natural luminance images used in this work, from the database of Tkačik et al. [36]. The marginal histogram of pixel values for this set of images (normalized to have zero mean and unit variance) is skewed (c). To generate our naturalistic stimulus ensemble, we randomly draw dectuplets of pixel values with spacing $d$ pixels by placing the template (d) at a random location on a randomly chosen image, and setting stimulus values $h_{i}^{s}$ to match the pixel values falling under each square. To maintain permutation symmetry of our stimulus ensemble, we permute which square corresponds to which stimulus index $i$ for each draw. The marginals [(a),(c)] are the same for all stimulus indices $i$ .
Reuse & Permissions
Figure 3
Triplet interactions can improve encoding of stimuli. Panels (a),(c),(e) show results for Gaussian-distributed stimuli, whereas panels (b),(d),(f) are for natural image pixel stimuli, which have skewed distributions. (a) For jointly Gaussian stimuli with pairwise correlation coefficients of 0.95 (dashed lines) or 0 (solid lines), encoders with triplet interactions allowed (green) or forbidden (red, $γ = 0$ ) have the same coding performance, which increases with neural reliability $β$ . (b) For natural image stimuli with pixel spacings of $d = 32$ (solid lines) or $d = 2$ (dashed lines), the triplet-allowed encoder (green) performs better. The shaded regions (similar in thickness to the lines) around the lines in panels (a),(b) show the standard deviation of the mean MI over five repeats of the optimization procedure with different sets of random stimuli. (c),(d) To summarize how performance gains vary with correlation level, we plot the ratio of the MI for the optimal triplet-allowed networks $(M I_{3})$ to the one for triplet-forbidden networks $(M I_{2}, γ = 0)$ as a function of $β$ . The darkest curve corresponds to the largest correlation $[ρ = 0.95$ for the Gaussian (c) and $d = 2$ for the natural image stimuli (d)], the lightest curve corresponds to the smallest correlation ( $ρ = 0$ for the Gaussian and $d = 32$ for the natural image stimuli), and intermediate shades correspond to intermediate levels of correlation. (e),(f) For $β = 1.5$ , we similarly plot the performance ratio as a function of mean firing rate for Gaussian (e) and natural image (f) stimuli, in cases with constrained firing rates (see text). In order to make a fair comparison, we estimate the MI of the triplet-allowed and triplet-forbidden networks at the same firing rate (see Appendix pp3 for details). $N = 10$ neurons for all cases.
Reuse & Permissions
Figure 4
Triplet interactions, when they are beneficial, sparsify the neural representation of stimuli. (a) For jointly Gaussian stimuli $(ρ = 0.95)$ and no firing rate constraint, optimal encoders have no triplet interaction ( $γ$ , green). At low $β$ , these optimal encoders have pairwise interactions ( $J$ , red) that enhance the input correlations, whereas at high $β$ they oppose them [14]. The optimal biases $h^{0}$ (blue) cancel the effective mean-field bias from pairwise interactions, $h_{eff} = (N - 1) J 〈σ〉$ . (b) For the natural image pixel stimuli $(d = 2)$ , the optimal encoders have negative triplet interactions (when triplet interactions are allowed: solid lines) for all $β$ —the encoder parameters do not change sign. When triplet interactions are forbidden (dashed lines), the magnitudes of the pairwise interaction and bias are smaller, and do change sign with increasing $β$ . The solid horizontal line (at zero) is to guide the eye. The shaded regions (similar in thickness to the lines) around the lines in panels (a),(b) show the standard deviation of the mean parameter values over five repeats of the optimization procedure with different sets of random stimuli. For other levels of stimulus correlation, we find qualitatively similar results (not shown). A comparison of the response distributions of the optimal encoders for the natural image pixel stimuli $(d = 2)$ with triplet interactions allowed (c) or forbidden [(d), $γ = 0]$ shows that the triplet interactions sparsify the responses by reducing the probability of the state in which all neurons are active.
Reuse & Permissions
Figure 5
Negative triplet interactions are better at suppressing the all-on state than are negative pairwise interactions. The plots here show the function $C = J α (α - 1) / 2 + γ α (α - 1) (α - 2) / 6$ for non-negative integer values of $α \in [0, 10]$ . For $J = 0.75$ and $γ = - 0.3$ (upper, pink curve), the states with many coactive neurons are suppressed by the recurrent interaction. For $J = - 0.75$ and $γ = 0.3$ (lower, brown curve), the opposite is true.
Reuse & Permissions

Physical Review E

covering statistical, nonlinear, biological, and soft matter physics