Perturbing low dimensional activity manifolds in spiking neuronal networks

Several recent studies have shown that neural activity in vivo tends to be constrained to a low-dimensional manifold. Such activity does not arise in simulated neural networks with homogeneous connectivity and it has been suggested that it is indicative of some other connectivity pattern in neuronal networks. In particular, this connectivity pattern appears to be constraining learning so that only neural activity patterns falling within the intrinsic manifold can be learned and elicited. Here, we use three different models of spiking neural networks (echo-state networks, the Neural Engineering Framework and Efficient Coding) to demonstrate how the intrinsic manifold can be made a direct consequence of the circuit connectivity. Using this relationship between the circuit connectivity and the intrinsic manifold, we show that learning of patterns outside the intrinsic manifold corresponds to much larger changes in synaptic weights than learning of patterns within the intrinsic manifold. Assuming larger changes to synaptic weights requires extensive learning, this observation provides an explanation of why learning is easier when it does not require the neural activity to leave its intrinsic manifold.


Non-orthogonal inside-manifold perturbations
In the derivation of inside-manifold perturbations, we assume thatQ is orthogonal. This corresponds to a rigid transformation of the manifold that can include rotations and mirrorings. If we allow for a more general linear transformation that may also include scaling and skewing of the manifold, the orthogonality argument no longer applies and we can no longer expect an inside-manifold perturbation to leave the weights approximately unchanged.

The 2-dimensional case
Consider a general inside-manifold perturbation matrix Q ∈ R 2×2 with elements given by Q = q 11 q 12 q 21 q 22 (S1) From Eq 17 in the manuscript, we expect that an inside-manifold perturbation would give a new weight matrixW = KQQ T Φ. The crucial step for this operation is the matrix QQ T , which in the 2-dimensional case becomes QQ T = q 2 11 + q 2 12 q 11 q 21 + q 12 q 22 q 11 q 21 + q 12 q 22 q 2 12 + q 2

22
(S2) This gives the perturbed matrix weights w ij = (q 2 11 + q 2 12 )k i1 φ 1j + (q 11 q 21 + q 12 to compare with unperturbed matrix weights The similarity betweenw ij and w ij is therefore heavily dependent on the choice of Q.

Elements drawn from a normal distribution
If the elements of Q are drawn from N (0, 1), the expectedw ij from Eq S3 becomes The expected weight change after an inside-manifold perturbation with a normally random matrix is thus a factor of 2. This can also be seen directly from the fact that the expected matrix in Eq S2 becomes 2I. Had the elements of Q instead been drawn from from a normal distribution with σ 2 = 1/2 the expected weight change after the perturbation would have been 0. Note however, that although for the average perturbation matrix Q chosen this way the perturbed weights will be as similar to the original weights as for an orthogonal perturbation matrix, for most choices of Q the deviation from the original weight matrix will be greater than with an orthogonal Q.
For a perturbation matrix of dimension D, the expected matrix E[QQ T ] will be D on the diagonals and 0 otherwise, leading to the expected E[w ij ] = Dw ij .

All elements equal
Assume that instead of having normally distributed elements, all elements of Q were identical, i.e.
for some real scalar q, Eq S2 gives If the elements of K are independently and identically distributed with mean zero, the distribution of φ 1j k i2 + φ 2j k i1 will be the same as the distribution of w ij . In this case, one can expect that for a manifold perturbation given by Eq S6, about half of the variance of the perturbed weight matrixW can be explained by the original weight matrix W .

Scaling
A consequence of choosing a non-orthogonal Q is that the radius of QQ T is not necessarily 1. The simplest case of this is setting Naively, applying this transformation would just scale the weight matrix by a factor of s 2 . This would not change the neural modes per se, as they are usually defined to have unit norm. Instead, it would be the amplitude of the latent variables that is increased, or, equivalently, the average firing rate of the network. However, the firing rates of all networks do also depend on other parameters. Arguably, keeping them constant while scaling the synaptic weights might not be realistic, because we expect firing rates to be within some biologically plausible range. In fact, in the NEF implementation Nengo citenengo, the optimization step for finding Φ is set so that the resulting firing rates should be within some target range, rather than scale with K. Thus, scaling K by a factor s would rather correspond to scaling Φ by a factor of 1/s.
In conclusion, an experiment in which K is scaled would not test the animal's ability to relearn an inside-manifold perturbation, but rather its ability to modulate the global firing rate of the circuit.