Synapse-type-specific competitive Hebbian learning forms functional recurrent networks

Significance Cortical circuits perform diverse computations, primarily determined by highly structured synaptic connectivity patterns that develop during early sensory experience via synaptic plasticity. To understand how these structured connectivity patterns emerge, we introduce a general learning framework for networks of recurrently connected neurons. The framework is rooted in the biologically plausible assumption that synapses compete for limited synaptic resources, which stabilizes synaptic growth. Motivated by the unique protein composition of different synapse types, we assume that different synapse types compete for separate resource pools. Using theory and simulation, we show how this synapse-type-specific competition allows the stable development of structured synaptic connectivity patterns, as well as diverse computations like response normalization and surround suppression.

1 Linear competitive Hebbian learning finds principal components Before considering inhibitory plasticity, we recapitulate how linear Hebbian learning finds the principal eigenvector of a neuron's inputs.Although first described by Oja (1), we will mostly follow the derivation by Miller and MacKay (2) that we will later extend to inhibitory neurons.

Hebbian plasticity without normalization is unstable
We consider a single neuron that receives input from a set of excitatory neurons (Fig. S1A).Its output firing rate r is a weighted sum of the firing rates of its pre-synaptic inputs y.One can conveniently write this as a dot product: τ r r = −r + i w i y i = −r + w y, [1] where w is a vector that holds the synaptic weights, and τ r defines the timescale at which the activity changes.In the following, lowercase letters in bold indicate vectors, and uppercase letters in bold matrices.Following Hebb's principle, synaptic weight changes depend on the pre-and post-synaptic firing rates.In vector notation: τ w = yr [2] where the constant τ sets the timescale of plasticity.Assuming that synaptic weights change on a much slower timescale than firing rates, τ r τ, we make the simplifying assumption that r reaches its fixed point instantaneously, i.e., τ r 1 and r = w y, and consider the same plasticity timescale for all synapses τ = .Then, the average change of the synaptic weights can be expressed as a linear transformation of the original weight vector: w = yr = yy T w = Cw, C ≡ yy , [3] where • is a temporal average and C is the covariance matrix of the synaptic inputs y, assuming inputs have zero mean, y = 0.In the following, we only consider the average weight changes and omit the angled notation for convenience.To solve this differential equation, we express the weight change in the eigenvector basis of the covariance matrix C, which is symmetric and positive-semidefinite and, therefore, has a complete set of orthonormal eigenvectors with non-negative eigenvalues.
⇒ w v = exp(Λt)w v (t 0 ).[5] Here, Λ is the diagonal eigenvalue matrix, and each column of V holds mutually orthogonal eigenvectors, i.e., VV = , and V −1 = V .Each eigenvector component grows exponentially at a rate given by the respective eigenvalue, which we identify with the attraction of the input component.We call eigenvector components with positive eigenvalue attractive, and the eigenvector component with the largest eigenvalue the most attractive input mode.We will later see that eigenvalues that describe the dynamics of input modes can become negative (Sec.2).We will call such input modes with negative corresponding eigenvalue repulsive.
In summary, we find that unconstrained Hebbian plasticity results in the unlimited growth of synaptic weights and is therefore unstable.One way to constrain this unlimited growth is to modify the Hebbian learning rule such that the total synaptic weight is maintained.

A B C
Figure S1: (A) Feedforward excitatory circuit.A post-synaptic neuron with output firing rate r receives synapses w from a set of excitatory neurons with firing rates y E .(B) The normalization operation constrains synaptic weight changes w to a hyperplane that is perpendicular to the constraint vector c by subtracting a multiple γ of the weight vector w.See text for details. Figure adapted from Miller and MacKay (2).(C) Feedforward inhibitory circuit.A post-synaptic neuron with output firing rate r receives excitatory synapses w E from a population of N E excitatory neurons with firing rates y E , and inhibitory synapses w I from a population of N I inhibitory neurons with firing rates y I .The gray horizontal line indicates the separation between two hypothetical brain regions or cortical layers.

Weight constraints stabilize unlimited Hebbian growth
Hebbian plasticity and weight normalization can be considered as two discrete steps.First, growing weights according to the Hebbian rule.Second, normalizing to maintain the total synaptic weight.In this section, we will follow Miller and MacKay (2) and show how one can integrate these two discrete steps into one and derive the effective weight change w.One can write the two steps as w = w(t) + Cw∆t, w(t + ∆t) = W c w w, W ≡ c w(t), [6] This update rule maintains the projection of w onto the constraint vector c by multiplicatively scaling the weight vector after the Hebbian learning step, i.e., w.Alternatively, if we let W be a constant (cf.Eq. 6), the projection onto c would be constrained to be equal to that constant.In the following, we instead assume that the weights are already properly normalized and set the projection value as it was before the plasticity timestep, i.e., equal to W as defined above.

w(t + ∆t) = β [w(t) + Cw(t)∆t], β (w(t), ∆t) = c w(t) c [Cw(t)∆t + w(t)]
, [7] where β describes the multiplicative normalization that depends on the size of the timestep ∆t and the previous weight w(t).It is straightforward to check that the projection of the weight vector onto the constraint vector c does not change, i.e.,

c w(t + ∆t) = c w(t).
[8] Then, the effective weight change w is given as where, in the first and last steps, we used the definition of w(t + ∆t) in Eq. 7. Next, we take the limit In summary, we get (cf.Fig. S1B): Here, γ is a scalar normalization factor that depends on the current weight w.
An alternative way to derive w is to guess the shape of the multiplicative normalization term in Eq. 16 and require that the change along the constraint vector is zero 1  Note that for c being a constant vector of ones, the L1-norm of the weight vector is maintained.However, c does not have to be constant.For example, for c = w the L2-norm is maintained.Also, note that one can analogously derive effective plasticity rules when weights are constrained via subtractive normalization with the ansatz w = Cw − ζk, where k is a vector of ones (2).

Fixed points
From Eq. 16 it is clear that multiples of eigenvectors v of C are fixed points, for which w * = 0. Explicitly, for a scalar constant a and w * = av one gets: Note that this is independent of the choice of the constraint vector c.We next consider the stability of these eigenvector fixed points.

Stability analysis
In the previous sections, we showed how multiplicative normalization constrains the norm of the weight vector and therefore prevents the otherwise unlimited growth of Hebbian plasticity.However, even when the total synaptic weight is constrained, synaptic weights might still be unstable and never settle into a fixed point, e.g., experiencing oscillatory dynamics and unstable fixed points.Following Miller and MacKay (2), we will now explore under what conditions fixed points are stable.Formally, a fixed point in a linear system is stable when the largest eigenvalue of the Jacobian is negative, or marginally stable when it is equal to zero (3).The weight dynamics around a fixed point w * can be approximated with its Taylor expansion: = J * (w − w * ), [21] where w * is zero, by definition, and J * is the Jacobian evaluated at the fixed point.The Jacobian is defined as ... A fixed point is stable if small perturbations away from the fixed point, ∆w = w − w * , decay to zero, i.e., d dt ∆w = w − w * = w ≈ J * ∆w, [23] where we approximated w with its Taylor expansion (Eq.19), since the perturbation is small, i.e., w is close to the fixed point.The result is a linear differential equation that one can solve as where all vector components decay to zero if all eigenvalues of J * are negative 1,2 .As we will see later, it is useful to rewrite the weight dynamics (Eq.16) as It follows 1 : where w * = w * = av * is the fixed point with v * being an eigenvector of C. The scalar a is the length of the fixed point weight vector w * (which cancels) and λ * is the eigenvalue to v * .To find the eigenvalues of the Jacobian, λ J , we diagonalize J by switching to the eigenbasis of C. When V is the matrix that holds the eigenvectors of C as columns one gets where Λ is a diagonal matrix that holds the eigenvalues of C. Without loss of generality, we can assume that the first column of V is equal to v * .Then e * = V v * is a column vector of zeros, except for the first entry, which is equal to one.Then, the first bracket becomes an upper triangular matrix with ones on the diagonal, except for the first diagonal entry, which is zero.From this, it follows 2 that the eigenvalues of the Jacobian are If λ * is the largest eigenvalue, i.e., w * is a multiple of the principal eigenvector of C, then all λ J are negative or zero, and the fixed point is marginally stable.If there exists a λ > λ * , the corresponding λ J is positive and the fixed point is unstable.Therefore, the eigenvector corresponding to the principal eigenvalue is the only (marginally) stable fixed point.In summary, linear Hebbian learning combined with multiplicative normalization becomes selective for the principal eigenvector of the input covariance matrix and thus performs principal component analysis (PCA).Next, we consider what happens when a neuron also receives inhibitory input.

Classic Inhibitory plasticity prevents stimulus selectivity
Previous work suggested a homeostatic inhibitory synaptic plasticity rule (4) that enforced a post-synaptic target firing rate r 0 : However, when combined with excitatory plasticity, this classic rule prevents the development of stimulus selectivity (cf.Fig. 1A, E & F).For completeness, we briefly recapitulate this result, presented in Clopath et al. (5): We consider a simplified circuit of a single post-synaptic neuron with firing rate r that receives lateral input from N I inhibitory neurons, while all neurons receive feedforward input from a population of N E excitatory neurons 3 (cf.Fig. S1C).Then, y E and y I are vectors that hold the firing rates of the excitatory and inhibitory populations.We now explore the self-organization of excitatory and inhibitory synaptic weights, w E and w I , that project onto the single post-synaptic neuron, while input synapses Q that project onto inhibitory neurons remain fixed.In Clopath et al. (5), the authors find that classic inhibitory plasticity is required to act faster than excitatory plasticity to enable stable weight dynamics (5).
For much faster inhibitory plasticity, the dynamics of excitatory and inhibitory weights decouples, and fixed points of the inhibitory weights w * I can be considered separately from the fixed points of excitatory weights.When excitatory and inhibitory inputs are equally stimulus selective, the fast dynamics of inhibitory weights ensures that the target firing rate is consistently met, r * ≈ r 0 , and plasticity of excitatory synapses only depend on pre-synaptic terms and constants 4 : Supplementary Material | PNAS When all pre-synaptic neurons have similar average firing rates, y E i ≈ y 0 , and weights change on a slower timescale than activities, as is the case biologically, the average excitatory synaptic weight change becomes where c is a vector of ones.The average synaptic weight change is identical across synapses, which prevents the development of stimulus selectivity (Fig. 1E & F).Therefore, classic inhibitory plasticity that enforces a target firing rate cannot explain the joint development of stimulus selectivity and inhibitory balance.Instead, we propose that, as excitatory weights, also inhibitory weights are constrained via a competitive process that normalizes the total inhibitory input that a neuron receives.

Synapse-type-specific normalization balances E-I receptive fields
Different from the normalization of excitatory weights, the normalization of inhibitory weights is not motivated by the requirement for stability.Inhibitory synaptic plasticity that depends on neural activity is self-limiting, since increasing inhibitory weights eventually prevent the neuron from firing, and thus prevent further plasticity.Instead, we motivate the normalization of inhibitory synaptic weights by the competition for a limited amount of synaptic building blocks that may also drive excitatory normalization (see Main text for details).In the following, we generalize the approach outlined in the previous Sections for excitatory weight normalization to the case of simultaneous excitatory and inhibitory normalization.We consider the same circuit architecture as in Section 1.2.3 (cf.Fig. S1C) with rate dynamics where is the unit matrix with appropriate dimension, 0 are matrices of zeros and appropriate dimensionality, and we defined the modified weight and input vectors, w and y.Similar to before, we assume fast activity dynamics, τ r 1, and write the Hebbian part of the time-averaged weight dynamics as where we defined the modified covariance matrix C. In general, we assume that all synapses of one type, excitatory or inhibitory, change equally fast (cf.Table 1).Then, the matrix τ holds the timescales of excitatory plasticity, τ E = τ E , and inhibitory plasticity, τ I = τ I , as matrices on the diagonal, and is zero otherwise.In the following, we drop the bracket notation • for better readability.As in the case of only excitatory input, we can implement multiplicative normalization by additional constraint terms.Now also for inhibitory weights (cf.Eq. 16): where 0 indicates vectors of zeros of appropriate dimension (N I and N E ) that we do not specify for better readability.The constraint factors γ and ρ follow from the requirement that the weight vector does not grow along the direction of the constraint vectors c E and c I .Here we choose them such that the sums over the excitatory and inhibitory weights remain constant, i.e., the L1-norm of the excitatory and inhibitory part of the weight vector is maintained 1 .
where the number of non-zero entries in c E and c I is equal to the number of excitatory N E and inhibitory neurons N I , respectively.Based on these requirements we derive expressions for the scalar constraint factors γ and ρ: Finally, we can write the weight dynamics as [45]

Fixed points
For the fixed points we have to find weight vectors w * for which the time derivative w * is equal to zero: which is equivalent to for λ E and λ I being arbitrary scalar.

Eigenvectors of the modified covariance matrix are fixed points
It is straightforward to check that multiples of eigenvectors v of the modified covariance matrix C with eigenvalue λ are fixed points: [49] In the following, we will refer to eigenvectors of the modified covariance matrix as fixed point eigenvectors, and to eigenvectors of the feedforward excitatory covariance matrix C as feedforward eigenvectors.Next, we will try to specify the eigenvectors of C. In general, eigenvectors of C depend non-trivially on the tuning of the laterally projecting population (cf.Sec. 3, Eq.121).However, the problem simplifies when the laterally projecting inhibitory neurons are tuned to multiples of eigenvectors of the excitatory population's covariance matrix.This is what one would expect when the post-synaptic excitatory neuron r and the inhibitory population y E both receive excitatory input from the same external brain region y E and synapses from the external population onto inhibitory neurons are plastic according to a Hebbian rule with multiplicative normalization (cf.Fig. S1C).Although we showed in Section 1.2.2 that without recurrent interactions only the principal eigenvector is a stable fixed point, we will find that with suitable recurrent interactions any feedforward eigenvector can be stable (cf.Sec. 3 & 5.2.3).Formally we set where each row of Q = A V is the feedforward weight vector of an inhibitory neuron which is equal to a positive multiple, a, of an eigenvector v of the excitatory covariance matrix C = y E y E .Then V holds all eigenvectors as columns, and A is a matrix where each multiple is the only non-zero element per column, such that AA is a diagonal matrix.We will now show that in this scenario multiples of the excitatory and inhibitory part of the eigenvectors of the modified covariance matrix C are fixed points.As a first step, we explicitly calculate the eigenvectors.
Supplementary Material | PNAS 2.1.2Eigenvectors and eigenvalues of the modified covariance matrix In the previous section, we have seen that eigenvectors v of the modified covariance matrix C are fixed points.In this section, we will find an explicit expression for these eigenvectors when inhibitory neurons are tuned to feedforward eigenvectors, i.e., inhibitory neurons are tuned to eigenvectors v of the excitatory covariance matrix C. Making use of Eq.50 the modified covariance matrix becomes Then, a full set1 of linearly independent eigenvectors V and their inverse V −1 is given as 2 .
where each column of V is an non-normalized eigenvector.The eigenvalue spectrum is Similar to before, we call eigenvectors of the modified covariance matrix with positive eigenvalue attractive.Different from the case of only excitatory feedforward input, eigenvalues of the modified covariance matrix can also be negative.In this case, we call the corresponding eigenvector repulsive (cf.Sec.1.1).
For eigenvectors in the right matrix column of V in Eq. 52, the excitatory and inhibitory components of the membrane potential exactly cancel, post-synaptic firing rates are zero, and no plasticity is induced: For multiple postsynaptic neurons with firing rates r, where each neuron is tuned to one of these eigenvectors, one gets ⇒ r = y E , −y E VA VA = 0. [55] Since these eigenvectors result in post-synaptic firing rates of zero, and they define the null space of the C matrix (Eq.53), we call them 'null eigenvectors' or 'null fixed points', and all eigenvectors that are not null eigenvectors 'regular' eigenvectors or fixed points.Note that for each additional inhibitory neuron that is tuned to a feedforward eigenvector, there is an additional null eigenvector, since inhibitory synaptic weights can now shift between the original, and the additional inhibitory neuron to cancel post-synaptic firing.Overall, there are always N I null eigenvectors and N E regular eigenvectors 3 .Note that A is a matrix with exactly one non-zero element per column (cf.Eq. 50f.), and we can see from Eq. 52 that the excitatory part of each null eigenvector is proportional to the excitatory part of one regular eigenvector.In the following, when we speak of regular eigenvectors and corresponding null eigenvectors, we mean eigenvectors with proportional excitatory components.
We have already shown in Section 2.1.1 that eigenvectors of C are fixed points.Each eigenvector specifies an exact ratio between the excitatory and inhibitory weight norm.Since our learning rule separately maintains the total excitatory and inhibitory synaptic weights, reaching any of these fixed points would require detailed fine-tuning at the point of initialization.In the next section, we show a more general set of fixed points that does not require any fine tuning of weight norms.

Non-eigenvector fixed points
In this section, we show that there exist fixed points that are not eigenvectors of the modified covariance matrix.In particular, arbitrary multiples of the excitatory and inhibitory parts of regular eigenvectors, i.e., of eigenvectors that result in non-zero post-synaptic activity, are fixed points.We make the ansatz that the matrix W * holds fixed points as columns and has the shape where K E and K I and are diagonal scaling matrices of arbitrary constants.The fixed point condition that follows from Eq.48 is [57] We now show that for any K E , K I we can find diagonal matrices Λ E , Λ I that fulfil this condition 1 .We write explicitly where we made use of the fact that independent of their subscript, the K, Λ, and AA matrices are diagonal and commute.By comparing the left and right sides of the equations, we find which are diagonal matrices, as required 2 .Before we consider the stability of these fixed points in Section 2.2, we first show that there is an additional set of fixed points.

General fixed points
Having covered various special cases of fixed points for the dynamics, we now consider the general problem.Recall that fixed points are defined to satisfy Equation 48: Expanding this using our expression for C (Eq. 51), we can see that this is equivalent to: and equivalently Inserting the first into the second expression, we can conclude that [70] If λ E = λ I 0, then we know that w * is an eigenvector of the modified covariance matrix, as discussed in Section 2.1.1.
In the case that λ E = λ I = 0, we have the null eigenvectors discussed in Section 2.1.2.We therefore now address the case that λ E λ I .We begin with the case λ I 0. Then we can insert Eq. 70 into Eq.68 to arrive at which, together with Eq. 70, gives necessary and sufficient conditions for a fixed point.From Eq. 71, we conclude that V w * E is an eigenvector of the diagonal matrix Λ − λ E λ I AA with eigenvalue λ E .When V w * E is one-hot, then the vector w * consists of an arbitrary multiple of the excitatory and inhibitory parts of a regular eigenvector, as covered in Section 2.1.3.
We now turn our attention to the case where V w * E is not simply one-hot.We can now say that for each component j of V w * E which is non-zero, the following equation must hold: This is a linear system in the pair of variables λ E and λ E /λ I .We work under the mild assumptions that the eigenvalues λ j , the diagonal elements (AA ) jj , and their product λ j (AA ) jj are distinct for each j.These conditions will in general hold in the absence of fine tuning.In this case, λ E and λ I provide two degrees of freedom and there will only be solutions when V w * E is (at most) two-hot, having non-zero components, j and k.Such solutions satisfy: which we can solve to obtain the expressions: The components of the two-hot solution are determined by the known initial values of k E = c E w E and k I = c I w I , which are kept constant throughout training 1 .Although two-hot fixed points do not require fine tuning of excitatory and inhibitory weight norms, we did not observe them in any of our numerical simulations and therefore assume they are unstable.
The final case to be considered is when λ I = 0, λ E 0. In this situation, Eq. 70 tells us that V w * E is in the kernel of A and therefore in the kernel of the diagonal matrix AA 2 .By using Eq.68, we can therefore conclude that We work under the assumption that, in the absence of fine tuning, Λ has distinct non-zero eigenvalues.In this case, the first term in Equation 75 is zero, and Aw * I must also be in the kernel of AA and therefore in the kernel of A .So w * I is in the kernel of A A and therefore the kernel of A. By Equation 68, this tells us that ΛV w * E = λ E V w * E , and therefore V w * E is an eigenvector of Λ with eigenvalue λ E .We therefore arrive at a fixed point for the system in which V w * E is one-hot with support on the kernel of AA , and w * I is in the kernel of A. This implies w * I y I = 0 (cf.Eq. 50) which is biologically implausible since we constrain synaptic weights w * I and firing rates y I to be positive.Under mild assumptions regarding Λ and AA , we have thus exhaustively characterized the fixed points of the system.

Stability analysis
We first consider the stability of fixed points that are regular eigenvectors of the modified covariance matrix and discuss the case of non-eigenvector fixed points afterwards.With Eq. 45, for the Jacobian J it follows (cf.Eq. 29) 1 Briefly, the two normalization conditions are k E = c w * E , and where we used Eq.70.Then, by inserting Eqs.74 we get two linear equations for the two unknown components of w * E , which can be solved in terms of k E , k I , λ i , λ j .We can then insert the solution for w * E into Eq.70 to obtain w * I , which together defines all components of the eigenvector. 2Note that ker(A ) = ker(AA ) and ker(A) = ker(A A), for any matrix A.
where v * E and v * I are the excitatory and the inhibitory part of the eigenvector fixed point w * = v * with eigenvalue λ * , with an additional set of zeros to reach the correct dimensionality of the vector (cf.Eq. 41).To find the eigenvalues λ J of the Jacobian, we switch to the eigenbasis of the modified covariance matrix1 : where we inserted VV −1 ≡ .The result is a block triangular matrix where each block on the diagonal corresponds to one regular eigenvector and its potentially multiple null eigenvectors.To better see this, we consider the first and second part of Eq. 77 separately.We define ≡ τ −1 , which remains a diagonal matrix with time constants for excitatory and inhibitory synapses on the diagonal, E = E and I = I .Inserting the definition of the eigenvectors matrix and its inverse (Eq.52) we write As one would expect, for E = I , this is equal to a scalar times the identity matrix.When we switch columns and rows such that pairs of regular and corresponding null eigenvectors form blocks, this becomes a block diagonal matrix.Note that this does not change the determinant or the eigenvalues of the matrix as for each row switch, there is a corresponding column switch that maintains the characteristic polynomial.Alternatively, we can assume that the matrix of eigenvectors V and its inverse V −1 are already appropriately sorted.Without loss of generality, we assume that the first columns of V are the fixed point's eigenvector v * and its corresponding null eigenvectors, and write2 where a * is a column vector that holds the multiples of the inhibitory neurons that are tuned to the feedforward eigenvector v * .As before, 0 are matrices of zeros and appropriate dimensionality, and ellipsis indicate continuing blocks on the diagonal with similar terms that belong to the non-fixed point eigenvectors and their null eigenvectors 3 .Similarly, we can write the second part of Eq. 77 as a block triangular matrix.Before sorting, we write where d E and d I are row vectors that hold the L1-norms of the eigenvectors' excitatory and inhibitory parts as a fraction of the L1-norm of the fixed point eigenvector's excitatory and inhibitory parts.The vector e * is zero except for one entry, equal to one, which corresponds to the fixed point feedforward eigenvector v * .We continue by multiplying the inverse eigenvector matrix V −1 from the left: where we defined the normalization matrix N of the inverse eigenvector matrix V −1 (cf.Eq. 52) to improve readability.It follows that the matrix above holds non-zero values in only a few rows, corresponding to the fixed point eigenvector (top block) and its null eigenvectors (bottom block).After rearranging, we get where d * E , d * I and d E , d I are the entries of d E , d I that correspond to the fixed point eigenvector and its null eigenvectors, respectively.As before, ellipsis indicate additional non-zero entries.To find the respective entries of d E , d I we use the definition of V (Eq.52) to write After rearranging the entries that correspond to the fixed point eigenvector and its null eigenvectors to the front we get [87] where e * selects the proper columns and c is a row vector of ones of appropriate dimensionality.We insert Eq. 87 & 88 into Eq.84 and find where we defined the matrix M * .In summary, we find that after rearrangement, Eq. 77 is a block triangular matrix.
where we used Eq.80 and Eq.89.Therefore, to find the eigenvalues, we consider each diagonal block separately.We make the simplifying assumption that there is exactly one inhibitory neuron tuned to each feedforward eigenvector.Then, a * → a * becomes a scalar, N and A = A become diagonal, and M * → 1.The transformed Jacobian remains triangular and becomes with 2 × 2 blocks on the diagonal of which we only show the first, that corresponds to perturbations in the direction of the fixed point eigenvector or its null eigenvector 1 .From the matrix product above, we see that their corresponding eigenvalues must be zero since the first two columns of the second to last matrix are zero.For perturbations in the direction of a non-fixed point eigenvector v † or its null eigenvector we have to consider the block matrix where V † is a two-column matrix that holds v † and its null eigenvector.The eigenvalues of this matrix are negative under two conditions.First, its determinant must be positive, and second, its trace must be negative.For trace and determinant, we find Finally, the two stability conditions read where, for the trace term, we made use of the equality λ † = λ † 1 − a †2 (cf.Eq. 53) to replace λ † .

Principal component analysis in inhibition modified input space
The first stability condition above states that only the fixed point v * with the largest eigenvalue, λ * > λ † , λ † , can be stable, and then only if it is not repulsive, i.e., provided that its corresponding eigenvalue is larger than zero.An eigenvector can become repulsive if inhibition is sufficiently strong, i.e., if This implies that post-synaptic neurons tuned to repulsive eigenvectors receive more inhibition than excitation, which results in negative firing rates r * = y E v * − y E v * a *2 < 0, for a * 2 > 1 (cf.Eq. 55).However, in biology, neurons with larger inhibitory than excitatory input are hyperpolarized and remain silent, which is why we assume λ * > 0. In the following, we call the combination of the excitatory feedforward attraction of an eigenvector λ * (cf.Sec.1.1) plus any contribution of laterally projecting neurons, in this case, minus the lateral inhibitory repulsion a * 2 λ * , the effective attraction, λ * , of a feedforward input mode.For I = E , the second condition reduces to1 λ † − 2λ * < 0, which holds if the first condition is met.Therefore, the post-synaptic neuron becomes tuned to the eigenvector of the modified covariance matrix with the largest eigenvalue, i.e., it performs principal component analysis on a modified feedforward input space, where the attraction of feedforward eigenvectors is modified by laterally projecting inhibitory neurons (cf.Eq. 53).We will further discuss the notion of a modified input space in Section 3.

Fast inhibition increases stability
In our networks, stationary states can still emerge when inhibitory plasticity is slower than excitatory plasticity.In the extreme case of static inhibition, I = 0, the second stability condition is still satisfied if the fixed point attraction λ * is larger than the feedforward attraction λ † of any other eigenvector, λ † − λ * < 0. When inhibitory weights are static, they remain tuned to the fixed point and the repulsive component of competing eigenvectors a †2 λ † do not matter for stability.This explains why we have to consider only the attractive part λ † of the effective attraction λ † in the first term of the second stability condition.However, for growing I > 0, the influence of the inhibitory part of competing eigenvectors increases, corresponding to an increasingly negative second term in the second stability condition 2 .Then, for sufficiently fast inhibitory plasticity I > E , the second condition always holds.Therefore, we consider slightly faster inhibitory than excitatory plasticity in our numerical simulations (cf.Table 1).

Stability of non-eigenvector fixed points
Before, we considered the stability of fixed points w * that are eigenvectors v of the modified covariance matrix C. Weight vectors of that shape put a strong constraint on the choice of the weight norms, as the ratio between the excitatory and the inhibitory weight norms is given by the norms of the excitatory and the inhibitory parts of the eigenvector (cf.Eq. 52).The issue was solved in that we found that arbitrary combinations of multiples of the excitatory and inhibitory components of regular eigenvectors are also fixed points (cf.Sec.2.1.3).We will now consider the stability of such non-eigenvector fixed points.
Let the shape of a fixed point w * be (cf.Eq. 56) where k E and k I are scalar constants.We recapitulate the general weight dynamics as given in Eq. 45: [99] Instead of evaluating the eigenvalues of the Jacobian, we now switch to a new coordinate system in which the Jacobian will have a familiar shape.This is possible since fixed points and their stability do not depend on the choice of coordinates.We define: from which the weight dynamics can be written as where we inserted K − 1 /2 K 1 /2 = .We now make use of the following identities: We find that the K 1 /2 matrices inside the bracket cancel, and we can pull K − 1 /2 from the right side to the left side of the bracket: We introduce the following definitions Note that C is not the modified covariance matrix expressed in the new coordinate system but a new modified covariance matrix that corresponds to an altered input space where excitatory and inhibitory input firing rates y E , y I are scaled by k I , respectively.In summary, we can write the plasticity of the weight vector in the new coordinate system as 1 We are interested in the stability of the fixed points given in Eq. 98.In the new coordinate system, they become [106] 1 Here, τ −1 and K −1/2 are both diagonal matrices and commute.

Supplementary Material | PNAS
It is straightforward to proof that w * is an eigenvector of the new modified covariance matrix C with eigenvalue λ * = (k E − k I a *2 )λ * : With C defined in Eq. 51 we get where we defined a * 2 as the entry of the diagonal matrix AA that corresponds to the eigenvector v *1 .Note that this is independent of the change of variables, however, only in the new coordinate system one can identify the new modified covariance matrix with an actual input space 2 , where pre-synaptic firing rates are scaled by k E , k I (Eq.104).
In theory, we can now proceed in finding the eigenvalues of the Jacobian3 , as explained in Section 2.2. .As before, one finds that stability is largely determined by the eigenvalues of the modified covariance matrix, which now are λ .Apart from providing a principled way to determine if a non-eigenvector fixed point is stable, our formulation provides additional insight: Let's assume the total synaptic inhibitory weight of a neuron is very small, much smaller than any eigenvector of C would suggest, i.e., k I 1, while the excitatory weight norm is equal to one, which implies k E = 1.As one would expect intuitively, the neuron does not exhibit much of the repulsion of the inhibitory neurons (cf.Eq. 104 for k I 1), and its stability would be primarily determined by the excitatory attraction of the different eigenvector modes, i.e., λ = (k E − k I a * 2 )λ ≈ λ.In the extreme case, when the inhibitory weight norm is zero, i.e., k I = 0, only the activity of the excitatory population is relevant.
While the effective plasticity timescale τ = τK in Equation 105 depends on the magnitude of the excitatory and the inhibitory part of the specific fixed point under consideration, this does not mean that the speed of synaptic plasticity is different from the original formulation in Equation 45.For example, when we consider a fixed point with a decreased inhibitory weight norm k I < 1, the effective inhibitory plasticity appears to increase, since τ I = τ I k I .However, this effect is balanced by the decrease in pre-synaptic inhibitory firing rates, which decreases with decreasing k I .Similarly, the coordinate system in which we describe the weight dynamics also does not affect the speed of plasticity 4 .From Equation 103 we see that we can freely shift scaling matrices K between the modified covariance matrix and the plasticity timescale by pulling diagonal matrices of the same shape as K through the bracket (cf.Eq. 102).However, in Section 2.2 we only considered the stability of fixed points that are regular eigenvectors of the modified covariance matrix.If we had chosen, e.g., C = K − 1 /2 CK 1 /2 and τ = τ, then w * = K 1 /2 v * (cf.Eq. 106) would not be a regular eigenvector of C (cf. Eq. 107f.).Therefore, our derivation in Section 2.2 would not apply, and we would need to find a different way to proof stability.

Lateral input stretches and compresses the feedforward input space
Before we consider how synapse-type-specific Hebbian plasticity affects learning in fully plastic recurrent networks, we first build additional intuition for how static lateral input affects the weight dynamics.From the previous section we know that in this case the eigenvalues of the modified covariance matrix are the key factors that determine fixed point stability, and from Sections 1.1 & 2.1.2we know that these eigenvalues describe the Hebbian growth towards the corresponding eigenvector that can be attractive or repulsive, corresponding to a positive or negative eigenvalue.When a neuron receives only feedforward excitatory input (Fig. S2A), the weight dynamics is described by a true covariance matrix with eigenvalues equal to the variances along the principal components of the feedforward input space (cf.Sec. 1).Then the weight vector in the fixed point aligns with the direction of maximal variance in the ).(B) Top: Same as in A for a neuron that receives additional input w q from a laterally projecting excitatory neuron r q which is tuned to an eigenvector q of the original input covariance matrix.Bottom: the effective input space y eff of the target neuron (dark blue triangle) is warped such that the variance along the eigenvector q (blue arrow) is stretched in proportion to the absolute value of the weight vector q.The contour lines of the original input distribution from A are shown in light gray for reference.(C) Top: Same as B for a laterally projecting inhibitory neuron.Bottom: Now, the effective input space is compressed.See text for details.
input space (Fig. 1G).In the following, we introduce a similar perspective and show that additional lateral input can be interpreted to stretch and compress the original feedforward input space, while the feedforward component of the weight vector performs PCA on this modified input space.We consider a circuit of two neurons that both receive feedforward input from a population of input neurons y (Fig. S2B, top).Let the first neuron have a fixed, non-plastic set of feedforward weights q and firing rate r q = q y [111] We let the first neuron project laterally onto the second neuron via a synaptic weight w q , without receiving any lateral input itself.Then the equilibrium firing rate of the second neuron is r = w q r q + w y [112] where we assume that both w and w q are plastic according to a stabilized Hebbian rule.
From the perspective of the second neuron, the input space is increased by one dimension due to the additional lateral input, i.e., we can write Eq. 112 as where, we defined the new input vector y and the combined input weights w.Effectively this is still a feedforward network without feedback, and the static covariance matrix C of the new inputs y fully determines the average synaptic weight dynamics1 : where C is the covariance matrix of the original input y.We are interested in the eigenvectors v and eigenvalues λ of this matrix for two reasons.First, because they describe the attraction towards different input modes due to the Hebbian term in our competitive plasticity rule.Second, because eigenvectors are fixed points with their stability Supplementary Material | PNAS mostly determined by the eigenvalues.Eigenvectors and eigenvalues must satisfy where v q and v F are the lateral and the feedforward components of the eigenvector, respectively.In the following, we focus on the feedforward component for different q.It follows q Cv F + q Cqv q = λv q . [118] q C v F + qv q = λv q . [120] Inserting the first into the second expression gives v q = q v F which, when inserted into the first expression, results in: This is again an eigenvector equation, where the feedforward components of the original eigenvector v F , are themselves eigenvectors of the matrix C( + qq ).Note that for q = 0 we recover the case without lateral projections and feedforward components are multiples of eigenvectors of C with attractions λ = λ.For general q, the solution is not straightforward: We consider the equation in the input eigenspace, where Eq. 121 becomes with Λ being the diagonal matrix of feedforward eigenvalues λ and the subscript (•) v indicates a vector in the eigenbasis of C. In this basis, eigenvectors of C are unit vectors, i.e., v v = e, where e is a vector of zeros with a single entry equal to one, corresponding to the respective eigenvector.When q contains components of more than one eigenvector, the matrix q v q v is not diagonal and eigenvectors of C, do not solve the equation.Here we consider a simplified case: When the first neuron had plastic feedforward input, we know from Section 1 that it would converge to a multiple of an eigenvector of the feedforward covariance matrix1 , q ∝ v † , with Cv † = λ † v † .Then, q v q v = e † e † is diagonal with a single non-zero entry, and from Equation 122 it is obvious that feedforward eigenvector components of C are eigenvectors of the feedforward covariance matrix C that solve Eq. 121.
To find the eigenvalues λ, we first consider feedforward eigenvector components v F that are orthogonal to q: Then it follows from Equation 121 that the corresponding eigenvalue of the modified covariance matrix equals the eigenvalue of the original covariance matrix, which is, by definition, equal to the variance σ ‡2 of the input distribution along the respective eigenvector.
Therefore, input modes that are orthogonal to the tuning of the laterally projecting neuron maintain their attractions, equal to the respective eigenvalues of C, and the laterally projecting neuron does not affect the Hebbian growth dynamics in the input subspace orthogonal to q2 .The remaining feedforward eigenvector component is proportional to q: where we again made use of Equation 121.Here, a q is equal to q , the L2-norm of q, and σ 2 q = λ † a 2 q is the firing rate variance of the laterally projecting neuron 1 .Therefore, the second neuron adjusts its feedforward weights w as if the variance along the eigenvector q was increased by σ 2 q (Fig. S2B, bottom).In that sense, the second neuron 'perceives' its feedforward input space as stretched and we speak of a modified input space (cf.Sec.2.2.1) that is described by a modified covariance matrix C. We note that it is possible to choose q such that an arbitrary direction of the input space becomes stable.For q = C −1 h Eq. 121 is 2 [126] For increasing h , the principal eigenvector transitions from v F ∝ v, for h = 0, to v ∞ F ∝ h for h → ∞.In the following, we only consider the case when q is parallel to one of the eigenvectors of C.Then, for sufficiently large a q and q ∝ v † , an arbitrary non-principle eigenvector v † with attraction λ † = λ † (1 + a 2 q ) can become stable.In that case, the corresponding fixed point is of the following shape 3 : When the laterally projecting neuron is inhibitory (Fig. S2C, top), the modified covariance matrix becomes (cf.Eq. 51) and it follows that the input space is compressed along q ∝ v † (Fig. S2C, bottom): [129] In the case of lateral inhibition and sufficiently large vector norms a q , an eigenvector can become repulsive, i.e., its eigenvalue becomes negative.Geometrically, this corresponds to a reflection of the input space along q through the origin, which can no longer be visualized as intuitively as in Fig. S2.
We can generalize the overall approach to multiple excitatory and inhibitory neurons such that the effective attraction towards a feedforward eigenvector becomes where λ = σ 2 , the vectors a E , a I hold the feedforward vector norms of the laterally projecting neurons that are tuned to the respective feedforward eigenvector, and σ A = √ λa A , A ∈ {E, I}, hold the standard deviations of their firing rates.This allows writing the regular fixed points as 4 This implies that for regular fixed points, the total synaptic weight is distributed among lateral synapses in proportion to the standard deviation of their pre-synaptic activities.Note that the different weight norms of the excitatory and inhibitory part of non-eigenvector fixed points can distort this relation (cf.Sec.2.1.3).In summary, we demonstrated how static lateral input can be interpreted to reshape the feedforward attraction landscape of afferent neurons.Note that these results are independent of what causes the laterally projecting neurons' tuning.The second, afferent neuron does not 'see' what inputs to the laterally projecting neurons cause their 1 From Equation 111 we immediately find σ 2 q = r 2 q − r q 2 = q Cq = λ † a 2 q , for q = a q v † , where we assumed zero mean input, y = 0. 2 Note that C −1 = (C −1 ) since C is a true covariance matrix, i.e., C and C −1 are symmetric. 3If a q is too small so that λ † < λ ‡ = λ ‡ , the principal feedforward eigenvector v ‡ of C with eigenvalue λ ‡ is stable and w * = (v ‡ , 0) . 4If none of the laterally projecting neurons is tuned to a specific feedforward eigenvector v ‡ , i.e., v ‡ ⊥ q i i, the corresponding fixed point becomes v ‡ = (v ‡ , 0 , 0 ) .tuning.For example, in addition to feedforward input, laterally projecting neurons might be integrated into a recurrent circuit of neurons that are all tuned to the same eigenvector 1 .Then σ2 E , σ 2 I result from recurrent interaction in addition to the norm of the feedforward weight vectors.However, for the dynamics of the second neuron, it would not make any difference as long as the firing rate statistics of its pre-synaptic inputs were the same.In the following sections, we will consider circuits where the firing rate statistics emerge from recurrent interactions.

Eigencircuits
In the previous section we considered neurons that receive feedforward input from an excitatory population and lateral input from neurons with fixed feedforward tuning (Fig. S2).We found that the attraction of different feedforward input modes is determined by the eigenvalues of a modified covariance matrix, composed of a feedforward contribution and a contribution due to the laterally projecting neurons that is proportional to the variance of their firing rates (Eq.131).In this section, we consider networks of recurrently connected, laterally projecting neurons and explore the variances of their firing rates.
First, we consider a network of excitatory and inhibitory neurons y E , y I that are laterally connected to themselves and each other and receive feedforward input from the same excitatory population y.We assume that the activity in the network is dominated by feedforward input such that neurons become selective for different eigenvectors of the feedforward covariance matrix C = yy , e.g., the steady state firing rate y a of a neuron that is tuned to an eigenvector v a is proportional to v a y (Fig. 4A), where the proportionality factor depends on the number and firing rates of other neurons that are tuned to the same eigenvector (see Sec. 4.1).Then the average Hebbian growth of a synapse that connects two neurons that are tuned to different eigenvectors is 2 : [133] Due to the competition for synaptic resources, the synapse loses out to the non-zero growth of synapses that connect neurons that are tuned to the same eigenvector, and decays to zero over time (Fig. 4B).Eventually, the circuit is separated into sub-circuits that are tuned to different eigenvectors with recurrent connections within, but not between sub-circuits.Since there is one sub-circuit per eigenvector of the feedforward covariance matrix, we call these decoupled sub-circuits 'eigencircuits' (cf.Fig. 4).

Variance propagation
In Section 3, we have seen that the attraction and the stability of a feedforward eigenvector are closely related to the firing rate variances of laterally projecting neurons, independent from how these variances arise.In the effective feedforward circuits that we considered, it was straightforward to compute variances based on feedforward weight norms (Eq.131 f.).We now show how variances can be determined in recurrent eigencircuits, which allows to quantify the effective attraction of an input mode.We consider a generic eigencircuit and investigate how variances propagate through the network, i.e., our goal is to express the standard deviation of a neuron's firing rate as a function of the standard deviations of its pre-synaptic input firing rates.For a neuron in an eigencircuit, all pre-synaptic inputs with non-zero synaptic weight are tuned to the same feedforward eigenvector v.We only consider these non-zero entries and assume that the steady state firing rate of an arbitrary neuron can be written as (Fig. S3A) Note that before, a E and a I referred to feedforward weight norms (cf.Sec. 3).Now these vectors more generally express how firing rate variances relate to the input variance along the eigencircuit's feedforward eigenvector, without making any assumptions about how this tuning arises.We will show in Section 5 that this assumption is correct and specify how the entries of a E , a I relate to the recurrent excitatory and inhibitory weights (cf.Eqs.161 & 162).For the weight vectors, we require that the excitatory and inhibitory parts are normalized to maintain the total amount of inhibitory and excitatory synaptic resources: where W E , W I are scalar weight norms, and v E , v I are the excitatory and inhibitory parts of the fixed point eigenvector (cf.Sec.5), with entries that are proportional to the pre-synaptic standard deviations (cf.Eq. 132).Then, the p-norm, • p , is maintained due to competition for synaptic resources 1 .For the post-synaptic firing rate, it follows The first bracket is a scalar pre-factor which makes it straightforward to compute the standard deviation: For a network in the steady state, i.e., when synaptic weights converged, this equation puts the standard deviation of neural firing rates in relation to each other, i.e., it provides the standard deviation of a post-synaptic neuron's activity as a function of the standard deviations of its pre-synaptic input neurons' activities 2 .It describes how standard deviations and variances 'propagate' through the network.In the next section, we will use this variance propagation equation (Eq.140) to express the standard deviations in terms of only the weight norms and the feedforward standard deviation σ.

Consistency conditions provide eigencircuit firing rate variances
We now consider a single eigencircuit where n E excitatory and n I inhibitory neurons are recurrently connected, and are tuned to the same feedforward eigenvector with standard deviation σ (Fig. S3B).In their steady state, all neurons have to fulfil the variance propagation equation (Eq.140).In the fully connected eigencircuit, the firing rate variance of each neuron depends on the firing rate variances of all other neurons, and all neurons have the same set of nonzero pre-synaptic inputs.This provides N = n E + n I consistency conditions for the N unknown standard deviations.For example, the condition for a single excitatory neuron i reads where we chose the L1-norm, p = 1, for normalization (but see Sec. 4.3), and W AB , A, B ∈ {E, I} are the total synaptic weight that a neuron of type A receives from neurons of type B. We make the simplifying assumption that all neurons have similar weight norms, i.e., W i AB ≈ W AB , i, A, B ∈ {E, I}.Then, also the standard deviations of their activities are similar, and we approximate σ i A ≈ σ A , i, A ∈ {E, I}: The standard deviations of excitatory and inhibitory neural firing rates become After some algebra, this yields the standard deviations as This provides standard deviations as a function of the number of neurons in the eigencircuit 1 , n E , n I , their weight norms, W AB , and the standard deviation of the feedforward input activity along the corresponding eigenvector, σ.Via Eq. 131, we can determine how the eigencircuit modifies the attraction of its feedforward eigenvector, i.e., the effective attraction λ is where we defined the attraction of the eigencircuit, λ eig , and λ is the attraction of the respective feedforward eigenvector.In the following, we refer to λ interchangeably as the effective attraction of the eigencircuit or the effective attraction of the feedforward input mode.
In summary, we assumed that neurons are tuned to feedforward eigenvectors (Eq.135) and showed how the network decomposes into recurrent eigencircuits.We demonstrated how variances propagate through such eigencircuits, and quantified how eigencircuits modify the attraction of their feedforward eigenvector (cf.Sec. 3) by laterally projecting onto other neurons (cf.Fig. 4C).In the following (Sec.5), we will show that eigencircuits are indeed stable fixed points of fully plastic recurrent networks and investigate their stability.

A note on the choice of weight norm
The choice of the weight norm that is maintained via multiplicative normalization is non-trivial.Biologically we motivated normalization by the competition for a limited amount of synaptic resources.We assumed the simplest case, where the L1-norm is maintained, and each resource unit translates to one unit of synaptic strength.An alternative choice would be to maintain the L2-norm.In the variance propagation equation (Eq.140) this corresponds to p = 2 which becomes [148] Following a similar logic as in Section 4.2, the eigencircuit consistency condition for a single inhibitory neuron becomes (cf.Eq. 142): where we once more assumed that all neurons have similar weight norms, W i AB ≈ W AB , i.The variance of an excitatory neuron becomes For an increasing number of excitatory neurons n E , the firing rate variance of a single excitatory neuron grows and diverges for Φ 2 n E = 1.For even larger n E , variances would have to be negative to fulfil the consistency condition, Neurons that are part of the same eigencircuit are recurrently connected to each other.Synaptic weights between neurons that are tuned to different eigencircuits are zero.The excitatory neuron in eigencircuit A is perturbed in the direction of eigencircuit B (dashed lines).(D) Equivalent circuit with one excitatory and one inhibitory neuron.We consider a fixed point, where both neurons are tuned to the same feedforward eigenvector with eigenvalue λ * .The neurons form an eigencircuit with attraction λ * eig .The excitatory neuron is perturbed in the direction of another feedforward eigenvector with attraction λ † (dashed line).Firing rates, y E , y I , y, and recurrent and feedforward synaptic weights, w EE , w EI , w IE , w II , w EF , w IF , are shown in black (cf.Eqs.153 & 153).See text for details.
which is not possible.It follows that for sufficiently large n E there exist no fixed points.This is not unique to the L2-norm but holds for any p > 1.Such norms allow for a larger total synaptic weight (in terms of its L1-norm) when distributed across multiple synapses.Additional neurons provide additional recurrent synapses, which leads to the growth of the effective recurrent excitation until activities can no longer be stabilized by recurrent inhibition.For a suitable choice of the weight norms, Φ can, in principle, become small enough to balance the number of excitatory neurons in any eigencircuit to maintain positive variances.However, this requires additional fine-tuning and fails when n E becomes unexpectedly large.

E-I networks with fully plastic recurrent connectivity
We now consider fully connected networks of excitatory and inhibitory neurons where all connections, feedforward and recurrent, are plastic according to the competitive Hebbian learning rule we introduced in Section 2. We will first show that eigencircuits are fixed points and then consider their stability with respect to a weight perturbation.Specifically, we would like to know when a neuron from one eigencircuit becomes attracted to a different feedforward eigenvector.We start with some simplifying assumptions.
Since each neuron can be bidirectionally connected to all other neurons, the dimensionality of the weight dynamics grows quadratically with the number of neurons.We are only interested in the general principles and consider a simplified circuit of two excitatory and two inhibitory neurons.One possible fixed point configuration is shown in Figure S3C (without dashed lines), where neurons are configured in two eigencircuits, A,B, with one excitatory and one inhibitory neuron per eigencircuit 1 .In this fixed point, all neurons receive feedforward input from a population of input neurons but synapses that connect neurons of different eigencircuits are zero (cf.Sec. 4).When a neuron in eigencircuit A is perturbed towards the other eigencircuit B (Fig. S3C, dashed lines), the tuning and the firing rates of all neurons in eigencircuit A change.However, neurons in eigencircuit B are unaffected because there are no connections projecting from eigencircuit A to eigencircuit B. Therefore, we only consider the recurrence within eigencircuit A, and think of input from other eigencircuits as effectively feedforward and static: That is, we construct an equivalent circuit where we perturb an excitatory neuron that is part of an eigencircuit, ' * ', in the direction of another eigencircuit, ' † ', that does not contain any neurons and has feedforward attraction equal to the effective attraction of eigencircuit B, that is1 (Fig. S3D) The configuration and attraction of the perturbed eigencircuit ' * ' is equal to eigencircuit A, i.e., λ * = λ A , λ * eig = λ A eig .In Section 5.2.3 we will explain in more detail why these two circuits (Fig. S3C & D) are highly similar with regards to their stability.
In the equivalent circuit (Fig. S3D), we now consider the generic equilibrium firing rates of the n E = 1 excitatory and n I = 1 inhibitory neuron without taking any tuning into account (Fig. S3D) where y holds the firing rates of a population of N F input neurons and we did not assume any specific tuning of the feedforward weights w EF , w IF .Since the network is linear, we can solve for the firing rates: where we defined the effective feedforward vectors a E , a I .The weight dynamics is where ellipsis indicate similar terms for afferent weights of the inhibitory neuron.We define the modified covariance matrix and write the average synaptic change as2 (cf.Eq. 40) where Γ is a diagonal matrix that holds the scalar constraint factors, and τ holds the timescales for excitatory synapses, τ E = τ E , and inhibitory synapses, τ I = τ I , on the diagonal.We make the simplifying assumption that the plasticity of excitatory and inhibitory synapses is equally fast, τ E = τ I = τ.Then τ = τ , which does not affect the fixed points or the stability of the system3 , and we set τ = .
Note that this is a highly non-linear dynamical system since the modified covariance matrix not only depends on the feedforward inputs y but also on the plastic synaptic weights w, in addition to the weight dependence of the normalization factors Γ. Next, we show that the eigencircuit configuration we discussed in the introduction to this section is in fact a fixed point of the weight dynamics.

Fixed points
In general, fixed points w * must fulfil the following condition where C * is the modified covariance matrix evaluated in the fixed point.We consider the special case when the two neurons form a single eigencircuit, tuned to the feedforward eigenvector v * .Then we can write the excitatory and inhibitory firing rates as 1 where a E and a I depend on the excitatory and inhibitory weights and can be determined via Eq.155 & 156.This demonstrates that when neurons are tuned to the same feedforward eigenvector v * , their firing rate is proportional to the projection of the activity vector y onto the eigenvector v * , and justifies our assumption in Eq. 135.The modified covariance matrix in the fixed point becomes which can be diagonalized by the eigenvector matrix V * and its inverse: where the subscript (•) \ * indicates that a matrix does not contain an entry that corresponds to the input mode v * .In general, C * has one diagonal block of dimension D = N F + N E + N I per neuron in the circuit, i.e., N E + N I blocks 2 .Then, C * is of dimension Because C * has a block diagonal structure (Eq.163), with the first D × D block driving development of weights onto the excitatory neuron and the second D × D block driving development of weights onto the inhibitory neuron, the eigenvector matrix V * and its inverse have the same block diagonal structure.Since each block has the same sub-structure, we only show the first block in Eq. 164.Assuming that all neurons in the circuit are tuned to a feedforward eigenvector, we have N F + n E + n I eigenvectors of C * per eigencircuit and block, where n E and n I are the number of excitatory and inhibitory neurons in the respective eigencircuit: N F regular eigenvectors, and n E + n I null eigenvectors (cf.Sec.2.1.2).For the specific circuit at hand, we have one excitatory and one inhibitory neuron, N E = N I = 1, recurrently connected in the same eigencircuit, i.e., there are N F − 1 eigencircuits with n E = n I = 0 and one eigencircuit with n * E = n * I = 1.The first column of V * in Eq. 164 corresponds to the N F − 1 eigencircuits with n E = n I = 0, i.e., with one regular eigenvector per feedforward eigenvector v v * , but without null-eigenvectors.The corresponding eigenvalues are Λ \ * , which are also eigenvalues of the feedforward covariance matrix C. For the eigencircuit corresponding to v * there are 1 + n * E + n * I = 3 eigenvectors of C * .The first is a regular eigenvector and the last two are null eigenvectors, where the excitatory feedforward component is cancelled by either a negative lateral 1 Note that here the superscript ' * ' indicates a variable that is evaluated in the fixed point of the weight dynamics and not a fixed point of the firing rate activity.Different input patterns y result in different neural activities y * E , y * I . 2 Remember that N F is the number of input neurons, and N E , N I are the total excitatory and inhibitory neurons in the circuit, respectively.excitatory component 1 , or a positive lateral inhibitory component 2 .The null eigenvectors have eigenvalues equal to zero, and the eigenvalue of the regular eigenvector is λ * = λ * (1 + a * 2 E − a * 2 I ).Similar to the feedforward case, arbitrary multiples of the separately normalized parts of eigenvectors of C * are fixed points.The only exception is the rightmost null eigenvector (cf.Sec.2.1.3).There, the inhibitory and the excitatory weights are aligned such that the post-synaptic activity is zero, which does not allow for arbitrary scaling of the excitatory and inhibitory weight norms.Inserting these fixed points into Equations 155 & 156, provides conditions to determine a * E and a * I .

Stability analysis
We are interested in the stability of the circuit described in the introduction of Section 5 and consider the stability of a regular eigenvector v * This means we do not consider arbitrary scalings of the excitatory and inhibitory parts of eigenvectors of C * , but assume that weight norms are fine tuned to match the norms of the excitatory and inhibitory parts of the regular eigenvector3 v * .When are such eigenvectors stable, and when are they attracted to a different input mode?To answer this question, we consider small fixed point perturbations ∆w(t 0 ), where the excitatory neuron shifts its tuning in the direction of a different feedforward input eigenvector v † : [167] where e † is a vector of zeros with a single non-zero entry that corresponds to the feedforward eigenvector v † (cf.Eq. 164).The system is stable with respect to a perturbation if the perturbation decays to zero over time.To check this, we consider the following differential equation that holds for small perturbations (cf.Sec.1.2.2) where J * is the Jacobian evaluated in the fixed point.We will consider the dynamics in the non-orthogonal eigenbasis V * of the modified covariance matrix C * evaluated in the fixed point w * = v * .Note that V * is not time-dependent, 1 In our simulations, we constrain synaptic weights to be positive.Then null eigenvectors with negative weights are only relevant in combination with regular eigenvectors: When a null eigenvector is added to a regular eigenvector, the net synaptic input does not change.For example, a decrease in recurrent excitation due to a negative excitatory component of the null eigenvector is balanced by an increase in feedforward excitation. 2 We can generalize this approach to the case where neurons are tuned to different feedforward eigenvectors.For example, consider we add a second excitatory neuron that is, however, tuned to a different feedforward eigenvector, v † .This gives rise to an additional null eigenvector, (v † a † E , 0, 0, −1, 0 ) , in the first block of V * (Eq.164).In addition, one of the regular eigenvectors in the first column block of V * (Eq.164) becomes (v † , 0, 0, a † E , 0 ) .Importantly, this is the case for each diagonal block of V * , i.e., we get D additional null eigenvectors and D altered regular eigenvectors per additional neuron.This ensures that we always have N E + N I null eigenvectors and N F regular eigenvectors per block, which allows to diagonalize C * which is of dimension (N E + N I )D × (N E + N I )D, independent from the feedforward tunings of neurons -with the caveat that all neurons must be tuned to feedforward eigenvectors.
because it is evaluated in the fixed point.In this static basis, we can express perturbations as [170] where the subscript (•) v indicates a vector or matrix expressed in this basis.The perturbation dynamics becomes where we defined the transformed Jacobian, J * v = V * −1 J * V * .Without loss of generality, we assume that eigenvectors in V * are sorted such that the first entry of e † is non-zero, i.e., the first column of V * is proportional to the initial perturbation ∆w(t 0 ) (cf.Eq. 167).Next, we will derive the transformed Jacobian.

The transformed Jacobian
First, we consider the regular Jacobian J * .We rewrite the dynamics in Eq. 159 as 1 where the second term in the bracket corresponds to the normalization of all excitatory synapses onto the excitatory neuron, additional normalization terms are indicated by ellipsis 2 (cf.Eq. 45), and c is a vector of ones.Then the Jacobian has the following shape (cf.Eq. 29) where v * EF , v * EE have the same shape as w EF , w EE in Eq. 172 with entries corresponding to the respective entries of the regular eigenvector v * (cf.Eq. 166).Note that we accounted for the weight dependence of the modified covariance matrix C which results in the tensor dC /dw.To find the transformed Jacobian V * −1 J * V * , we consider the first bracket: The first entry remains equal to the identity matrix, as the eigenvector matrix and its inverse cancel.We consider the columns v * b of V * separately.Then, we can write 1 Remember that we set τ = . 2In general, there are 2 × (n E + n I ) normalization terms.
where H b is a diagonal matrix with entries corresponding to the respective normalization constraint, of which we give h b EE and h b EI as examples.Then each column v * b of V * is transformed into a multiple of the separately normalized parts of the fixed point eigenvector v * (Eq.166).After transformation, the bth column becomes where, as before, ellipsis indicate potentially non-zero entries.Importantly, after the transformation, the first N F − 1 entries are zero, independent of the column index, b, because v * is orthogonal to the columns of V \ * .Overall, we can write where the block structure arises from the block structure of V * −1 (cf.Eq. 177).
After transformation, the second bracket of Eq. 173 becomes We next consider the first columns of dC dw * w * , for which we compute the matrix dC , where w b EF is the bth feedforward weight onto the excitatory neuron.
where we used the definition of C from Eq. 158.The vectors a E and a I are defined in Eq. 155 & 156.It follows: where e b is a vector of dimension N F with entries equal to zero, except for the bth entry equal to one.Additionally, we have (cf.Eqs.161 & 162) where g (•) are scalars.
We find other columns in a similar fashion and write where, again, g (•) are scalars.After applying the transformation, we get The fully transformed Jacobian is (cf.Eq. 179) Finally, by inserting Eq. 178 & 192 we find where Λ \ * contains eigenvalues of C * that correspond to regular, non-fixed point eigenvectors.1

Stability conditions
The dynamics of a general fixed point perturbation ∆w v in the eigenbasis of C * is (cf.Eq.171) Note that the transformed Jacobian (Eq.195) has a triangular block structure, and each row of J * v corresponds to the growth of a perturbation in the direction of a different eigenvector of C * .We are only interested in perturbations that grow in the direction of a non-fixed point feedforward eigenvector, V \ * .Therefore, we focus on the first rows of J * v , which correspond to growth in these directions.Except for the first diagonal block, these rows are zero.It follows that perturbations ∆w v (t 0 ) that do not already contain components in the direction of non-fixed point eigenvectors, also do not develop such components in their later dynamics.In contrast, perturbations in the direction of a nonfixed point feedforward input mode, e.g., ∆w v ∝ e † , can induce perturbations within the original eigencircuit that corresponds to the feedforward eigenvector v *2 .For example, a decrease in feedforward and recurrent excitatory synaptic weights within the eigencircuit balances the increase of feedforward excitatory synaptic weights due to the perturbation towards a different eigencircuit, to maintain the weight norm.However, as explained above, these 'second-order' perturbations, without components in the direction of non-fixed point feedforward eigenvectors, V \ * , are contained within the eigencircuit, i.e., they can not induce subsequent perturbations in the direction of non-fixed point feedforward input modes, V \ * .Therefore, to answer the question of when an eigencircuit is stable, we only consider the dynamics along the direction of the original perturbation (cf.Eq.167) by projecting the dynamics onto the perturbation vector at time zero, ∆w v (t 0 ) ∝ e † (cf.Eq.170): [197] which provides the eigencircuit stability condition for the excitatory neuron If Eq. 198 holds, perturbations in the direction of non-fixed point eigenvectors decay to zero, and the eigencircuit is stable.For β E we have (cf.Eqs.186 & 182 f.)Following the same framework, we find the stability condition when perturbing the inhibitory neuron: [204] We will now interpret these results.

Eigencircuit stability depends on recurrent connectivity
We first consider the case when the effective attraction of all eigencircuits is the same, i.e., λ * = λ † (cf.Eqs. 198 & 203).Then the stability is fully determined by β E , and β I .In feedforward circuits we have not found any β -terms, because in that case, the modified covariance matrix does not depend on any plastic synaptic weights (cf.Eq. 51 & Sec.2.2).This is not the case in recurrent circuits where the perturbation induces a change in the tuning of laterally projecting neurons.
To build some intuition, we consider a simple example: Think of a recurrent network of two excitatory neurons with identical weight norms , and an external population of excitatory neurons projecting feedforward input to both.In the fixed point, the neurons are tuned to two different feedforward input eigenvectors of equal attraction and are recurrently connected to themselves but not each other (Fig. S4A).Then the effective attraction of the two eigencircuits is the same.In general, neurons receive synaptic inputs, but have no information about the overall network structure, e.g., which synaptic inputs are feedforward or recurrent.Taking this perspective, we unfold the recurrent network and observe that the effective mode attraction is a combination of the feedforward input and the recurrent self-excitation (Fig. S4B).When we perturb one neuron towards the opposing input mode (Fig. S4C, dashed lines), the tuning of the perturbed neuron changes slightly in the direction of that mode (Fig. S4C, dark blue).From the perspective of the perturbed neuron, this tuning change leads to an attraction increase of the opposing eigencircuit, which is now more attractive than the original eigencircuit of the perturbed neuron (Fig. S4D), and the perturbation grows in the direction of the more attractive mode -the fixed point is unstable.Similarly, if the neurons were inhibitory instead, the perturbation would decrease the attraction towards the opposite input mode which would stabilize the network.
In our mathematical analysis of the circuit shown in Figure S3D, the attraction increase or decrease due to the tuning change of recurrently projecting neurons is reflected in the β -terms in Equations 198 & 203, which emerge from the weight dependence of the modified covariance matrix C (cf. Eq. 173).For example, when perturbing the excitatory neuron, the increase in attraction from the perspective of the perturbed neuron is (cf.Eq. 198) where the brackets reflect the tuning shifts of the excitatory and the inhibitory neuron1 in response to the perturbation of w EF in the direction of v † , which is then weighted by the respective synaptic connection onto the excitatory neuron, w * EE , w * EI .When the inhibitory neuron is perturbed instead, the terms for β I follow the same logic (cf.Eq. 204).Without going through the lengthy mathematical derivation, we now give some intuition about β -terms of the network perturbation in Figure S3C.In the fixed point, the perturbed excitatory neuron receives recurrent input from all neurons in its eigencircuit, including itself.In the following, superscripts indicate the corresponding eigencircuit, A or B.Then, as for the equivalent circuit (cf.Fig. S3D), β A E comprises two terms, one due to the tuning shift of y A E , and a second due to the tuning shift of y A I .Assuming the same weight norms, this is exactly equal to the β E for the equivalent circuit (Eq.198).Different from β E , β A E is weighted with the effective attraction λ B = λ B + λ B eig , instead of only the feedforward attraction (cf.λ † in Eq. 205), because the perturbation comprises not only the feedforward eigenvector component but the whole eigencircuit (cf.dashed lines in Figs.S3C & D).This is why, for the equivalent circuit, we chose the feedforward attraction λ † = λ B = λ B + λ B eig (Eq.152).Then, the diagonal entries corresponding to the respective perturbations in the upper left blocks of the transformed Jacobians are the same2 (cf.Eq. 195), i.e., [206] We find that perturbations in both circuits initially follow the same dynamics, while the later dynamics diverges: At time t 0 , there are no lateral projections from eigencircuit B towards eigencircuit A (cf. Fig. S3C), since in the fixed point there are no recurrent connections between eigencircuits (cf.Sec.4), and the perturbation at time t 0 only introduces connections from eigencircuit B onto eigencircuit A. However, as we just discussed, the perturbation introduces a tuning shift in neurons of eigencircuit A in the direction of eigencircuit B. This shift leads to non-zero correlations between neurons of both eigencircuits, and synaptic weights from eigencircuit A onto eigencircuit B start to grow.These growing synapses shift the attraction of neurons in eigencircuit B and thus impact the dynamics of perturbation components in the direction of eigencircuit B. Therefore, the transformed Jacobian of the original circuit (Fig. S3C) has a more complex structure than the Jacobian for the equivalent circuit 3 .However, since we consider an initial perturbation that is aligned with a regular eigenvector, i.e., ∆w v (t 0 ) ∝ e B is one-hot (cf.Eq. 170), the top left diagonal block of the Jacobian still determines the initial dynamics 4 .

Supplementary Material | PNAS
In summary, recurrent synapses can stabilize or destabilize a circuit with respect to small perturbations away from a fixed point.These stabilizing and destabilizing effects are described by β -terms that depend on the specific weight configuration in the fixed point (cf.Eq. 199), which again depends on the weight norms that constrain the total synaptic weights.In the following, we consider the case when synaptic weights are tuned such that β -terms are small.In the equivalent circuit (Fig. S3D) this is the case when the influence of the tuning shifts of the excitatory and the inhibitory neuron balance each other1 (cf. the first and second terms in Eqs.202 & 204).

Decorrelation condition
We now consider how neurons self-organize to represent all parts of their input space instead of clustering all their tuning curves around a dominant input mode.We consider the fixed point stability of different eigencircuit configurations.In particular, we consider the case when recurrent connectivity motifs do not influence the stability of an eigencircuit.The β -terms in Equations 198 & 203 describe the change in the covariance structure of the network due to a small perturbation (cf.Sec.5.2.3).Since we consider the stability of a single neuron in a larger network of many neurons, N E , N I 1, these changes in the covariance structure are small, and the dynamics is dominated by the total attractions of the eigencircuits.Therefore, in the following, we consider β E and β I to be small, approximately equal to zero This can be achieved by a suitable choice of weight norms 2 .Then, all eigencircuits are marginally stable when they are equally attractive, i.e., (cf.Eqs.198 & 203 for β E/I = 0) a, b. [207] For homogeneous input spaces, where the feedforward attraction of all input modes is the same, i.e., λ a = λ b = λ, a, b, the only alternative stable configuration is when all neurons are tuned to the same feedforward input mode and form a single eigencircuit.Such a configuration does not reflect the tunings of biological neural populations, where all parts of the input space are represented.To prevent such a global clustering of neural tunings, we require that the corresponding eigencircuit is unstable.When all β -terms are approximately zero, this is the case when the effective attraction of the only occupied eigencircuit, λ * , is smaller than the attraction of one of the N F −1 unoccupied 3 input modes, λ † = λ † (cf.Eq. 198 & 203): ⇒ N E σ 2 E − N I σ 2 I < 0, [210] where σ 2 E , σ 2 I are the average variance, and N E , N I the total number of inhibitory and excitatory neurons.When this condition is satisfied, the only stable solution is when the effective attraction of all eigencircuits is identical.The simplest configuration where this is the case is when each eigencircuit contains the same number of excitatory and inhibitory neurons.

Figure S2 :
Figure S2: Input space modification due to lateral input.(A) Top: a single neuron with firing rate r receives synaptic inputs w from a population of excitatory neurons y.Bottom: input distribution projected onto the first two input dimensions.Each dot represents the firing rates of the first two neurons during one input pattern.(Contour lines in light gray).Under a linear Hebbian learning rule, the neuron becomes selective for the direction of maximum variance, the first principal component (cf.Sec. 1).(B) Top: Same as in A for a neuron that receives additional input w q from a laterally projecting excitatory neuron r q which is tuned to an eigenvector q of the original input covariance matrix.Bottom: the effective input space y eff of the target neuron (dark blue triangle) is warped such that the variance along the eigenvector q (blue arrow) is stretched in proportion to the absolute value of the weight vector q.The contour lines of the original input distribution from A are shown in light gray for reference.(C) Top: Same as B for a laterally projecting inhibitory neuron.Bottom: Now, the effective input space is compressed.See text for details.

Figure S3 :
Figure S3: (A)A neuron with firing rate r (gray, center) receives synaptic inputs as part of a recurrent eigencircuit.The neuron receives excitatory synapses w from a population of input neurons y (dark purple, bottom).Excitatory (purple, triangles) and inhibitory neurons (light purple, circles) with firing rates, y E , y I , that are part of the same eigencircuit, project laterally onto neuron r via excitatory w E and inhibitory w I synapses.Recurrent synaptic connections that are not inputs of neuron r are shown in light gray -Not all synaptic connections are shown, for clarity.(B) Recurrently connected eigencircuit of n E = 1 excitatory neuron (purple triangle) and n I = 2 inhibitory neurons (light purple circles) that are tuned to the same feedforward eigenvector (dark purple circle, bottom).The standard deviation σ of input firing rates along the input eigenvector propagates through the network and results in firing rate standard deviations of σ E and σ I (cf.Eq. 145 f.).(C) Two excitatory neurons (triangles, top) and two inhibitory neurons (circles, top) in a recurrent circuit receive feedforward excitation from two input neurons (purple and green circles, bottom) that correspond to two different eigenvectors with eigenvalue λ A , λ B .Neurons are configured in a fixed point with two eigencircuits, A and B, with eigencircuit attractions λ A eig , λ B eig (cf.Eq. 147).Neurons that are part of the same eigencircuit are recurrently connected to each other.Synaptic weights between neurons that are tuned to different eigencircuits are zero.The excitatory neuron in eigencircuit A is perturbed in the direction of eigencircuit B (dashed lines).(D) Equivalent circuit with one excitatory and one inhibitory neuron.We consider a fixed point, where both neurons are tuned to the same feedforward eigenvector with eigenvalue λ * .The neurons form an eigencircuit with attraction λ * eig .The excitatory neuron is perturbed in the direction of another feedforward eigenvector with attraction λ † (dashed line).Firing rates, y E , y I , y, and recurrent and feedforward synaptic weights, w EE , w EI , w IE , w II , w EF , w IF , are shown in black (cf.Eqs.153& 153).See text for details.

Figure S4 :
Figure S4: (A) Two excitatory neurons (triangles) are tuned to two different, but equally attractive input modes (circles, green and purple).(B)The same circuit as in A, unfolded to highlight pre-synaptic partners.Both input modes are balanced in their attraction.(C) Perturbing the purple excitatory neuron towards the green input mode (dashed lines) shifts its tuning (dark blue) such that it now responds to both the green and the purple input modes.(D) The unfolded circuit from C. Due to the perturbation, the green input mode is now more attractive, and the previously purple excitatory neuron shifts its tuning.See text for details.

Movie S1 :
Decorrelation of feedforward tuning curves of excitatory neurons in plastic recurrent networks.Development of feedforward tuning curves of N E = 10 excitatory neurons (cf.Figs.3A & B).Synaptic weights were initialized randomly.Different color shades indicate weights of different post-synaptic neurons.