About the de Almeida-Thouless line in neural networks

.


Introduction
Replica symmetry breaking in neural networks has attracted increasing attention in recent years [4,12,13,31,43], however there is, as yet, no general broken replica-symmetry theory for these systems and no simple method to systematically detect their transition from the replica-symmetric (RS) to the replica-symmetry-broken (RSB) phase.While the first point is still out of reach, the second question can be addressed by adapting approaches originally developed for spin glasses.Indeed, the instability line of the RS phase in the Sherrington-Kirkpatrick (SK) spin glass model was derived by de Almeida and Thouless (AT) many decades ago [25], using a method based on replicas.Since their seminal work, rigorous techniques have been developed and tested in archetypical mean-field as well as short-ranged spin glass models, by many researchers (see e.g.[14,17,18,30,32,34,38,39]).As neural networks are particular realizations of spin glasses, it is quite natural to ask if we can devise a systematic method to derive the RS instability line also for these systems.In this work we answer affirmatively to this question, using the Hopfield model of neural networks and a model of dense associative memory, which extends Hebbian learning to multi-node interactions, as case studies.To this purpose, we devise a method inspired by the approach proposed by Toninelli in [41], which builds on Guerra's work on broken replica-symmetry bounds [29].As a technical note we remark that at difference with conventional spin glasses, here we focus on the RS instability in the parameter space (α, T ) where α is the storage load of the network and T is the noise level, rather than in the space (h, T ) (i.e.magnetic field, temperature) conventionally used in spin glasses.
For the Hopfield model, our method recovers the instability line obtained by Coolen [19] using the AT approach, as a special limit, which corresponds to assuming a continuous transition from the RS to the RSB phase, in the order parameters.Therefore, our method provides a generalization of the AT approach, which can be applied to systems with a discontinuous transition from the RS to the 1RSB phase.Another advantage of our method, when compared to the involved calculations of the AT method in the Hopfield model [19], is its remarkable simplicity.This allows for straightforward application to more complex neural network models, such as dense associative memories with P -node interactions.We supplement the results in the main text with appendix A where we show our method at work on conventional spin-glass models, namely the Sherrington-Kirkpatrick model, the Ising Pspin and the spherical P -spin model, the latter providing an example of system which exhibits a discontinuous phase transition from the RS to the RSB phase.In all cases, we retrieve the AT lines known for these models [19,24,25,27] in a specific limit, confirming the validity of our approach as a generalization of the AT method.As expected from the decomposition theorem of multi-node Hebbian networks proved in [4], for dense associative memories with P -node interactions we retrieve the instability line of the Ising P -spin model derived in Appendix A. Appendices B and C provide further technical details.

The Hopfield model
In this section we illustrate the method for the Hopfield model with N Ising neurons σ i ∈ {1, −1}, i = 1, . . ., N and K = αN stored patterns ξ µ , µ = 1, . . ., K. Each pattern ξ µ is a sequence of N Rademacher entries (i.e.Bernoulli variables) ξ µ i , i = 1, . . ., N , with distribution The Hamiltonian of the model is and we denote the associated Boltzmann factor, at inverse temperature β = 1/T , as In the so-called 'retrieval' phase, the equilibrium local configurations are correlated only with a single pattern, say ν.As the couplings J ij = K µ=1 ξ µ i ξ µ j are symmetric w.r.t.permutations of the patterns, it is assumed without loss of generality that ν = 1.It is then convenient to define as the order parameters of the system, the so-called Mattis magnetization which quantifies the alignment of the system configuration σ with the retrieval pattern ξ 1 , and the two-replica overlap q(σ (1) , σ (2) ):= 1 (2) i (2.5) which quantifies the correlations between two configurations σ (1) , σ (2) of the system, with the same realization of the patterns (i.e.quenched disorder).
The RS analysis assumes that the order parameters m and q self-average around their equilibrium values m and q, in the thermodynamic limit, namely lim lim where 2) |ξ)δ(q − q 12 (σ (1) , σ (2) )), with E ξ denoting the expectation over the pattern distribution (or 'quenched' disorder).Under this assumption, the free-energy, averaged over the pattern distribution, f , is given by (see [5]) where z is a random Gaussian variable with zero average and unit variance, E denotes the average over z and α is the load capacity of the network.In this limit, the order parameters q and m fulfill the celebrated Amit-Gutfreund-Sompolinsky self-consistency equations [5,20]: (2.10) On the other hand, within one step of the replica-symmetry breaking (1RSB) scheme [1,21,37] it is assumed that the distribution of the two-replica overlap q, in the thermodynamic limit, displays two delta-peaks at the equilibrium values q0 and q1 > q0 and the concentration on these two values is ruled by the parameter θ ∈ [0, 1], while m self-averages as in the RS case : lim lim Within this assumption, the disorder-averaged free-energy is given by (see e.g.[1]) where, for mathematical convenience, we defined and we have denoted with E 1 , E 2 the averages w.r.t. the standard normal variables z (1) and z (2) , respectively.From now on, we imply the dependence of all the functions on β and α.By extremizing the 1RSB free-energy w.r.t.its order parameters q0 , q1 , m1 , it is possible to show that the latter fulfill the following self-consistency equations , (2.17) The key idea of our method is to assume that at the onset of the RS instability, one of the two deltapeaks in equation (2.12) has vanishing weight, i.e. either θ → 0 or θ → 1. Consistency with the RS theory then requires the dominating peak to be located at the value q of the RS order parameter, so either lim θ→0 q1 = q or lim θ→1 q0 = q.As we will see below, both these relations are generally satisfied, hence we appeal to the physical interpretation of RS breaking to determine which scenario applies.Noting that when the RS theory becomes unstable, a multiplicity of states emerges with mutual overlap q0 and self-overlap q1 , within the single state (or each of the states) predicted by the RS theory, it is natural to assume that the value of the mutual overlap between the newly born states is equal to the self-overlap q of the state(s) assumed by the RS theory.Thus, we will assume that at the onset of the RS instability, q0 → q, hence θ → 1.For the Hopfield model, taking this limit in (2.17), and using we get lim (1)  (2.19) where in the second line we have used sinh(A + B) = sinh A cosh B + sinh B cosh A and cosh(A + B) = cosh A cosh B + sinh B sinh A (A denoting the first two terms on the RHS of (2.18) and B the last one) and have performed the integral over z (2) using the oddity of the sinh function.As (2.19) is identical to (2.9), in the limit θ → 1, q0 is indeed equal to the RS order parameter q, as anticipated.Similarly, we can show that and one can easily verify that f 1RSB ( m1 , q1 , q0 |θ)| θ=1 = f RS ( m, q), as expected from the fact that, for θ = 1, Eq. (2.12) reduces to (2.7) and one retrieves the RS scheme.Our purpose is then to prove that for values of θ close but away from one, the 1RSB expression of the quenched free-energy is smaller than the RS expression, i.e. f 1RSB ( m1 , q1 , q0 |θ) < f RS ( m, q), below a critical line in the parameters space (α, β).

.34)
Figure 1: RS overlap q (top row) and the difference between q1 ( m, q) and q (bottom row) versus the scaled parameter T / √ α, for different values of α (as shown in the legend), for the Hopfield model (left) and Hebbian networks with P -node interactions, for P = 4 (mid) and P = 6 (right).
While this limit is a priori unjustified, as q and q1 ( m, q) should be solved from the self-consistency equations (2.9) and (2.25), respectively, one can check numerically that the solutions of these equations are virtually indistinguishable for any temperature (see Fig. 1, left panel), and the resulting RS instability line is almost identical to the AT line derived in [19], see Fig. 2 (left panel).Small deviations can be appreciated in the retrieval region (see right panel), but these are likely due to numerical precision.As our method does not rely on the assumption q1 ( m, q) → q, it provides a more general approach than the one originally devised by de Almeida and Thouless, that can be carried over to systems with discontinuous phase transitions, where q1 differs from q0 even at the onset of the RS instability, implying q1 ( m, q) ̸ = q.Before concluding this section, we note that, although we have disregarded the limit θ → 0 as lacking physical interpretation, such limit is still well defined mathematically and one may ask what would be the outcome of a similar analysis carried out in this limit.We will perform such analysis in Appendix B. Intriguingly, we find that the analysis for θ → 0 gives the same instability line as the analysis for θ → 1, in all the models we considered, except in the spherical P -spin model.Reassuringly, in the latter case, the analysis at θ → 0 leads to a lower temperature for the RS instability, confirming that the limit θ → 1 gives the physical transition and it is therefore the relevant one.

Hebbian networks with P -node interactions
In this section we consider generalizations of the Hopfield model, where neurons interact in P-tuples of even P ≥ 4 (rather than pairwise, i.e.P = 2).Such networks were shown to store many more patterns T c versus α) obtained via our method (blue crosses) and the AT line obtained in the limit q1 ( m, q) → q (red dashed curve), for the Hopfield model model.The several branches separate the paramagnetic (P), spin-glass (SG) and retrieval (M) region.The black curve show the critical temperature T ⋆ at which q becomes non-zero within the RS theory, i.e. the spin-glass (SG) transition.
than the number of their nodes, so they work as dense associative memories [33].They are also known to be dual to deep neural networks [3,4] and to exhibit information processing capabilities that are forbidden in shallow networks, such as the existence of a region in the parameter space where they can retrieve patterns although these are overshadowed by the noise [2].As before, we consider a network of N interacting Ising neurons σ i ∈ {1, −1}, with K stored patterns ξ µ and P -node interactions J i1...i P = 1 The Hamiltonian of this model can be written as The order parameters are still the Mattis magnetization m and the two-replicas overlap q, as introduced in (2.4), with their RS distributions given in (2.6) and (2.7) and their 1RSB generalizations given in (2.12) and (2.11).The quenched free-energy in RS assumption is (see [28]) with β ′ := 2β/P !, where β is the inverse temperature and α = lim N →∞ K/N P −1 is the network load.E is the average w.r.t. the standard Gaussian random variable z, and q and m satisfy the self-consistency equations: On the other hand, the quenched free-energy within the 1RSB approximation (see [4]), reads as where E 1 , E 2 are the average w.r.t. the standard normal random variables z (1) and z (2) , respectively, and In this approximation the self-consistency equations for the order parameter q1 , q0 and m are as in (2.17), with the argument of the hyperbolic cosine and tangent replaced by (3.5).As before, for θ = 1, q0 = q and the 1RSB expression for the quenched free-energy reduces to the RS one.
From now on, we imply the dependence of the functions on β ′ and α.Our objective is to prove that for θ close but away from one, the 1RSB quenched free-energy is smaller than its replica symmetric counterpart i.e. f 1RSB ( m1 , q1 , q0 |θ) < f RS ( m, q) above a critical value of the effective parameter √ αβ ′ .To this purpose, we proceed as in the Hopfield model: we expand, to the leading order in θ − 1, the 1RSB quenched free-energy around its RS expression, as shown in (2.21).Since the self-consistency equations also depend on θ, we need to expand them too.Following the same steps as in the Hopfield model, we can write m1 as in (2.28) with C( m, q, q1 ( m, q)) as in (C.9),where q1 ( m, q) is the solution of the self-consistency equation q0 as in (2.27), with A( m1 , q, q1 ( m, q)) given in (C.7), and q1 as given in (2.26) with B( m, q, q1 ( m, q)) given in (C.8).With the above expressions in hand, we can now calculate the derivative of f 1RSB w.r.t.θ when θ = 1, as needed in (2.21) Again, we have that K( m, q, q) = 0, regardless of the value assigned to m, (this follows from the fact that for θ = 1, q is an extremum of the free-energy).Next, we inspect the sign of K( m, q1 ( m, q), q).To this purpose, we study K( m, q, x) for x ∈ [0, q] and locate its extrema, which are found from where the last equality follows from (3.6).Under the assumption that the extremum x = q1 ( m, q) is global in the domain considered and reasoning as in the Hopfield case, we have that K( m, q1 ( m, q), q) > 0 if x = q1 ( m, q) is a maximum and K( m, q1 ( m, q), q) < 0 if it is a minimum.In particular, if is positive, K( m, q1 ( m, q), q) < 0 and f 1RSB < f RS .This happens when the expression in the curly brackets of the equation above is negative, i.e. when the parameter αβ satisfies the inequality As noted in [4,15], Hebbian networks with P -node interactions are equivalent to Ising P -spin models under a suitable rescaling of the temperature β ′ √ α → β ′ .With this rescaling, (3.11) retrieves indeed the RS instability line of the Ising P-spin model, that we have for completeness derived in Appendix A, using our method (see eq. (A.30)).In the limit q1 ( m, q) → q, (A.30) retrieves the AT line of the Ising P -spin model [27].In Fig. 1 (mid and right panels) we plot the difference between q1 ( m, q) and q for Hebbian networks with P -node interactions (obtained solving numerically the self-consistency equations (3.6) and (3.3)) as a function of the scaled parameter T / √ α, for different values of α.As for the Hopfield model, we find that q1 ( m, q) is indistinguishable from q, hence the limit q1 ( m, q) → q can be justified a posteriori.T c versus α) obtained via our method, i.e. by expanding the free-energy around θ = 1 (blue dots) and the AT line obtained in the limit q1 ( m, q) → q (red curve), for the Hebbian network with P -node interactions, with P = 2, 4, 6 from left to right.The black curves show the critical temperature T ⋆ at which q becomes non-zero within the RS theory, i.e. the spin-glass (SG) transition.
In Fig. 3 we show the RS instability lines resulting from our method and the classic AT line obtained in the limit q1 ( m, q) → q, for different values of P .The two lines coincides for all values of P .As explained earlier, we could have expanded the free-energy around θ = 0 (as opposed to θ = 1).In Appendix B we show that such analysis leads to the same line.

Discussion
In this work we proposed a simple and systematic method to derive the critical line in the parameter space (α, β), below which the 1RSB expression for the free-energy is smaller than the RS expression, in Hebbian neural networks.The same analysis for spin-glass models is carried out in appendix A. For the Hopfield model, our approach recovers the critical line obtained by Coolen using the AT approach [19] as a special limit.Similarly, we recover the known AT lines of all the spin-glass models considered in the appendix, in the same limit, showing that our method provides a generalization of the approach originally devised by de Almeida and Thouless.Owing to its simplicity, our method allows for straightforward application to Hebbian networks with multi-node interactions, for which the AT-line was unknown.
The key idea of our method is to regard the 1RSB theory, which assumes two delta-peaks in the overlap distribution P (q), located at q1 and q0 < q1 , with weights 1 − θ and θ, respectively, as departing continuously from the RS theory, which assumes only one peak at q.This leads us to assume that at the onset of the RS instability, the 1RSB overlap distribution is dominated by one peak, so that either θ → 0 or θ → 1.Then, consistency with the RS theory requires either lim θ→0 q1 = q or lim θ→1 q0 = q.As the physical interpretation of RS breaking suggests that the mutual overlap between 1RSB states should be equal to the self-overlap of the RS states, we regard the 1RSB theory as a continuous variation of the RS theory, when the parameter θ is decreased from one.Crucially, we do not make any assumption on the location of the peaks of the 1RSB theory, which are fixed by the 1RSB self-consistency equations.We then compare the 1RSB and the RS free-energies when θ (the only free-parameter in our analysis) is close to one, by performing simple expansions to linear orders in θ − 1.In doing so, we solely require that f 1RSB , q1 , q0 and m1 are differentiable up to the first order in a neighborhood of θ = 1 and that the derivative of f 1RSB exists at θ = 1.Although our method is similar in spirit to the one introduced by Toninelli in [41], there is a crucial difference, in that the latter relies on the assumption q1 → q, which is, in our view, unjustified a priori.In fact, while q0 = q for θ = 1, q1 may differ from q, even in the limit θ → 1.This consideration also leads to a departure of our approach from the method originally devised by de Almeida and Thouless, which relies on a variation of the RS free-energy as the order parameters are varied continuously around their RS values.In contrast, we study the variation of the RS free-energy as the statistical weight of the order parameters is varied continuously (rather than the actual value of the order parameters).This approach allows us to determine the instability line of the RS theory also in spin-glass models which exhibit a discontinuous phase transition.As a prototypical example of this class of models, we consider in Appendix A the spherical P -spin model [21].In conclusion, in this work we have presented a new method to find the instability line of the RS approximation.A compelling advantage of our method, when compared to the approach by de Almeida and Thouless [25], is that it does not require to compute the full eigenspectrum of the Hessian of the quadratic fluctuations of the free-energy around its RS value and it does not rely on the availability of an "ansatz-free" expression for the free-energy.This makes the computations easier and affordable also for neural networks.The method still requires the availability of explicit expressions for the RS and 1RSB free-energies, which however can be computed using different techniques (e.g.Guerra's interpolation) in addition to the replica trick.Our approach can in principle be extended to compute the stability of the k-RSB solution, expanding the corresponding k + 1-RSB free-energy, provided one has an explicit expression for the two.For example, it would be interesting to see whether the well-known transition from 1RSB to full-RSB occurring at the Gardner temperature [27] in the Ising P -spin model can be recovered within our approach, by studying the stability of the 2RSB solution.In addition, in recent years there has been a boost of renewed interest in mixed P -spherical models, as introduced in [6,22], as they have been shown to display unexpected dynamical behaviour [26] and new types of spin-glass phases [23], [42] as well as for their relevance to the modelling of random lasers [7][8][9][10][11].These models similarly display transitions from 1RSB to full-RSB and would be good lab systems to test extensions of our theory.Another interesting avenue for future work would be a generalization of the 1-RSB scheme, which does not assume the Mattis magnetization to be selfaveraging.Preliminary work in [1] suggests that the RS assumption is not the right approximation for Mattis magnetization.Finally, an attractive perspective would be to apply our approach to predict the onset of ergodicity breaking in systems with sparse interactions.In such systems, the free-energy is typically expressed, already at the simplest RS level, in terms of order-parameter functions to be determined self-consistently.At 1RSB level, recursive equations for functional distributions of such functions must be solved.Working out the fluctuations of ansatz-free free-energies around their RS value, as it would be required by the de Almeida and Thouless approach would be unfeasible and no similar approach has been proposed to date.An alternative approach has been devised in [40], for the Bethe lattice with regular degrees, however it strongly relies on the assumptions of homogeneity in the network nodes and large degrees.We envisage that, owing to its simplicity, our method may carry over to more general sparse systems, where it would require an expansion of the functional 1RSB self-consistency equation to linear orders around the RS equation, which should be feasible.

A Applications to Spin-glasses models
In this appendix, we derive the critical line for the instability of the RS theory for three archetypical spin-glass models, namely the SK model, the Ising P -spin and the spherical P -spin model, using the technique developed in the main text.In all cases, we will recover the AT lines known in the literature for the three models, in a specific limit, confirming the validity and higher generality of our approach.

A.1 Sherrington-Kirkpatrick Model
The Sherrington-Kirkpatrick (SK) model [36] is a system of N Ising spins σ i ∈ {±1} interacting via symmetric pairwise interactions J ij which are i.i.d.Gaussian variables with zero average and variance J 2 /N .The Hamiltonian of the model is and the order parameter is the two-replica overlap q as defined in (2.4).The quenched free-energy in RS assumption is [36] −βf RS (q|β, J) = ln 2 + where E is the average w.r.t. the Gaussian variable z and the order parameter q fulfills the selfconsistency equation The 1RSB approximation of the quenched free-energy is (see e.g.[35]) where and E 1 , E 2 are the average w.r.t. the standard Gaussian variables z (1) and z (2) , respectively.From now on, we imply the dependence of g, f RS and f 1RSB on β and J.The order parameters q0 and q1 fulfill the following self-consistency equations Noting that lim θ→1 q0 = q and f 1RSB (q 1 , q0 |θ = 1) = f RS (q), our objective is to expand the 1RSB quenched free-energy for θ ≃ 1.To this purpose, we expand the self-consistency equations for q0 and q1 to linear order in θ − 1. Proceding as in the Hopfield model, we get q0 = q + (θ − 1)A(q, q1 (q)) (A.7) q1 = q1 (q) + (θ − 1)B(q, q1 (q)) (A.8) where q1 (q) solves the self-consistency equation q1 = E 1 E 2 cosh g(q 1 , q) tanh 2 g(q 1 , q) E 2 cosh g(q 1 , q) (A.9) and A(q 0 , q1 ) and B(q 0 , q1 ) are given in (C.13) and (C.14), respectively.Next, we derive w.r.t.θ the 1RSB free-energy (A.4) where we replace q0 and q1 with (A.7), (A.8), obtaining B(q 1 (q), q)q + 1 θ E 1 E 2 cosh θ g(q 1 (q), q) log cosh g(q 1 (q), q) E 2 cosh θ g(q 1 (q), q) + β 2 J 2 2 B(q 1 (q), q)θE 1 E 2 cosh θ g(q 1 (q), q) tanh g(q 1 (q), q) E 2 cosh θ g(q 1 (q), q) 2 , (A.10) which for θ = 1, using (A.3) and similar manipulations to those used in (2.19), evaluates to − E ln cosh β √ qz + E 1 E 2 cosh g(q 1 (q), q) log cosh g(q 1 (q), q) E 2 cosh g(q 1 (q), q) . (A.11) We then study the sign of (A.11), where q and q1 (q) are the solutions of the self-consistency equations (A.3) and (A.9), respectively.To this purpose, it is useful to study the behaviour of the function K(q, x) for x ∈ [0, q].For x = q, we have K(q, q) = 0, while the extremum of K(q, x) is found from = 0, (A.12) as x = E 1 E 2 sinh g(x, q) tanh g(x, q) E 2 cosh g(x, q) ≡ q1 (q) (A.13) via Eq.(A.9).Given that K(q, x) vanishes for x = q, if the extremum x = q1 (q) is global in the domain considered, we must have that K(q 1 (q), q) > 0 if x = q1 (q) is a maximum and K(q 1 (q), q) < 0 if x = q1 (q) is a minimum.Therefore, if is positive, K(q, q1 (q)) is negative, so the RS theory becomes unstable for We note that for q1 (q) → q, such condition retrieves the well-known AT line [25] 1 By numerically solving the self-consistency equations (A.3) and (A.9), we can verify that q1 (q) = q for all temperatures, hence this limit can be justified a posteriori for the SK model (see Fig. 4, left panel).We anticipate that this remains true for the Ising P -spin model analysed in the next section.4: RS overlap q (top) and difference between q1 (q) and q (bottom) versus the ratio T /J, for the SK model (left) and the Ising P -spin model, with P = 4 (mid) and P = 6 (right panel).The dotted vertical line marks the onset of the RS instability.

A.2 The Ising P-spin model
In this section, we consider a system of N Ising spins σ i = ±1, i = 1, . . ., N governed by the Hamiltonian ..,N i1,...,i P =1,...,1 J i1,...,i P σ i1 . . .σ i P (A.17) where J i1,...,i P are Gaussian i.i.d.variables, J i1,...,i P ∼ N (0, J 2 ).As in the previous cases, the order parameter of the model is the two-replica overlap q defined in (2.4).The quenched free-energy in RS assumption, at inverse temperature β reads as [27] −β where β ′ = 2β/P !, E is the average w.r.t. the Gaussian variable z and q fulfills the following selfconsistency equation: The 1RSB expression of the quenched free-energy in the thermodynamic limit is [27] −β where E 1 , E 2 are the average w.r.t. the standard Gaussian variables z (1) and z (2) and q0 , q1 fulfill the following self-consistency equations From now on, we imply the dependence of g and of the RS and 1RSB quenched free energies f RS , f 1RSB on β ′ and J.The aim is to prove that the 1RSB approximation of the quenched free-energy is smaller than the replica symmetric one, above a critical value of the parameter β ′ J.

A.3 P-spin spherical model
In this section we consider the spherical P -spin model, introduced for the first time in [24].The Hamiltonian of the model is the same as in (A.17), however the N spins σ i are now real variables, satisfying the so-called 'spherical' constraint As in previous cases, the order parameter is the two-replica overlap q introduced in (2.4).In the thermodynamic limit, at inverse temperature β, the quenched free-energy under the RS assumption (2.6) is given by where q fulfills the self-consistency equation Denoting the LHS with g(q 1 ) and noting that g(0) = g(1) = 0, and reasoning as for T ⋆ , a non-zero solution exists when the RHS is smaller than g(q 1 ) evaluated at its maximum point q⋆ 1 = (P −2)/(P −1), giving T < P (P − 2) P −2 /2(P − 1) P −1 [16].Next, we compute the derivative w.r.t.θ of −2βf 1RSB at q0 = q and q1 = q1 : Substituting q = 0 and the value of q1 = q⋆ 1 this evaluates to which is always negative for P > 2, implying This shows that at the temperature T c = P (P − 2) P −2 /2(P − 1) P −1 , where a non-zero overlap q1 first emerges, the RS theory becomes unstable, as known in the literature [16,24].It can be easily verified that T c > T ⋆ for all P > 2.
B Expanding around θ = 0 Although we have so far regarded the limit θ → 1 (where q0 = q and f 1RSB = f RS ) as the physical one, in the opposite limit, θ → 0, we would equally find, for all the models considered above, f 1RSB = f RS (with q1 = q), suggesting that a similar analysis could have been carried for θ → 0.
In this section we present such analysis for the Hopfield model, the Hebbian networks with multi-node interactions and the spherical P -spin.The same analysis can be carried out for the other spin-glass models considered in Appendix A. Given the strong similarity of the SK and the Ising P -spin models with the Hopfield and the dense associative memory models, respectively, we will not report such analysis here.

B.1 The Hopfield model
From (2.17), one finds where we have used that for θ = 0, ∆ 2 (0, q0 , q1 ) = ∆ 1 (q 1 ) and the relation with F any smooth function, a 1 , a 2 , a 3 ∈ R, and λ, Y and Z i.i.d.standard normal random variables.As (B.1) is identical to (2.9), in the limit θ → 0, q1 is equal to the RS order parameter q.Similarly, one can show that lim and can easily verify that f 1RSB ( m1 , q1 , q0 |θ)| θ=0 = f RS ( m, q).Our purpose is then to prove that for small but finite values of θ the 1RSB expression of the quenched free-energy is smaller than the RS expression, i.e. f 1RSB ( m1 , q1 , q0 |θ) < f RS ( m, q), below a critical line in the parameters space (α, β).

(B.18)
Again, we have that K( m, q, q) = 0, regardless of the value assigned to m (this follows from the fact that for θ = 0, q is an extremum of the free-energy).Next, we inspect the sign of K( m, q, q0 ( m, q)).
To this purpose, we study K( m, q, x) for x ∈ [0, q] and locate its extrema, which are found from where the last equality follows from (B.17).Under the assumption that the extremum x = q0 ( m, q) is global in the domain considered and reasoning as in the Hopfield case, we have that K( m, q, q0 ( m, q)) > 0 if x = q0 ( m, q) is a maximum and K( m, q, q0 ( m, q)) < 0 if it is a minimum.In particular, if is negative, K( m, q, q0 ( m, q)) > 0 and f 1RSB < f RS .This happens when the expression in the curly brackets of the equation above is negative, i.e. when the parameter αβ ′ 2 satisfies the inequality The resulting critical line is found to be identical to the critical line (3.11)obtained from the expansion around θ = 1 (see Fig. 5, mid and right panels).

B.3 Spherical P -spin
Expanding for small θ the self-consistency equations (A.37), (A.38) to linear orders, we get where the expression for A(q 0 , q1 ) and B(q 0 , q1 ) are provided in (C.19) and (C.20), respectively.If θ = 0, summing the two equations gives showing that, to orders O(θ 0 ), q1 = q, while q0 fulfills the following self-consistency equation whose solution is denoted with q0 (q).The latter equation is solved by q0 (q) = 0 (which corresponds to the paramagnetic solution and remains valid when ergodicity is broken, as explained earlier).Similarly, q1 = 0 is always a solution of (B.25), however the 1RSB scenario requires q1 > 0. As explained in the previous section, such non-zero solution appears at T ≤ T ⋆ = 2(P − 2) P −2 /P P −1 , which is below T c for any P > 2, hence we can immediately conclude that the instability of the RS theory occurs at the larger temperature T c , without further comparing the free-energies 1RSB and RS for θ close to zero.

C Contributions to sub-leading orders
In this appendix we provide expressions for all the functions that we left unspecified in the main text, as they did not contribute to leading orders, including the functions A(q 0 , q1 ) and B(q 0 , q1 ) for all the models considered.

Figure 2 :
Figure 2: RS instability line (i.e.T c versus α) obtained via our method (blue crosses) and the AT line obtained in the limit q1 ( m, q) → q (red dashed curve), for the Hopfield model model.The several branches separate the paramagnetic (P), spin-glass (SG) and retrieval (M) region.The black curve show the critical temperature T ⋆ at which q becomes non-zero within the RS theory, i.e. the spin-glass (SG) transition.

Figure 3 :
Figure 3: RS instability line (i.e.T c versus α) obtained via our method, i.e. by expanding the free-energy around θ = 1 (blue dots) and the AT line obtained in the limit q1 ( m, q) → q (red curve), for the Hebbian network with P -node interactions, with P = 2, 4, 6 from left to right.The black curves show the critical temperature T ⋆ at which q becomes non-zero within the RS theory, i.e. the spin-glass (SG) transition.

Figure
Figure4: RS overlap q (top) and difference between q1 (q) and q (bottom) versus the ratio T /J, for the SK model (left) and the Ising P -spin model, with P = 4 (mid) and P = 6 (right panel).The dotted vertical line marks the onset of the RS instability.

Figure 5 :
Figure 5: RS instability lines for the Hopfield model (left) and Hebbian networks with P -node interactions, with P = 4 (mid) and P = 6 (right), obtained via our method for θ → 0 and θ → 1.In the all the case the lines obtained for θ → 0 and θ → 1 are indistinguishable.