The generative neural microdynamics of cognitive processing

The entorhinal cortex and hippocampus form a recurrent network that informs many cognitive processes, including memory, planning, navigation, and imagination. Neural recordings from these regions reveal spatially organized population codes corresponding to external environments and abstract spaces. Aligning the former cognitive functionalities with the latter neural phenomena is a central challenge in understanding the entorhinal-hippocampal circuit (EHC). Disparate experiments demonstrate a surprising level of complexity and apparent disorder in the intricate spatiotemporal dynamics of sequential non-local hippocampal reactivations, which occur particularly, though not exclusively, during immobile pauses and rest. We review these phenomena with a particular focus on their apparent lack of physical simulative realism. These observations are then integrated within a theoretical framework and proposed neural circuit mechanisms that normatively characterize this neural complexity by conceiving different regimes of hippocampal microdynamics as neuromarkers of diverse cognitive computations


Microdynamical organization of hippocampal reactivations: an algorithmic marker of cognitive computation
An essential feature of many cognitive algorithms is the presumed multiphase nature of the associated processing.Working memory, a paradigmatic cognitive mechanism for the study of neural computation, is typically modeled as being composed of storage and retrieval phases [1].Cognitive control is not a singular computation but is composed of goal identification, action selection or suppression, and error monitoring [2].Planning models may be decomposed into phases of sampling and updating, each phase being a technical area of research in its own right [3].The complementary learning systems perspective on long-term memory storage emphasizes a short-term episodic tracing phase followed by a consolidation phase in which information is transferred to the cortex [4].Latent representations in neural networks are trained through alternating wake and sleep phases [5].More broadly, dual process theories of the brain provide a useful framework for embedding multiphase componential mechanisms across a range of areas in cognition and behavior [6].Indeed, the importance of such dynamic multiphase contributions has led to modular cognitive and neural network architectures that implement these functionalities [7].We refer to this multiphase characteristic of cognitive computational models as the macrodynamics of cognition.
Sketching an organization of cognitive processes as a hierarchical program of neural dynamical systems raises the question of what constitutes the microdynamics of cognitive processing?The properties of a cognitive microdynamic should include being a process with a singular purpose within the context of a broader macrocognitive computation that evolves and terminates on short timescales.Attractor dynamics, whereby a neural network state evolves to a stable fixed point, reflect such properties in the context of memory retrieval [8] and contextual inference [9].Indeed, attractor microdynamics are a crucial and ubiquitous motif linking the computational and mechanistic levels of brain function [10].Our view is that new varieties of microdynamics, which are more generative in nature and consequently difficult to reconcile in a purely attractor-based theory, have been recently detected within the representational content of theta cycle activity and hippocampal sharpwave ripples (SPW-Rs) [11,12].These neural phenomena contain subsecond (4e10 Hz) and ultrafast (up to 250 Hz) sequential reactivations traversing hippocampal spatial representations [13e15].
In the last decade, innovative analysis techniques have revealed that these neural phenomena can have highly variable statistical and structural profiles as a function of the behavioral and neurophysiological state of the animal [16e21] (Figure 1).Given the subtle differences in activation patterns and associated analysis strategies, these hippocampal sequences have been variously referred to as replay, generative replay, sequential hippocampal reactivations, or trajectory events and have typically been studied independently [22].
Here we synthesize a collective computational perspective for these phenomena through the lens of optimized generative sampling of trajectories [23,24].Descriptive templates for the relevant nonlocal hippocampal reactivation phenomena are provided with apparently disordered, physically unrealizable dynamics highlighted (Figure 1).We then discuss how different generative regimes are optimal for distinct cognitive functions such as memory retrieval, planning, and learning.Finally, we outline a novel recurrent network model of the entorhinal-hippocampal circuit (EHC), which exhibits attractor-based and generator-based modes of operation, and suggest possible circuit-level mechanisms that may be involved in flexibly modulating nonlocal hippocampal sequence generation.

Wandering and jumping through internal spaces: the diffusive and superdiffusive regimes
Internal spatial models, such as cognitive maps, have the capacity to encode sophisticated relational knowledge in domains such as navigation and semantic memory [27].However, they are computationally inert in the absence of some form of dynamic that executes a cognitive Organizing the diversity of sequential hippocampal reactivations.Generating sequences from internal models is an essential element of many cognitive algorithms.Given the systematic variability in the structure and statistics of generated sequences across behavioral states and environmental scenarios, we conceptualize this variability as reflecting the evolution of cognitive computations at a macrodynamical scale.In the illustrated macrodynamical trajectory (cyan), an animal initially requires a localized position representation before generating superdiffusive exploratory trajectories in order to forage, for example (black dot: hippocampal location encoding, black curve: large discontinuity in successive locations, i.e., jumps).During sleep, diffusive reactivations facilitate the consolidation and generalization of spatial knowledge structures from episodic traces.In turn, this refined internal model provides the representational scaffold for forward jump trajectory events in the awake, immobile state [16,18].More generally, cognitive macrodynamics may be sequentially structured as described, parallelized across distinct brain regions, or interleaved over time [20].The importance of sequence generation as a cognitive tool is reflected in the diversity of microdynamical structure, which we organize according to its apparent spatiotemporal order or disorder.Instances whereby nonlocal trajectories violate physical reality (i.e.there does not exist a behavior that would induce such a spatial trajectory under localized hippocampal activations) suggest that the role of sequence generation may not be simply explained by veridical simulation, behavioral representation, or memory retrieval hypotheses.calculation [28,29].Classically, an early influential idea regarding how generative dynamics aid in memory processing was spreading activation theory [30,31].This theory postulated that human knowledge was stored in long-term memory in a semantic graph format and that conceptual associations between semantic memories could be achieved by iteratively activating adjacent nodes on the graph.This algorithm effectively implemented parallel search until a connecting path between memories was identified.In mathematical terminology, this exploratory stochastic process is a (parallelized) random walk or, equivalently, a discretized diffusion (Figure 2a, Diffusion).Such diffusive wandering may be ubiquitously applied to internal models, whether spatial or graph-based, in order to generate possible exploratory trajectories [32,33].However, diffusions tend to be ineffective for exploration, particularly when the scale of the environment is unknown, and instead, superdiffusive sampling strategies perform better [23].That is, when exploring large spaces, either structured (e.g.mazes) or spatially homogeneous (e.g.open fields), efficient sequential sampling patterns are stereotyped by the interlacing of local "spreading" searches with occasional jumps for global repositioning.In a wide range of ethological studies, it has been observed that the behavioral patterns of many species [34], including rodents foraging an arena [18], and humans performing semantic memory searches [35,36], exhibit this highly distinctive pattern of interleaved local searches and global jumps characteristic of such anomalous processes.More recently, fine-grained analyses of hippocampal SPW-Rs have revealed analogous superdiffusive structure within the nonlocal sequential position coding in hippocampal SPW-Rs [21,23].Specifically, in these neural microdynamics, which may generate candidate paths for spatial exploration, hippocampal activity successively encodes nearby positions but additionally has a small probability of generating a large jump to a more distal region (Figure 2a, Superdiffusion) [16].Furthermore, such jump trajectories may be guided specifically to motivationally salient positions such as a home or reward location, regardless of the physical position of an animal, thus implementing an efficient spatial retrieval algorithm [23] (Figure 2a, Remote activation).Such remote activation microdynamics is another desirable feature, as it expedites rapid behavioral orientations towards locations that may be critical to survival.We suggest that recently observed goal-sensitive distortions in grid cell representations [37,38] embed goal locations directly within the grid code, which can then be parsimoniously addressed via superdiffusive hippocampal reactivations [23].
Given the empirical observation of superdiffusive trajectories in hippocampal reactivations, along with their apparent superiority for exploratory search processing compared to diffusions, it is striking to note that diffusive trajectories have also been recorded in SPW-Rs [18].Furthermore, diffusive hippocampal generation occurred while animals were resting, consistent with a oneiric contribution to a distinct cognitive function compared to superdiffusive trajectories that occur in the awake, immobile state (Figure 2a, Diffusion).Theoretically, it is suggested that diffusive trajectory sampling, as opposed to superdiffusive or episodic (i.e.replaying spatial behavioral traces through the environment), is optimized for learning and consolidating spatial knowledge [23].Indeed, it can be shown that diffusions uniquely reflect fundamental inductive properties regarding the structure of space, such as isotropy and finite scale, thus facilitating generalization and interpolation of spatial experience [18,23].

Entorhinal cortex as a cortical control interface for hippocampal microdynamics
Several computational studies have posited grid cells in the medial entorhinal cortex [39,40] as matrix factorizations of the correlational structure of hippocampal population place codes [41e43].Position encoding in the hippocampus can then be achieved via a linear readout from such grid cells, which reflect a lowdimensional spatial representation (Figure 2, Position encoding).This circuit motif underpins models of path integration in EHC [44].Path integration models predict stable position encoding localized to the animal and thus do not explain nonlocal hippocampal reactivations or their apparent regulation via entorhinal input [18, 45,46], nor do broader theoretic perspectives suggest that hippocampal activity contributes to a variety of cognitive functions beyond path integration [47].We suggest that incorporating a generator operational mode into network models of EHC, in addition to the established attractor and integrator modes, may provide an opportunity to develop a unifying normative perspective on EHC function [10,48].Indeed, circuit mechanisms that can support non-local reactivations have been proposed [49], including firing rate adaptation, which may be tuned to generate superdiffusive trajectories [25] (Figure 2c, d).Beyond superdiffusions, the generator network outlined here can support the controlled generation of nonlocal, heteroassociative, and compositional trajectories, thus accounting for a wide variety of hippocampal microdynamics [23].
This generative perspective presented here suggests that localized position encoding (Figure 2a, Localization) is one point on a spectrum of microdynamical regimes of hippocampal sequence generation controlled by distributed input from the medial entorhinal cortex (mEC) [23,24] (Figure 2b).This perspective is formalized computationally, with grid cells serving as factorized population codes of spatial generators, which are read out to flexibly shape propagator distributions over states in a hippocampal cognitive map.Intuitively, generators specify the short timescale dynamics of nonlocal reactivations.Factorized representations of generators across grid cells enable a linear readout from mEC to the future propagator distribution of nonlocal reactivations at any timescale [23].Furthermore, the temporal, statistical, and structural profiles of reactivation propagators may be determined by a proposed circuit mechanism of flexible modulation.This hypothetical mechanism modulates the relative activity gains of grid cells in a distributive manner at the level of grid modules across the dorsoventral axis of mEC (Figure 2b).This theory explains why the covariance in grid cell population activity may flexibly alter across grid modules between behavioral states (such as awake and sleep) but remain stable within modules and why grid cells may spatially rescale as a function of arousal [50].We suggest that this modulation mechanism could serve as a control interface by which higher-order brain regions such as the prefrontal cortex may leverage its dense projections to the mEC in order to shift the microdynamical regime of hippocampal sequence generation according to the desired cognitive computation.It is predicted that hippocampal microdynamics should depend on the distributed flexible modulation of grid modules, which in turn relates to the cognitive state of the animal (Figure 1).Potentially, comparisons between dorsoventral mEC activity profiles across distinct behavioral states of the animal (e.g.mobile or immobile), taken as a correlate of its cognitive state (e.g.exploring or decision-making), may provide an opportunity to test elements of this theory [51].

The hippocampal microdynamics for rapid inference
Another consideration for the design of generative sequence modeling in the brain pertains to its use in online planning algorithms, whereby the future rewards and costs associated with a particular course of action are estimated via internal simulation [3,28,52,53].The ability of a biological agent to plan effectively is dependent on its ability to make rapid and accurate inferences in noisy and complex environments [54].For example, in computational models of decision-making, an important quantity to be predicted is the total expected reward E p p ½rðxÞ where r(x) is the reward obtained in a particular environment state x2X since the objective of the agent is to maximize this quantity [55].The distribution p p (x) is the probability of encountering a state x and is dependent both on the environment dynamics and the decision policy p of the agent.Given a sample of N states È x i É i¼1;.;Ndrawn from p p (x), this quantity can be estimated as and then input into a choice comparison algorithm in order to produce a considered course of action.The expected error in this estimate scales with a measure t int of sampling inefficiency known as the integrated autocorrelation time, which is the sum of autocorrelations in the generative process over time [56].
Essentially, t int reflects a speed-accuracy trade-off in simulation-based inference as it specifies how much the estimation error is reduced by increasing the time taken to generate more samples (Eqn.( 2)).Consequently, generative processes with high autocorrelations, such as diffusions (Figure 2A, Diffusion), are highly inefficient since many samples must be drawn over a relatively long period of time in order to achieve a low estimation error.Thus, in an ecological context, animals depending on an autocorrelated internal sampling process to make rapid decisions may suffer greatly from decision errors rooted in erroneous inference or slow response times [57].Therefore, in order to minimize planning time, a normative strategy is to construct minimally autocorrelated state sequences for fast and flexible prospective evaluation [7].
Given the direct benefits of minimizing such autocorrelations for the purposes of statistical estimation, it has been hypothesized that hippocampal sequence generation for simulation-based inference may pursue a minimal autocorrelation objective [23].This mode of sequential sampling re-activates map positions in an order distinct from that in which they were, or could be, experienced, and thus results in physically unrealizable trajectories.That is, there is no behavior that the animal could produce in the real world that would induce such a trajectory via local hippocampal place activations.Since it cannot reflect a time-compressed encoding of behavior, such sequences may be deemed to purely contribute towards a cognitive objective [58].We refer to such generative sequence production as teleological simulation since the microdynamical structure of the simulation is physically disordered in order to optimize the auxiliary objective of efficiency in approximate inference [23].This simulation framework stands in contrast to chronological simulation, whereby a veridical rollout of the environment dynamics is generated [52], as in standard approaches to planning such as Q-DYNA [59], as well as physical reasoning models based on forward simulation [28,60].
This theory explains the phenomenon of "generative cycling" recorded in hippocampal theta sequences [19] (Figure 2A, generative cycling).In a T-maze experiment, it was observed that, as the rodent approaches the junction, theta sweeps out of the animal and then cycles back and forth between the two arms of the maze.Such sequential dynamics emerge in the generator network architecture (Figure 2B), with the modulatory input to the entorhinal layer optimized for minimizing autocorrelations in the sequences generated downstream in the hippocampal layer [23].Potentially, the application of this computational hypothesis in alternative environment structures may help to explain apparently disordered hippocampal reactivations that have hitherto lack a coherent explanation.
Another mechanism by which possibly physically unrealizable trajectories may be generated in cognitive maps is via composition [24,61].The generator-based network architecture enables generators to be composed simply by stacking them in a deeper network.The resulting compositional propagator reflects the dynamics encoded by both generators.A demonstration of this compositional mechanism may be constructed to generate extended replay sequences across multiple SPW-Rs, which have been established to be causally dependent on mEC input [46] (Figure 2, Extended replay).

Conclusions
Though ultrafast sequential reactivations in SPW-Rs have been classically observed to recapitulate an episodic trace of the previous experience of a rodent and thus referred to as "replay" [62,63], further experiments across distinct environments, tasks, neurophysiological, and behavioral states have revealed a great degree of variability in the statistical and structural profiles of SPW-Rs [16,18,21,46].Similarly, theta sequences, previously established to represent localized sweeps near the rodent during locomotion [64], have now been observed implementing remote activation "jumps" [65] and generative cycling [19].Initial spatial decoding analyses suggested that hippocampal reactivations reflected an essentially spatiotemporally contiguous sequence of positions in an environment.However, more recently, it has been observed that generative hippocampal sequences are generally discontiguous [17] and often systematically disordered, such that they could never correspond to the experience of the animal either in the past or the future.This physical unrealizability indicates that such apparently disordered sequences are optimized for cognitive computation [13] rather than behavioral representation or memory retrieval [66,67].Indeed, this qualitative characteristic may serve to adjudicate between competing hypotheses regarding the functional implications of nonlocal hippocampal reactivations.Such diversity of hippocampal microdynamics and their functional roles are integrated within a framework conceptualizing the EHC as implementing a flexible generative process for trajectories traversing internal relational models or cognitive maps [23,24,58,68e70].Starting from this perspective, we reviewed the computational hypothesis that the microdynamical structure of hippocampal trajectories is modulated by distributed entorhinal input according to the specific cognitive computation currently being executed in the brain.Fundamentally, this perspective proposes that the entorhinal-hippocampal network forms a generative neural system that flexibly modulates the microdynamical structure of sequential representation in cognitive maps rather than encodes such maps per se.
The algorithms used by animals and humans to explore environments efficiently [71,72], consolidate novel experiences into long-term knowledge representations [4], represent spatial uncertainty [73], and plan [3] all depend on the internal sequential activation of state and action representations.However, the statistical and structural regime of sequence generation is a critical factor in determining the performance of the downstream cognitive algorithm [56].For example, the structure learning of internal models and planning from these learned internal models are best facilitated by profoundly contrasting regimes of generative sampling, despite the dependency between these two cognitive computations.Specifically, the former is optimized by random walk dynamics, leading to highly autocorrelated state sequences, whereas the latter specifically requires minimally autocorrelated state-space trajectories (Figure 2).How might the brain switch between such different generative modes for sampling trajectories from the same underlying space?One possibility is that the brain retains multiple copies of the same spatial states but with different map structures (e.g.diffusive or minimally autocorrelated), which can be retrieved depending on the desired generative mode.For example, the natural associations between semantic representations might be augmented with judiciously chosen, but essentially illusory, associations in order to sculpt a small-world semantic graph for more efficient search even with diffusive propagation [74].An alternative hypothesis is a single neural population code, which represents an internal model in a format that can be used to flexibly generate distinct neural microdynamics.We suggest the latter is facilitated by the entorhinal grid code [23].Given this, complementary analyses of dynamics (i.e. the sampled non-local trajectory, a marker of cognitive function) and representation (i.e. the environment and task elements, i.e. the relevant positions, objects, and stimuli) in the interrogation of hippocampal population coding may be fruitful for teasing apart the macro-and micro-components of cognition since distinct patterns of hippocampal microdynamics may be generated from the same cognitive map being represented in the hippocampus [75].Indeed, relatively unsupervised approaches to analyzing hippocampal data may prove crucial in uncovering novel regimes of hippocampal microdynamics [21,76].
In particular, the generator model provides a mechanistic account regarding how the entorhinal cortex may provide modulatory input that adapts the generative dynamics of hippocampal sequences according to the contemporaneous cognitive computations being performed by the brain [23].This broadens the role of mEC input beyond the hippocampal inheritance of phase precession [77,78].More speculatively, we suggest that top-down input from prefrontal cortex (PFC) may induce the appropriate distributed activity profile in mEC via gain modulation and grid rescaling [79].
Experiments involving simultaneously recordings in the entorhinal cortex and hippocampus, or prefrontal and entorhinal cortices, in conjunction with comparisons across behavioral states would be helpful in testing these hypotheses for the PFC-EC-HC pathway [14,80,81].Many computational issues also remain unaddressed.Can the brain learn novel microdynamics via the adaptation of PFC input to EC [82]?Or is there a preconfigured basis set of possible sequence generation regimes [70]?Can schedules of distinct microdynamics be optimized and executed in serial, parallel, or interleaved [7]?
The theoretic analysis of the entorhinal-hippocampal circuit may benefit from further evaluation of its apparent generative functionality, an integrative analysis of generator-based and attractor-based modeling, and drawing ideas from recent innovations in generative models in machine learning [83,84].Regarding the latter, direct analogies may be established by observing that diffusion models have a qualitatively similar architecture to the compositional stacking of generators [24].
Procedures for training such models by backpropagation and anomalous extensions of diffusive processing may underpin a bidirectional flow of inspiration between the computational neuroscience and machine learning perspectives here.Furthermore, the proposed flexible modulation mechanism in the entorhinal cortex based on input from the prefrontal cortex is reminiscent of the feature-wise linear modulation strategy for conditioning neural networks based on higher-order, typically textbased, input [85].Models of the hippocampus as transformers or modern Hopfield networks [86] could be augmented with modulated entorhinal input for flexible generativity in addition to its role in path integration.This would provide a cohesive framework relating localizing attractor-based models with nonlocal reactivation phenomena [21,25,49,87] and graft generative mechanisms into more flexible associative memory models [88].* * .Stella F, Baracskay P, O'Neill J, Csicsvari J: Hippocampal reactivation of random trajectories resembling brownian diffusion.Neuron 2019, 102:1-12.This study uses a novel analysis drawn from stochastic process theory to show that the spatial reactivations within SPW-Rs have a diffusive reactivation structure inconsistent with memory retrieval of experience.

Figure 1 Current
Figure 1

Figure 2 Hippocampal
Figure 2 , et al.: Constant sub-second cycling between representations of possible futures in the hippocampus.Cell 2020, 180:1-16.It is demonstrated that, as a rodent approaches a junction, theta sequences alternate between encoding two discontiguous locations to the left and right of the junction.This physically unrealizable microdynamic is referred to as "generative cycling."20.Wang  M, Foster D, Pfeiffer BE: Alternating sequences of future and past behavior encoded within hippocampal theta oscillations.Science 2020.21.Krause EL, Drugowitsch J: A large majority of awake hippocampal sharp-wave ripples feature spatial trajectories with momentum.Neuron 2022, 110:722-733.e8.22. Genzel L, et al.: A consensus statement: defining terms for reactivation analysis.Philos Trans R Soc Lond B Biol Sci 2020, , Stachenfeld KL, Botvinick MM, Gershman SJ: Flexible modulation of sequence generation in the entorhinal-hippocampal system.Nat Neurosci 2021, 24: 851-862.A theoretical framework is proposed for hippocampal microdynamics based on fine-grained stochastic analyses of generative sampling processes and a hypothesized entorhinal-hippocampal circuit mechanism.24.McNamee D, Stachenfeld K, Botvinick M, Gershman S: Compositional sequence generation in the entorhinalhippocampal system.Entropy 2022, 24:1791.25.Dong X, Chu T, Huang T, Ji Z, Wu S: Noisy adaptation generates lévy flights in attractor neural networks.In Ranzato M, Beygelzimer A, Dauphin Y, Liang P, Vaughan JW.Advances in neural information processing systems, 34.Curran Associates, Inc.; 2021:16791-16804.Neural microdynamics for cognition McNamee 7