Why trace and delay conditioning are sometimes (but not always) hippocampal dependent: A computational model

Abstract

A recurrent-network model provides a unified account of the hippocampal region in mediating the representation of temporal information in classical eyeblink conditioning. Much empirical research is consistent with a general conclusion that delay conditioning (in which the conditioned stimulus CS and unconditioned stimulus US overlap and co-terminate) is independent of the hippocampal system, while trace conditioning (in which the CS terminates before US onset) depends on the hippocampus. However, recent studies show that, under some circumstances, delay conditioning can be hippocampal-dependent and trace conditioning can be spared following hippocampal lesion. Here, we present an extension of our prior trial-level models of hippocampal function and stimulus representation that can explain these findings within a unified framework. Specifically, the current model includes adaptive recurrent collateral connections that aid in the representation of intra-trial temporal information. With this model, as in our prior models, we argue that the hippocampus is not specialized for conditioned response timing, but rather is a general-purpose system that learns to predict the next state of all stimuli given the current state of variables encoded by activity in recurrent collaterals. As such, the model correctly predicts that hippocampal involvement in classical conditioning should be critical not only when there is an intervening trace interval, but also when there is a long delay between CS onset and US onset. Our model simulates empirical data from many variants of classical conditioning, including delay and trace paradigms in which the length of the CS, the inter-stimulus interval, or the trace interval is varied. Finally, we discuss model limitations, future directions, and several novel empirical predictions of this temporal processing model of hippocampal function and learning.

Introduction

Classical conditioning, in which a cue (the conditioned stimulus or CS) is paired with a reflex-evoking unconditioned stimulus (US) until the CS comes to produce an anticipatory response (the conditioned response or CR) has proven a useful testbed for examining the psychological principles and neurobiological substrates of learning. Under many circumstances, delay conditioning, in which the CS and US overlap and co-terminate, is spared or even mildly facilitated following hippocampal damage (e.g., Berger et al., 1983, Gabrieli et al., 1995, Ito et al., 2005, Ito et al., 2006, Schmaltz and Theios, 1972); conversely, under many conditions, hippocampal lesion disrupts trace conditioning, in which CS offset occurs before US onset, producing a temporal gap known as the trace interval (e.g., Beylin et al., 1999, McGlinchey-Berroth et al., 1997, Solomon et al., 1986, Weisz et al., 1980). This has led some researchers to assume that the hippocampus plays a more important role in trace than in delay conditioning (Beylin et al., 1999, McGlinchey-Berroth et al., 1997, Solomon et al., 1986)—as we discuss below, this assumption ignores empirical findings on the role of the hippocampus in delay conditioning. Along the same lines, some studies of humans and non-human animals use trace conditioning as a canonical task to demonstrate evidence of hippocampal dysfunction in transgenic models, healthy aging, and pharmacological models (Brown et al., 2010, Disterhoft et al., 1999).

However, there are strong reasons to challenge this delay/trace dichotomy. First, it has long been known that the hippocampus shows learning-related changes during acquisition of the delay CR in intact animals and humans. These changes include the development of responses by some hippocampal pyramidal neurons that precede the behavioral eyeblink CR and mirror its form (Berger et al., 1976, Berger et al., 1983, Berger and Thompson, 1978, Green and Arenos, 2007, Thompson et al., 1980). Initially, these responses occur in the US period, but increases in the CS period occur at about the time that behavioral CRs appear, and decline with continued training (for review, see Christian and Thompson, 2003). Similar hippocampal activity occurs in rabbits given trace conditioning (Weiss et al., 1996) but not in rabbits given unpaired CS/US trials (Solomon et al., 1986).

Functional imaging studies in humans show a similar pattern of learning-related activity in the hippocampus during delay eyeblink conditioning to those observed in animals (Blaxton et al., 1996, Cheng et al., 2008, Knight et al., 2004 Stein and Helmstetter, 2004; Logan and Grafton, 1995, Schreurs and Alkon, 2001). Thus, these data suggest that – even if the hippocampus is not necessary for acquisition of a delay CR – it nevertheless normally plays a role. This challenges the simple view of delay conditioning as hippocampal-independent, and begs a more nuanced view of the difference between brain substrates that are sufficient to mediate a learned response, versus those that are normally involved.

Second, under many conditions, delay conditioning is spared or slightly enhanced following hippocampal lesion. Specifically, the ability of hippocampal lesioned animals to acquire a delay eyeblink CR depends on the length of the CS interval. Specifically, while hippocampal-lesioned rats can acquire an eyeblink CR when the delay between CS onset and US onset is short, they are impaired when the delay is lengthened (Beylin et al., 2001). Thus, short-delay conditioning is spared by hippocampal lesion, but long-delay conditioning is not. Further, disruption of the hippocampus, via electrical stimuli or pharmacological intervention, can retard acquisition of even a short-delay CR (Kaneko and Thompson, 1997, Sakamoto et al., 2005, Salafia et al., 1979, Salafia et al., 1977, Solomon and Gottfried, 1981, Solomon et al., 1983). Together, these results document that delay conditioning is not always spared following hippocampal lesion or disruption.

Third, although trace conditioning is often disrupted by hippocampal lesion, this is not always the case. For example, Thompson et al. (1980) speculated that the hippocampus might be involved in trace conditioning, to bridge the temporal gap between CS and US. Solomon et al. (1986) presented an early study showing that dorsal hippocampal lesions disrupted trace eyeblink conditioning in rabbits, by decreasing the number of CRs. However, other studies followed that reported no trace conditioning impairment in hippocampal-lesioned animals (James et al., 1987, Port et al., 1986).

Another factor affecting the hippocampal-dependence of trace conditioning may be differences in the trace interval used. In studies where the trace interval has been explicitly varied, a deficit in trace conditioning appears only for long trace intervals. Thus, for example, hippocampal-lesioned rabbits are impaired on eyeblink CR acquisition with a long (500 ms) but not a short (100 ms CS or 300 ms) trace interval (Moyer et al., 1990). There may also be interactions between CS duration, and trace interval: Steinmetz and colleagues (Walker and Steinmetz, 2008) found that hippocampal-lesioned rats were impaired relative to controls on acquisition of an eyeblink CR when the CS duration was 50 ms and the trace interval was 500 ms, but not when the CS duration was 500 ms and the trace interval was 50 ms. In addition, Shors and colleagues (Beylin et al., 2001) showed that – although hippocampal-lesioned rats were impaired at both trace and long-delay eyeblink conditioning – once the lesioned animals had acquired a long-delay CR, they could then learn and perform the trace CR. Together, these results document that, at least under some circumstances, subjects with hippocampal lesion can acquire a trace CR as well as matched controls.

In summary, while the idea that trace conditioning is hippocampal-dependent whereas delay conditioning is hippocampal-independent provides a useful rule of thumb, it is not sufficient to adequately address the full range of existing data. An additional complication involves the inter-stimulus interval (ISI). When the ISI is short, response systems (such as the brainstem and cerebellum for eyeblink conditioning) can successfully learn CS–US associations and produce a well-timed CR. When the ISI is longer, the hippocampus helps to bridge the temporal gap between CS and US, facilitating production of a well-timed CR. Thus, in the case of eyeblink conditioning, both short-delay and short-trace paradigms can be acquired without impairment by hippocampal-lesioned animals; however, both long-delay and long-trace paradigms are disrupted following hippocampal lesion.

Consistent with this view, although infant rats (with immature hippocampus) can acquire short-delay conditioning, they are impaired at both long-delay and trace conditioning, which emerge in parallel during later development (Barnet and Hunt, 2005, Ivkovich et al., 2000). Thus, these data all suggest that in addition to the presence of a trace interval, the duration between CS onset and US arrival determines whether hippocampal mediation is required, as is the case in short- and long-delay conditioning.

Interestingly, when Hoehler and Thompson (1980) first speculated that trace conditioning might be hippocampal dependent, they did so based on their studies of ISI manipulations in eyeblink conditioning, and on their findings that the hippocampus appeared to be involved in forming a temporal map of the learned behavioral response to be made, allowing for the CR to be accurately timed even when the ISI is beyond the timing parameters that are “optimal” for the basic associative substrate. Thus, for example, the “optimal” parameters for eyeblink conditioning (operationalized in terms of acquisition speed) may be a few hundred milliseconds and may reflect temporal processing mechanisms in other structures such as the cerebellum.

As discussed by Christian and Thompson (2003), optimal temporal parameters for learning in the cerebellum are 50–200 ms between CS and US presentation. Most trace conditioning studies on animals use a trace interval of 500 ms to induce hippocampal involvement in task learning. Optimal parameters for fear conditioning tend to be an order of magnitude longer (usually>1 s, see for example, Bevins and Ayres, 1995), and may reflect temporal processing mechanisms in the amygdala. But in fear conditioning, just as in eyeblink conditioning, hippocampal lesions affect trace conditioning as a function of the trace interval, so that hippocampal lesions impair expression of contextual fear conditioning with long but not short trace intervals (Chowdhury et al., 2005, Pang et al., 2010). In other words, we argue that the hippocampus plays a similar role in both eyeblink and fear conditioning, but temporal differences in optimal ISI length for eyeblink and fear conditioning acquisitions are, respectively related to processing in the cerebellum and amygdala.

Extending this principle beyond classical conditioning, other brain systems mediate other behavioral responses, and each may have an operating window of temporal delays that can be spanned; in each case, these brain systems alone may be capable of mediating learning with sufficiently short delays, but as the delay is lengthened, hippocampal mediation becomes critical. This basic idea is consistent with a large number of theories of hippocampal-region function (e.g., Hoehler and Thompson, 1980, Rawlins, 1985, Wallenstein et al., 1998) and finds broad support from a range of preparations. For example, in delayed non-matching-to-place in an eight-arm radial maze, rats with hippocampal lesion can learn under short delays, but are impaired when the delay period is extended (Lee and Kesner, 2003). Similarly, in delayed non-match to sample (DNMS), primates with lesions limited to the hippocampus (sparing nearby medial temporal areas) can learn the non-matching task as well as controls, but show increasing impairments as the delay between sample and response is lengthened (Zola-Morgan and Squire, 1986). In each case, the important factor determining hippocampal dependence is not presence or absence of a stimulus-free gap, but rather the length of the interval across which information must be maintained before responding.

Again, similar findings have also been reported in human studies. For example, humans with bilateral hippocampal damage are impaired relative to healthy controls on temporal and spatial estimation at long, but not short, delays (Kesner and Hopkins, 2001). Patients with medial temporal lobe and hippocampal lesions also perform much better on relational memory tasks when the interval between learning and memory test is short than long. Specifically, Squire and colleagues (Jeneson et al., 2010) found that patients with medial temporal lobe damage do not show impairment in performing relational learning tasks when the delay between learning and test is short (1 s). Similarly, Ryan and Cohen (2004) have tested amnesic patients on a relational memory task using both short and long delay periods. They have found that patients are impaired only for the long-interval condition.

Thus, a conceptualization of eyeblink conditioning in which the length of the ISI, in addition to the presence of a trace interval, determines hippocampal dependence would appear to help integrate our understanding of the eyeblink conditioning literature with that of other preparations.

In this paper, we present an extension of our prior trial-level computational models of hippocampal function and stimulus representation. Our models assumed that the hippocampal region interacted with other brain systems, such as cortex and cerebellum, during associative learning, specifically by forming new stimulus representations that provided information about stimulus–stimulus and contextual regularities (Gluck and Myers, 1993, Gluck and Myers, 2001, Myers and Gluck, 1994, Moustafa et al., 2009). These models were correctly able to account for the effects of hippocampal lesion and disruption on various trial-level phenomena such as acquisition, discrimination, latent inhibition, and contextual shift effects. A later extension which modeled the effects of cholinergic manipulations by altering the hippocampal region learning rate was correctly able to address the effects of cholinergic agonists and antagonists on classical conditioning (Myers et al., 1996, 1998; Moustafa et al., 2010). However, these earlier models simulated trial-level information only, meaning that they could simulate whether a CR is given on a particular trial, but could not address within-trial events, such as the relative timing of CS and US onset. As such, these earlier trial-level models could not address the differences between delay and trace conditioning, nor the effects of manipulating the length of the ISI or trace interval. The need to address these aspects of the empirical data partially motivates the current work.

Our new model simulates performance in various delay and trace eyeblink conditioning data within a unified framework. Specifically, the current model includes adaptive recurrent collateral connections that aid in the representation of intra-trial temporal information. With this model, as in our prior models, we argue that the hippocampus is a general-purpose system that learns to predict the next state of all stimuli given the current state of variables encoded by activity in recurrent collaterals. As such, the model correctly predicts that hippocampal involvement in associative learning, including classical conditioning, should be most critical not only when there is an intervening trace interval, but also when there is a long delay between CS onset and US onset, as in short-delay vs. long-delay conditioning.

Fig. 1 shows a schematic diagram of the current model, which builds off our prior models of hippocampal-region processes in classical conditioning. Like our earlier models (Gluck and Myers, 1993, Gluck and Myers, 2001, Moustafa et al., 2009), the present model conceives of the hippocampal region (Fig. 1, green) as a predictive autoencoder, which learns to predict the next state of the world given current inputs. In the process, the hippocampal-region network forms new stimulus representations in its internal layer that compress (or make more similar) the representations of co-occurring inputs while differentiating (making less similar) the representations of inputs that make different predictions about future events such as US arrival. In other words, the essential function of the hippocampus in our model is monitoring environmental regularities and using prior experience and current inputs to predict what (out of all possible events) is likely to happen next. Classical conditioning is a good example of prediction processes because the most salient event – the US arrival – can be predicted with high accuracy by learning the CS and the ISI. In our model, the hippocampal network learns not only whether a particular CS will be followed by a US, but when this will occur.

Also as in our prior models, the hippocampal-region network communicates with a second motor network (Fig. 1, blue), which is assumed to represent some of aspects cortical and cerebellar substrates of motor learning. The motor output network is modeled as a single adaptive node that learns to map from weighted inputs specifying the presence of CSs, as well as contextual or background stimuli and an efferent copy of the CR. The activities of the hippocampal-region network hidden layer units are also provided as inputs to the motor output network, allowing the motor output network to incorporate the adaptive representations formed in the hippocampal-region network into its own ongoing learning. The output from the motor response network represents the behavioral CR. The difference between this output (CR) and the US constitutes an error signal that can be used to train the connection weights in the motor response network, using an error correction rule such as the least-mean squares or LMS rule (Widrow and Hoff, 1960); full details of the learning rule and other model details are provided in Section 4.

The major differences between this model and the prior models are (1) the consideration of each trial not as a discrete event, but as a series of timepoints, (2) the addition of recurrent pathways within the hippocampal-region and motor output networks. We discuss each of these points below; full simulation details are provided in Section 4.

First, to simulate within-trial events, each trial is divided into a number of timesteps, which represent small time intervals within a conditioning trial (e.g., 50 ms). Typically, contextual inputs are present during the entire trial, and are present alone during the first several timesteps of a trial; then one or more CSs may be presented for a specified number of timesteps; the US, when present, appears for a single timestep. The US may overlap with the CS (as in delay conditioning) or may occur after CS cessation (as in trace conditioning). A further series of context-alone presentations ends the trial (simulating the intertrial interval or ITI). Fig. 2 provides schematic illustrations of some example paradigms that we simulate in our model.

A second difference between the prior and current models is that Fig. 1 includes recurrent connections within the hippocampal and motor output models. Specifically, the hippocampal region network includes recurrent connections within the internal layer, while the motor output network contains a feedback CR pathway, carrying information regarding the current state of the CR. Provision of feedback within the hippocampal network allowed the activation of internal layer nodes at any time step to be a function of external (CS and contextual) input and also of the adaptive representation of input from a previous timestep. Because the weights on these recurrent connections are adaptive, it is possible for a sequence of activation patterns to be stored in the network that “buffers” input information over several timesteps. Importantly, this buffering function is not pre-wired into the network, but emerges dynamically as a result of training and learning. Anatomical studies support the existence of such recurrent loops in the hippocampal region, particularly hippocampal subfield CA3 (Amaral et al., 1990, Amaral and Witter, 1989) as well as the dentate gyrus (Amaral et al., 2007). Prior models of the hippocampus have also included recurrent connections to simulate conditioning tasks (see for example, Rodriguez and Levy, 2001).

In this model as in our prior models, hippocampal lesion is simulated by disabling learning in the hippocampal-region network, in which case the motor output network can still learn new responses by modifying weights from the CS and contextual inputs, but no new adaptive stimulus representations are formed in the hippocampal region (Gluck and Myers, 1993, Myers and Gluck, 1994). In addition, and also as in our prior models, the effects of cholinergic agonists and antagonists are simulated by raising (agonist) or lowering (antagonist) learning rates in the hippocampal-region network (Myers et al., 1996, 1998; Moustafa et al., 2010).