Detecting sub-second activation sequences with sequential fMRI pattern analysis

Mental computations are often reflected in fast changes of neural activation patterns, for instance during so called replay events in the hippocampus. A major challenge for human neuroscience is therefore to capture such fast changes with sufficient spatial resolution using noninvasive neuroimaging. Here, we demonstrate that functional magnetic resonance imaging (fMRI) with a conventional repetition time (TR) of 1.25 s can be used to investigate sequentially activated neural patterns separated by less than 100 ms. We investigated the statistical properties of neural activation patterns following the presentation of fast sequential visual stimuli by extracting multivariate probabilistic estimates for the presence of a neural event over time. The time-course of the probabilistic classifier evidence resembled the expected shape of the hemodynamic response function (HRF). Importantly, by disentangling temporally and spatially overlapping BOLD signals our analysis technique allowed us to detect the order of sequentially activated neural patterns separated by only 64 ms. Providing such enhanced temporal resolution, our method promises to lay the groundwork for investigations into cognitive processes that require extracting temporal information from fast neural event sequences, such as hippocampal replay.


Introduction
Many cognitive processes are underpinned by fast changes in neural representations that can occur on sub-second timescales. These include, for example, fast replay of memory representations (e.g., Zhang, Deuker, & Axmacher, 2017), temporal predictive coding (e.g., Ekman, Kok, & De Lange, 2017), and fast value comparisons (e.g., Rich & Wallis, 2016). Investigating these topics non-invasively in humans requires the detection of detailed temporal information with anatomical specificity, which remains a methodological challenge (Ghuman & Martin, 2019). Recently, we provided evidence for non-spatial replay in the human hippocampus during rest by using a novel statistical analysis that is sensitive to sequential neural events occurring as fast as one hundred milliseconds apart (Schuck & Niv, n.d.). Here, we evaluate experimentally how well the occurrence and specifics of fast neural event sequences can be detected in general using fMRI. Extending and validating our previous approach, we introduce a novel method that allows the detection of fast-paced neural events with fMRI in humans.

Tasks
The experiment involved an oddball and a sequence task. In the oddball task, participants were presented with a picture (500 ms duration) from one of five visual object categories (cat, chair, face, house, shoe; cf. Haxby et al., 2001). Participants were instructed to detect stimuli that were presented upside-down on 20% of trials (see Fig. 1a) and performed 600 oddball trials overall. The inter-stimulus interval (ISI) in the oddball task was exponentially distributed with a mean of 1.5 s (truncated at 1 s). During trials from the sequence task first a cue indicated a target object category, and then a sequence of the five known objects was displayed. Each image of the sequence was shown for 100 ms. Importantly, the ISI between images of a sequence varied between 32, 64, 128, 512 and 2048 ms. Participants' task was to detect the serial position of the cued target stimulus within the sequence, and correspondingly answer a two-choice question after a delay period without visual input (16s after sequence onset; see Fig. 1b). In total, the experiment involved 75 sequence trials (15 trials per speed condition).

fMRI pattern classification
For each trial, one brain activation volume recorded 4 s after stimulus onset was masked jointly by an anatomical inferior occipito-temporal mask and a functional task-selectivity mask (see below). Voxels present in both masks were used to train a multinomial logistic regression classifier (L2 regularization, C-parameter = 1.0) to decode object category. Classifier training on oddball trials was done using leave-one-run-out crossvalidation and feature selection was based on training data alone (M = 4592 voxels within the occipito-temporal mask with highest and lowest t-values in a stimulus-on contrast). The trained classifier was applied to held-out data from (1) the oddball task and (2) the sequence task.

Probabilistic pattern sequence analysis
The main question we asked was to what extend we can infer the order of images displayed in sequence trials from fMRI data. To this end, we applied the trained classifier to each volume following the onset of the visual object sequences and  Figure 1: Task design. (a) Oddball task: Participants were instructed to press a button in response to visual stimuli occasionally presented upside-down (20% of trials) but not respond to upright pictures. (b) Sequence task: Participants were instructed to detect the serial position of a cued target stimulus in a sequence of five visual images and select the correct answer from two options after a delay period without visual input. Colors are only shown for illustration purposes.
analyzed the time courses of the predicted probabilities for different object categories. Our analyses were twofold: First, we investigated the sequential ordering of relative activation differences within TRs across time. If fast neural sequences lead to largely overlapping blood-oxygen-level dependent (BOLD) responses, one might still expect that the first pattern is activated most, the second element activated second most etc. during the signal rise, and vice versa during the signal fall (for an illustration, see Fig. 2). To test this idea, we cumulated the probabilities for each serial event across time per participant and trial. In this analysis, faster accumulation indicates an earlier onset and sustained higher level of decoding probability (for a similar approach, see Michelmann, Staresina, Bowman, & Hanslmayr, 2018). Then, pattern sequentiality was indexed by the slope of a line fitted to the cumulative probabilities of each serial event at every time point (see Michelmann et al., 2018, their Fig. 3C). A negative slope at any given time point indicated forward decoding, reflecting that earlier events had a higher cumulative probability as compared to later serial events. Correspondingly, a positive slope implied backward decoding. The slopes were tested against 0 (the expectation of no order information) at every time point across participants with a series of two-sided t-tests, adjusted for multiple comparisons by controlling the false discovery rate (FDR).
Second, we analyzed the fMRI data from the sequence trials for evidence of sequentiality across TRs. To this end, we extracted the most likely pattern for each TR, and calculated the step-sizes between consecutively decoded serial events, as in (Schuck & Niv, n.d.). For example, decoding Event 2 ! Event 4 consecutively would correspond to a step-size of +2, while a 3 ! 2 transition would reflect a step size of -1, etc. Here, we assessed the frequency of unsigned step-sizes separately for the five different speed conditions and compared them either against the average frequency in a permutation again using two-tailed FDR-adjusted t-tests.

fMRI results
Decoding in slow oddball trials. Classification accuracy for the five different visual objects was higher than the chance baseline level (t (39) = 23.33, p < 0.001; see Fig. 4a). Applying the probabilistic classifier to a series of volumes from stimulus onset in oddball trials revealed a slow and delayed increase in classification probability for the true stimulus class, as expected based on hemodynamics (see Fig. 4b). Decoding sequential pattern activation: Within-TR ordering. Time-courses of decoding probabilities following sequence trials revealed that pattern sequentiality can be detected in the classifier probability time-courses (see Fig. 5). Sequences with 2048 ms between serial events resembled overlapping single trial decoding time-courses as shown in Fig. 4b. At faster speeds, decoding probabilities decline, but order-dependent probability ordering can still be seen at speeds of 64 ms (mostly backward, reflecting the falling slope). As described above, we quantified this impression by cumulating the probabilities separately for each serial event and normalizing the resulting time courses to a maximum of 100% (see Fig. 6). Examining the slopes of the linear fits to the cumulated probabilities of serial events at each time point not only revealed significant forward sequentiality at slower speeds (TRs 3-11 and 3-5 at 2048 and 512 ms ISI, respectively; all p's < .001) but also faster speeds (TR 3 for 128 and 64 ms ISI, p's = .012 and p = .029). Furthermore, at 32 ms we found significant backward sequentiality at later timepoints (TRs 8, 11, and 12, p's = .045, .027, .012, respectively; FDR-corrected). There was no significant evidence for backward sequentiality for other speed conditions. Decoding sequential pattern activation: Between-TR ordering Next, we investigated evidence of sequentiality across TRs. First, we analyzed whether the average serial position of decoded events changed over TRs. If our fMRI analyses are sensitive of pattern sequences, then one might expect earlier events to be decoded primarily in earlier TRs, whereas later events are primarily decoded in later TRs. In-

Discussion
Our data establishes the feasibility of investigating fast sequential neural event sequences with fMRI despite its sub-  stantial temporal limitations. Our findings also verify and extend our recent fMRI study in which evidence for sequential reactivation of previous task experience was found in sequential statistics of pattern activation in the human hippocampus during rest (Schuck & Niv, n.d.). Our methodology allows to study sub-second neural event sequences with millimeterrange spatial resolution using non-invasive neuroimgaing and has the potential to impact wider areas of human cognitive and brain research.

Acknowledgments
This work was funded by a research group grant awarded to NWS by the Max Planck Society (M.TN.A.BILD0004). We thank Gregor Caregnato for help with participant recruitment, Unsigned step-size between decoded events Average frequency relative to permutation Figure 9: Average frequency of unsigned step-sizes between consecutively decoded serial events relative to a permutation. Average frequency (y-axis) of unsigned stepsizes (x-axis) between consecutively decoded serial events (across TRs) relative to the average step-size of all sequential permutations for each speed condition (gray panels). The horizontal line at zero reflects inequality from the permutation. All errorbars represent one SEM.