Clear Evidence for Electrophysiological Signatures of Duration and Rhythm Prediction, but not across Sensory Modalities

Predicting the timing of events is fundamental for adaptive behaviour. An electrophysiological marker (temporal) Mismatch Negativity (tMMN) is often found following violations of expected temporal pattern of event sequences. A confounding factor in interpreting temporal mismatch paradigms is that violations of expected interval are accompanied by violations in feature or rhythmic pattern. Recently, Chang and colleagues (2017) disentangled duration and rhythm in a mismatch paradigm, presenting sequences of visual or auditory pairs with constant or pseudo-random interpair-intervals. Multivariate pattern analysis of EEG data replicated tMMN for rhythmic patterns, and additionally found signatures of tMMN specifically for duration. Temporal generalisation analysis showed that signatures of duration prediction violation generalised across sensory modality. However, results were based on only 15 participants, and report accuracy of decoder performance, which is understood to not accurately represent differences. To address these issues, we reanalysed the data, replicating evidence for predictions about auditory duration, and generalization across constant/pseudo-random presentation, though found no evidence that predictions generalized across modality. These results raise questions about the conclusions of the original study. Consequently, we will run a full replication of the study to resolve the precise nature of the electrophysiological correlates of temporal prediction.


Introduction
The human brain presents the remarkable capacity to predict the timing of events in a sequence. This ability has been widely investigated and is reflected in an electrophysiological marker known as (temporal) Mismatch Negativity (tMMN) -elicited by violations of expectation (Garrido et al., 2009). Despite extensive investigation, the nature and neural underpinnings of tMMN are far from clear. Violations of the temporal properties of event sequences are studied in temporal oddball paradigms wherein an event is considered standard if it maintains the rhythmic structure of a sequence, and deviant if it violates it (Chen et al., 2010). However, understanding the precise nature of the violated prediction in tMMN is confounded by the fixed rhythmic structure of stimulus presentations, thus conflating predictions of interval duration with those related to rhythmic pattern. Recently, Chang, Seth and Roseboom (2017) demonstrated that these different contributions can be disentangled by presenting stimuli of different durations and modality within rhythmic (fixed ITIs -isochronous) or arrhythmic (pseudo-randomly sampled ITIs -anisochronous) structures. Combining a univariate ERP approach with multivariate pattern analysis and temporal generalization analysis (TGA - King & Dehaene, 2014), they showed that violation of duration predictions elicited a deviation in the EEG signal regardless of rhythmic structure, providing evidence for the coding of predictions related specifically to duration. Additionally, by training a classifier to distinguish standard and deviant trials in 1083 This work is licensed under the Creative Commons Attribution 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/3.0 one sensory modality and then testing it on another, they provided evidence for a supra-modal, modalitygeneral mechanism of duration prediction. In spite of these potentially exciting findings, the study was based on a sample of only 15 participants and reported only decoding accuracy in combination with the use of a support-vector machine (SVM) for the unbalanced design, which may have produced an overly optimistic estimate of the results. Here we will present a full reappraisal of these findings.

Procedure and Design
Fifteen healthy students were recruited from the University of Sussex. The experiment was a temporal oddball paradigm in which durations were defined by two transient stimuli. The duration could be 150ms or 400ms. Participants observed stimulus presentation passively. In one block of trials, the standard duration was 150ms (200 trials) and deviant 400ms (50 trials). In the other block of trials, the standard and deviant duration were switched. There were three experimental conditions: auditory isochronous (A-ISO), auditory anisochronous (A-ANISO), and visual anisochronous (V-ANISO). Auditory stimuli were 10ms pulses of 1500 Hz pure tones. Visual stimuli were 10ms flashes of luminance-defined Gaussian blobs. The inter-trialinterval (ITI, time between one pair and the next) was either fixed at 1750ms (isochronous) or drawn from two uniform random distributions between 1000-1500ms or 2000-2500ms.

EEG Acquisition and Preprocessing
64 channel EEG data were recorded at 2048 Hz and preprocessed with the EEGLAB toolbox (version 14.1.2.b - Delorme & Makeig, 2004) running on MATLAB (R2018a). Data were downsampled at 512 Hz and were separately high-pass filtered at 0.1 and lowpass filtered at 45 Hz. Bad channels were automatically detected with the clean_rawdata() function and subsequently spherically interpolated. EEG data were then re-referenced to the common average of all 64 electrodes. Eye movements and blinks were corrected with independent component analysis (FastICA -Hyvarinen, 1999). As recommended by Winkler and colleagues (2015), ICA was fitted to a version of the dataset that was high-pass filtered at 1Hz to improve the decomposition. To ensure compatibility, the 1Hz dataset presented the same features of the "original" one in terms of both cropping and removed sensors. The ICA weights were then projected to the 0.1 Hz filtered dataset for further classification. Eye components were automatically classified and removed by means of ICLabel (Pion-Tonachini, Kreutz-Delgado, & Makeig, 2019). The 0.1 Hz dataset was then epoched in respect to the first presented stimulus of each pair (S1) between -100 and 850 ms and each epoch was baseline corrected (-100 to 0 ms). Bad epochs were automatically detected with a customised script.

Multivariate Pattern Analysis (MVPA)
All decoding analyses were run with the Amsterdam Decoding and Modeling toolbox version 1.07 (ADAM -Fahrenfort et al., 2018) in MATLAB (R2018a) on a downsampled version of the preprocessed data (128 Hz). To test whether standard and deviant stimuli showed different electrophysiological patterns in each condition, a backward decoding classification algorithm (LDA) was used. All 64 EEG channels were features, with Standard and Deviant labels within each condition as classes. The classifier was trained and tested on each datapoint (122 per epoch) using 10-fold cross validation. Classifier performance was averaged across all folds. Classifier performance was measured by the Area Under the ROC Curve (AUC). Due to the imbalance between standard and deviant trials (4:1 ratio), deviant trials were oversampled by means of the ADASYN algorithm (He et al., 2008). In addition, withinclass balancing was ensured by means of undersampling.
Decoding across conditions. To assess whether duration predictions in both isochronous and anisochronous conditions, and across different sensory modalities, shared common underlying neural information, we used the temporal generalization analysis (TGA; King & Dehaene, 2014). The core feature of this approach lies in its ability to generalize classifiers' performance across time instead of training and testing on the same datapoint. The analysis followed the same 10-fold cross validation approach reported above, but in this case each classifier was trained on one condition and tested on another one (across rhythmicity: A-ISO vs A-ANISO and across modality: A-AISO vs V-AISO).

Statistical analysis.
Statistical significance at the group-level was assessed by t-testing against chance level (0.5 for AUC) and then applying cluster-based permutation to correct for multiple comparison (1000 random permutations).

Results
The reported analysis is specifically focused on violations elicited by short-ISI stimuli (150 ms). In this context, a prediction violation of duration is represented by a stimulus occurring earlier than predicted (unexpected presence). Contrasts are thus based on the comparison of same physical stimuli but having different contextual meaning (standard vs deviant) within sequences presented in the counterbalanced blocks.

Decoding Within Conditions
LDA was able to discriminate between standard and deviant EEG pattern with above chance performance in both the A-ISO for two clusters (time window: 229-526 ms, p<.001; time window: 573-636 ms, p<.005; peak AUC: 0.64) and A-ANISO conditions for one cluster (time window: 315-378 ms p<.01, peak AUC: 0.59). However, no period of significantly above chance decoder performance was found for the V-ANISO condition. Results are depicted in Figure 1 above.

Decoding Across Conditions
Decoding across rhythmicity with TGA in the auditory modality highlighted the presence of shared neural information when processing prediction duration violations. More specifically, when training the classifier to distinguish standard and deviant EEG patterns for A-ISO condition and then testing it on the A-ANISO condition we found a statistically significant on-diagonal cluster (p<.001, peak AUC: 0.59). In addition, when applying the reverse TGA (training on A-ANISO and testing on A-ISO) we also found a statistically significant on diagonal cluster (p<.001, peak AUC: 0.61). However, no significant cluster was detected when applying cross-decoding across sensory modality in the A-ANISO and V-ANISO conditions. Results are reported in Figure 2 below.

Discussion and Conclusions
Our re-analysis successfully replicated the well-known electrophysiological signatures of violations of rhythmic predictions, as well as the results reported by Chang et al. (2017) related specifically to predictions of durationbut only for the auditory, not visual modality. Additionally, again consistent with the original study, across-rhythmicity decoding using TGA highlighted the presence of similar neural information for violations of both rhythmic and arrhythmically presented stimuli. However, we found no evidence for the across-modality generalization of prediction violation reported in the paper. While supporting the existence of separate neurophysiological signatures for duration and rhythm in audition, these results, and the aforementioned limitations, call for further investigations to resolve whether electrophysiological responses to violations of duration predictions share any common components across sensory modality.