Monitoring of attentional oscillations through Spectral Similarity Analysis predicts reading comprehension

Highlights

• We developed a Spectral Similarity Analysis (SSA) method to detect mind wandering.

• SSA compares 5 min resting state EEG and 5 min reading EEG without interrupting reading.

• The resulting measure correlated with reading comprehension and executive control.

• Readers with more mind wandering moments tended to have worse comprehension.

• Readers with higher executive control tended to have fewer mind wandering moments.

Abstract

Deviations of attention from the task at hand are often associated with worse reading performance (Schooler, Reichle, & Halpern, 2004). Ironically, current methods for detecting these shifts of attention typically generate task interruptions and further disrupt performance. In the current study, we developed a method to (1) track shifts of attention away from the reading task by examining the similarity between 5 min of eyes-closed-resting-state EEG and 5 min reading EEG; and (2) investigate, during reading, how the ratio between attention shifts and focused reading relates to readers’ comprehension. We performed a Spectral Similarity Analysis (SSA) that examined the spectral similarity between EEG recorded during reading and at rest on a moment-by-moment basis. We then recursively applied the algorithm to the resting-state data itself to obtain an individual baseline of the stability of brain activation recorded during rest. We defined any moment in which SSA during reading was greater than the mean correlation between resting-state EEG and itself as an “attentional shift.” The results showed that the proportion of such attentional shifts recorded over the left visual region (O1) significantly predicted reading comprehension, with higher ratios (indicative of more frequent attentional shifts) relating to worse comprehension scores on the reading test. As a proof of its validity, the same measure collected during the reading comprehension test also predicted participants’ Simon effect (incongruent - congruent response times) which is a common index of selective attention. This novel method allows researchers to detect attention shifts moments during reading without interrupting natural reading process.

Introduction

Sustained attention over extended periods of time is important in reading comprehensions (Blankenship et al., 2019, Jangraw et al., 2018). For example, both young and older adults who achieved higher scores on sustained attention task performed better on reading task comprehension (Jackson & Balota, 2012). Conversely, reading comprehension decreases during mind wandering, a debated construct that indicates intentional or involuntary shifts in attention from a given task or from external stimuli (Christoff et al., 2016, McMillan et al., 2013, Seli et al., 2016, Seli et al., 2017). Although it is not always central to the definition, such shifts typically reflect states in which an individual’s attention is directed toward internally generated thoughts. In the past three decades or so, researchers have been exploring different tools and paradigms to capture this “flight of mind” and to understand how it affects people’s emotion (Killingsworth & Gilbert, 2010), learning (Risko et al., 2012, Unsworth and McMillan, 2017), exam performance (Mrazek et al., 2013, Smallwood et al., 2007), and visual recognition (Lutz, Lachaux, Martinerie, & Varela, 2002; see Mooneyham & Schooler, 2013 for a review). In fact, self-reported attention shift frequency is often associated with poorer performance on cognitive tasks (McVay and Kane, 2009, Schooler et al., 2004). Despite its importance, only a few studies to date have investigated attention shifts and their effects on reading, and nearly all of them have relied on self-report or thought probes to identify attention shifts moments during reading (Broadway et al., 2015, Foulsham et al., 2013, Reichle, Reineberg, & Schooler, 2010, Schooler et al., 2004, Smallwood, 2011, Smallwood et al., 2008, Unsworth and McMillan, 2013).

Schooler et al. (2004) developed a self-report paradigm to detect readers’ shifts of attention during their reading of the opening chapters of War and Peace. They first familiarized readers with the concept they called “zoning out”—that is, when readers lost track of what they were reading or found that their thoughts were not related to the text. Then, readers were asked to press a key when they realized they were “zoning out” while reading. Results showed that readers who reported more shifts of attention away from the text also achieved lower comprehension accuracy. Using a similar technique, Unsworth and McMillan (2013) found that such attentional shifts during reading comprehension were predicted by readers’ working memory capacity: low-capacity readers reported more attentional shifts away from the text than did high-capacity readers, resulting in lower comprehension scores for the political science texts they were reading. McVay and Kane (2012a) further explored the hypothesis that the relation between working memory capacity and reading comprehension was mediated by executive/attentional control, and in particular to the ability to maintain attention on the task at hand. Using data gathered from multiple cognitive tasks, a structural equation model showed that measures of executive control explained differences in attention shifts between high- and low-working-memory-capacity participants during reading.

Instead of self-reports, Smallwood and his colleagues used comprehension probes at critical and random regions of the text to examine how readers’ attention shifts affect their inference-making ability (Smallwood et al., 2008). When reading a Sherlock Holmes detective story, participants were probed during inference critical episodes (section of texts that were relevant to the inference comprehension question) and during random episodes (section of texts that were not critical to generating inferences). Results revealed that readers who had more attention shifts away from the task moments during the inference critical episodes failed to answer the inference question. Interestingly, readers who mind wandered more at the beginning of the story also had worse comprehension than those who mind wandered at the end, which is consistent with the greater importance of the initial parts of a text to establish characters and settings.

In addition to self-report, Reichle et al. (2010) investigated attention shifts by comparing readers’ eye movement patterns during mindless reading versus engaged silent reading. Participants in this experiment read the entire book of Sense and Sensibility in hour-long sessions over multiple days. Researchers explained to participants the concept of “zoning out” and asked them to self-report their mind wandering moments during reading by pressing the “Z” button. Additionally, researchers used a prompt 2–4 min after each self-reported mind wandering moment to check if readers were mind wandering at the time of the prompt. Results revealed that, compared to engaged silent reading moments, mind wandering led to longer fixation durations, fewer regressions, and lower sensitivity to word length and frequency. In other words, although readers’ eyes were still fixating on the words, their inattention changed the extent to which they encoded and comprehended the text they were fixating on. Another study also observed similar characteristics of eye movements during mind wandering, showing that when reading sentences that varied in word frequency, mind wandering events were characterized by slower reading speed, longer fixations, and reduced sensitivity to word frequency as compared to engaged reading (Foulsham et al., 2013).

Taken together, these results suggest that accidentals loss of attentional focus during reading decreases the depth of encoding and comprehension of materials encountered. They also show that systematic differences in how frequently an individual mind wanders are related to performance in cognitive tasks that require inference making, working memory, and selective attention.

One challenge with these studies, however, is that all of the methods used to assess whether participants were paying attention to the main tasks also interfere with it in some way (see Gruberger, Ben-Simon, Levkovitz, Zangen, & Hendler, 2011 for a review of methods; Konishi, Brown, Battaglini, & Smallwood, 2017). For example, probes that ask readers to report on their experience (e.g. “Are you focused on the text?”) are often delivered during the task, which may interrupt the construction of a situation model as well as other online reading processes. In addition, when readers are asked to monitor their own mind wandering and self-report “zoning out” episodes, they must divide attention between reading and monitoring their own thought processes. This creates a situation akin to dual tasking or divided attention. To make matters worse, the relation between attentional shits and reading comprehension may be mediated by an individual’s ability to recover from either the distraction or the need to divide attention between the main task and the monitoring process. To circumvent these limitations, the current study employs a paradigm that measures fluctuations of attention noninvasively using continuous electroencephalography (EEG).

Acknowledging the limitations of behavioral paradigms, an increasing number of researchers have recently investigated possible physiological correlates of various states of attention. For example, using a self-paced reading task, Franklin, Broadway, Mrazek, Smallwood, and Schooler (2013) monitored readers’ pupil dilation changes at the same time that they received experience probes asking about their attention to the task (Franklin et al., 2013). They found that larger pupil dilations were associated with moments of mind wandering and distraction from reading; however, when using pupillometry to investigate mind wandering during other tasks (e.g. a monotonous breath counting task), the opposite finding was demonstrated (Grandchamp, Braboszcz, & Delorme, 2014). These conflicting results suggest that pupillometry may not be a stable physiological measurement for detecting mind wandering during reading.

Researchers have also employed event-related potentials (ERPs) to investigate the neurophysiological correlates of attention during reading and found that the size of the N1 component, a negative peak that occurs 100 ms after the onset of the target word, significantly predicted readers’ comprehension (Broadway et al., 2015). However, readers were interrupted 96 times by the thought probes (“Just now, were you mind-wandering?”) during the task. Moreover, texts were presented one word at a time at a fixed rate, which makes it hard to explore natural reading processing that usually involve regressions and saccades.

In addition to ERPs, quantitative EEG has also been used to explore the neural markers of attentional engagement (e.g. Braboszcz and Delorme, 2011, Macdonald et al., 2011). Macdonald et al. (2011) observed that participants’ attention affected oscillations in the alpha band (8–13 Hz); specifically, high engagement correlated with low alpha power while low engagement led to high alpha power. This finding is consistent with the body of literature showing that increased alpha power, especially over posterior regions, is associated with relaxation and lack of visual processing (Romei, Rihs, Brodbeck, & Thut, 2008). Similarly, Braboszcz and Delorme (2011) found that the moment when participants realized they were off-task and their mind was wandering, power in power in the alpha (9–11 Hz) band decreased. In addition, however, they also found that, when participants recovered from mind wandering, power in the beta (15–30 Hz) band decreased as well, while power in the theta (4–7 Hz) and delta (2–3.5 Hz) bands increased. Taken together, these findings suggest that mind wandering is associated with broad changes across the different electrophysiological frequency bands.

Because EEG and ERPs have limited spatial resolution, a number of other researchers have investigated the neural underpinnings of mind-wandering using other methods. In a landmark study, Christoff, Gordon, Smallwood, Smith, and Schooler (2009) adapted an experience-probing paradigm in a fMRI study. While participants performed a go/no-go attention task, a thought probe was randomly presented to collect their mind wandering information about (1) Whether their attention was on-task or off-task; and (2) If they were aware of their being off-task and mind wandering (note that individuals might or might not be unaware of own their mind wandering, and unaware mind wandering led to worse performance than aware mind wandering; Smallwood et al., 2007, Smallwood et al., 2008). The results showed that self-reported mind wandering episodes showed increased activity in the medial frontal, posterior cingulate, and temporoparietal cortices, that is, the network of regions collectively known as the Default Mode Network (DMN). There were no activation differences between the meta-awareness mind-wandering moments and those in the absence of meta-awareness, suggesting that mind wandering recruits similar brain regions no matter participants are aware that they mind were wandering or not. Other researchers have reported the similar activation of default network regions during task-independent thoughts, which also considered as mind wandering (Stawarczyk, Majerus, Maquet, & D'Argembeau, 2011).

The association between mind wandering and the DMN provides clues as to the nature of mind wandering and the states that elicit it. The DMN emcompasses a set of regions that are consistently deactivated during performance of experimental task (Christoff et al., 2009, Mazoyer et al., 2001, Raichle and Snyder, 2007, Shulman et al., 1997). Further research showed that these brain regions that were consistently deactivated during cognitive tasks or goal-directed behaviors were also activated during resting state periods, during which participants are not asked to do anything in particular (Raichle et al., 2001, Mason et al., 2007; See Raichle & Snyder, 2007 for a review). Note that, although the neuroimaging literature refers to recordings made of participants lying in the scanner while “doing nothing” as “resting state”, the term does not imply that the participants’ minds are neither empty nor peaceful (Christoff, Ream, & Gabrieli, 2004). Instead, researchers posit that intrinsic activity in the DMN may represent spontaneous, unconstrained, and not goal-directed thought, such as mind wandering, or stimulus-independent thoughts (Christoff et al., 2004, Mason et al., 2007).

Although EEG provides no direct way of measuring DMN activity, the fMRI research strongly suggests that “resting state” periods provide a way to naturally elicit the spontaneous fluctuation of thought that underlie mind wandering. For this reason, this study will examine mind wandering by comparing resting-state EEG with EEG recordings obtained during an active, goal-directed reading task.

To investigate the electrophysiological correlates of attentional shifts during reading, we borrowed methods that had been previously developed and successfully applied to fMRI data. In the context of fMRI, two competing approaches have been used to study and decode specific human brain states —machine learning-based classification (Norman, Polyn, Detre, & Haxby, 2006) and representation similarity analysis (RSA: Kriegeskorte, Mur, & Bandettini, 2008). In essence, classification methods employ supervised learning methods to discover data features that are most predictive of certain mental states, while RSA relies on the similarity between brain responses associated to different states or stimuli. Of the two families, classification-based methods have been used the most, for example, in EEG-based brain-computer interfaces (Rao, 2013). Representation similarity analysis, however, has specific advantages. In particular, it provides a continuous metric (similarity between mental states) that can be generalized across mental states and across individuals and can be applied to small amounts of data. Furthermore, classification is sensitive to optimal feature selection (with generalization decreasing when too many features are selected), while representation similarity analysis can easily accommodate large numbers of features.

Based on these considerations, we developed a new algorithm that detects oscillations in attention. As a benchmark, the algorithm uses characterizations of individual reader’s electroencephalographic (EEG) data obtained from a 5-minute, eyes-closed, resting-state recording as a representative sample of off-task mental state. This variant of RSA, which we call Spectral Similarity Analysis (SSA, in analogy to RSA in fMRI data analysis) involves a continuous comparison of the moment-by-moment spectral characteristics (the percentage of signal explained by oscillations in individual frequency bands) of a neural time series against the spectral fingerprint of the “reference” data obtained during the eyes-closed, resting state period. The basic assumption of this method is that the more engaged in a task an individual is, the more dissimilar his or her EEG will be to the resting-state reference; whereas the less engaged an individual is in a task (e.g., during mind wandering moments) the more similar his or her EEG will be to the resting-state data. Thus, our algorithm provides a continuous metric over time instead of discrete categories at sparse time samples.

As noted above, we used resting-state EEG as our benchmark because it best captures our participants' mental states while they are not engaged in any specific task. It also seems to share the spectral characteristics that have been previous associated with mind wandering (such as increased alpha power and decreased theta power), and, most importantly, it provides an individualized baseline that accounts of individual variations of spectral characteristics across individuals. In fact, previous research has shown that resting-state EEG captures significant amounts of individual variability in cognitive functions (e.g., Gou et al., 2011, Prat et al., 2016). For example, researchers have found that young children’s gamma power at rest significantly correlated with their later language development and they argued that high gamma power during resting-state indexed better attentional control and easier access to working memory (Gou et al., 2011). Recently, researchers found that resting-state EEG significantly predicted bilingual adults’ L2 language acquisition speed—resting-state EEG explained 60% of the variability in L2 learning (Prat et al., 2016). Thus, resting state data provides both a way to induce and measure mind-wandering activity and to characterize individual differences in neural activity.

The use of resting-state EEG as a benchmark also alleviates another complication of this research--namely, the fact that it is difficult to experimentally detect attention shifts non-invasively. This is, in fact, one of the obstacles to the application of machine-learning classification methods to mind-wandering research: the use of probing paradigms tends to yield few reliable samples that can be used as the ground truth for training a classifier. Furthermore, the use of within-task probes might alter the way participants perform the task, thus obscuring some of the data. In contrast, in our method mind wandering is elicited during resting state, and the moment-by-moment similarity of task-based EEG to resting states provides clues to possible shifts of attention without the need to interrupt a task. As a matter of fact, in this study we will report data collected from an unmodified standard reading comprehension test, the Nelson Denny Reading Test (Brown, Fishco, & Hanna, 1993).

Capitalizing on this approach, current study aims to (1) compare the spectral similarity between resting-state EEG and EEG obtained during reading on a second by second basis to measure individual differences in the number of times that the two brain states come to resemble one another (i.e., mind wandering moments); and (2) to investigate whether frequency of attention shifts predicts readers’ comprehension during the task; and (3) to see whether attention shifts frequency also relates to executive functioning.

The moment-by-moment spectral similarity between the resting-state EEG and reading EEG provides a stability index of whole brain neural responses at a given moment during reading. Based on the previous literature (e.g. Feng, D’Mello, & Graesser, 2013), we hypothesized that attention shifts should significantly correlate with readers’ performance—a high attention shifts ratio should yield worse reading comprehension. We also hypothesized that the attention shifts ratio during reading should significantly correlate with readers’ performance on the Simon task, which reflects’ people’ executive control abilities, because previous literature has reported that both children and adults’ executive control abilities correlated with reading abilities (Blair and Razza, 2007, McVay and Kane, 2012a, Yamasaki and Prat, 2014) and attention shifts significantly correlated with people’s executive control abilities (McVay and Kane, 2009, McVay and Kane, 2010, McVay and Kane, 2012b).