Tracking drivers’ minds: Continuous evaluation of mental load and cognitive processing in a realistic driving simulator scenario by means of the EEG

Driving safety strongly depends on the driver's mental states and attention to the driving situation. Previous studies demonstrate a clear relationship between EEG measures and mental states, such as alertness and drowsiness, but often only map their mental state for a longer period of time. In this driving simulation study, we exploit the high temporal resolution of the EEG to capture fine-grained modulations in cognitive processes occurring before and after eye activity in the form of saccades, fixations, and eye blinks. A total of 15 subjects drove through an approximately 50-km course consisting of highway, country road, and urban passages. Based on the ratio of brain oscillatory alpha and theta activity, the total distance was classified into 10-m-long sections with low, medium, and high task loads. Blink-evoked and fixation-evoked event-related potentials, spectral perturbations, and lateralizations were analyzed as neuro-cognitive correlates of cognition and attention. Depending on EEG-based estimation of task load, these measures showed distinct patterns associated with driving behavior parameters such as speed and steering acceleration and represent a temporally highly resolved image of specific cognitive processes during driving. In future applications, combinations of these EEG measures could form the basis for driver warning systems which increase overall driving safety by considering rapid fluctuations in driver's attention and mental states.


Introduction
Driving a car is a complex task and requires continuous adaptation of attention based on the demands of the present driving situation. There is a long history of accessing the mental states of drivers in connection to drivers' fatigue for a review see [1]. Among the neurophysiological methods used, EEG measures have turned out to be a valuable and promising tool to achieve this goal by measuring voltage changes directly on the scalp with high temporal precision without interfering with the task. Most studies, however, did not Abbreviations: bERP, blink-evoked event-related potential; bERSP, blink-evoked event-related spectral perturbation; CNV, contingent negative variation; EEG, electroencephalography; EOG, electrooculography; ERP, event-related potential; fERP, fixation-evoked event-related potential; ERL, event-related lateralization.

Participants
In this study, we recruited 20 participants without any prior or present neurologic or psychiatric condition, using online announcements and postings on bulletin boards at the Technical University Dortmund, Germany. All participants had no motor impairment, had a normal or corrected-to-normal vision, and were right-handed. Also, they had a driver's license for at least 3 years. Due to technical issues, only 15 datasets could be used for further analysis. The participants' age ranged from 20 to 26 years (mean = 22.53, standard deviation = 2.13), 6 were female, 9 male. The study was approved by the local ethics committee of the Leibniz Research Centre for Working Environment and Human Factors (March 24, 2017) and all participants gave written informed consent.

Task and procedure
When participants arrived at the lab around 9-10 a.m., they were fitted with a 10-20 system 32 electrode cap (actiCap, Brain Products GmbH, Gilching, GER). Participants then read the experimental instructions, gave their informed consent, and filled out questionnaires about demographic data, driving history and driving habits. After preparation, participants executed several conditions in a cognitive-motor dual-task paradigm which will not be reported here in detail. Right after this task, participants were escorted to the driving simulator laboratory, where they performed the driving task. This task consisted of a longer car ride with different road sections and typical driving situations of a German street environment. The ride took place in a static driving simulator (ST Sim, St Software B.V. Groningen, NL, see Fig. 1).
After a 5-10-min practice drive to familiarize the participants with the driving simulator, the route started with a two-lane freeway section. In this study, the focus lay on driving situations in which conspicuities in driving performance are typically observed, such as intersections, turning maneuvers (especially left turns), lane changes, and situations that require interaction with other road users. The following scenarios were implemented along the entire route: 52 intersections (9 with a left turn, 19 with a right turn, and 24 going straight; 5 with traffic lights; 4 with a stop sign; a total of 28 with right of way), 2 roundabouts, 34 speed limits (15 urban and 19 extraurban), 2 freeway entrances and exits, 9 sections requiring evasive maneuvers (road works, obstacles, 5 extra-urban and 4 urban), 2 overtaking passages (optional) and 1 passage with prohibited overtaking, 1 foggy passage, as well as other critical events (including obstacles on the roadway, crosswalk with pedestrians on the roadway, bus stops, traffic-calmed zone). After completing the initial freeway section, participants entered a rural road, before driving back to the highway. After another section on the rural road, they finally drove into the city. The route had an average to upscale requirement character, comparable to that of an official driving test as estimated by an experienced driving instructor, with an estimated duration of 60 min.
Acoustic and visual navigation signals guided the drivers and indicated upcoming turn maneuvers. These consisted of spoken instructions from a female German speaker and the simultaneous presentation of directional arrows on a dashboard screen. If participants did not comply with the instructions, they were re-routed to a section of the track just before the turn. The participants were instructed to keep to the speed limit. The entire trip was about 51 km long and took the participant between 36 and 50 min to complete the first 50 km, the section that was analyzed.
After both the practice drive and the test drive, participants filled out a Motion Sickness Questionnaire [25] to register any account of discomfort due to exposure to the virtual environment. Fig. 1. ST-Stim Driving simulator with a custom-made cockpit mock-up and three displays. An EEG-amplifier was situated behind the head rest.

Data recording
Electrophysiological data were recorded using a regular 10-20-system electrode cap (actiCap) that was fitted over the head of the participant which was equipped with 32 active electrodes (Fp1, Fp2, F3, F4, F7, F8, Fz, FC1, FC2, FC5, FC6, C3, C4, Cz, T7, T8, CP1,  CP2, CP5, CP6, P3, P4, P7, P8, Pz, PO9, PO10, O1, O2, Oz). After aligning the cap properly, the electrodes were filled with electrolyte gel until an impedance of ≤10 kΩ was reached to ensure proper signal quality. The electrode cables were carefully aligned not to cross or to move relative to each other in order to prevent electromagnetic noise. The cables were then routed through loopholes near the participant's ear and plugged into a LiveAmp 32 amplifier (BrainProducts GmbH, Gilching, GER) placed in the actiCap's pocket at the back of the participant's head. Data were recorded with a sampling rate of 500 Hz and a bit depth of 24 bits directly onto a micro-SD card inserted into the amplifier. The EEG data were stored together with data from the driving simulator (speed and position of the steering wheel).

EEG-preprocessing
Data were initially band-pass filtered from 0.1 to 40 Hz and then entered into the PrepPipeline [26]. This function detects corrupt channels based on iterative procedures leading to a robust average reference and interpolates artificial channels. These data were high-pass filtered at 1 Hz and down-sampled to 250 Hz for ICA decomposition. Data were segmented into epochs of 1s length and checked for artifacts. Before entered into an ICA [27] data were compressed using PCA to the rank corresponding to the number of non-corrupt channels minus 1. Obtained ICs were categorized using ICLabel [28]. IC-weights were then written back to the initially 0.1Hz filtered data with a 500Hz sampling rate.
Based on ICA decomposition, eye blinks were identified by gaussian fits [7]. Saccades were detected by searching for maximal velocity in the IC identified to reflect horizontal eye movements. From this first estimation of possible saccades, templates (visually checked for a correct saccade shape) were determined for 3 expressions and correlated with the time courses in the ICs. Only segments with a correlation higher than 0.8 were specified in more detail and used for further analysis. For each saccade, both the onset and the offset, which reflect a new fixation, were determined. For further analyses, each saccade was additionally labelled for its direction (left/right) and whether it was directed outwards (away from a central fixation) or inwards (towards a central fixation). This factor was labelled "type of saccades". For EEG analyses, ICs that were classified by ICLabel as brain activity with a probability of ≤ 30% were removed.

Track-based determination of mental load
All data were intended to be analyzed in a scene-based manner [4]. Therefore, the car's exact position at any time point was calculated based on the directly transferred velocity and 85 predefined trigger points. The latter were moments of distinct events with a known position on the track. Since velocity and steering angle were transcribed to the EEG system only with a sampling rate of 100 Hz, moments of acceleration or breaking may bias the exact calculation of waypoints based solely on these data. This error was corrected by morphing the exact distance values between two trigger points. Finally, the entire track was divided into segments of 10 m length based on the distance covered by the driver, resulting in 5000 segments overall. For further analyses, each time point was assigned to these segments.
Then, EEG-data were entered into a continuous wavelet transform from 2 to 30 Hz in 29 steps separately for all EEG channels. Each trace was z-transformed for the entire time and averaged for the 5000 segments of 10-m length [4]. The outcome of this procedure was smoothed by a moving average ( ± 50 m), before Theta power for a frontal cluster (F3, Fz, F4, FC1, FC2) and Alpha power for a posterior cluster (P3, Pz, P4, PO9, PO10, O1, Oz, O2) were extracted. For the EEG-based estimation of task load, the ratio of Alpha and Theta power was used: For each subject and each segment, the difference between these two measures was calculated before the median across subjects of these difference values was calculated for each segment. These median-values were divided into terciles, assigning each segment either low (predominant Alpha power), medium (quite similar Alpha and Theta power), or high task load (predominant Theta power).

Behavioral data
Average driving velocity, steering acceleration, as well as the frequency of blinks and saccades, were determined for each segment. Average values were calculated for each of these measures and for each level of mental load.

EEG data 2.5.1. Eye-event-evoked EEG activity
For all eye-event-related analyses, eye events (blinks, saccade onsets and fixations) were labelled according to the determined task load of the segment in which they appeared. For a list of all analyzed eye-event-related EEG activity, see Table 1.
For eye-blink-related activity, data were selected time-locked to the center of the fitted Gaussian function that was used to detect blinks (which more or less accords to the maximum deflection in the time course of the eye-blink-related IC). Segments from − 500 ms to 1000 ms were extracted. The baseline was set before blink onset (− 400 to − 200 ms with respect to the maximum). In the averaged ERPs, peak amplitudes were estimated using a jackknife procedure [29] for an occipital N1 (detection window: 20 to 200 ms; electrode cluster: O1, Oz, O2), a parietal P2 (80 to 200 ms, electrode cluster: CP1, CP2, P3, Pz, P4), an occipital P2 (80 to 250 ms) and a fronto-central N2 (detection window: 100 to 600 ms, electrode cluster: Cz, FC1, FC2, Fz).
The fixation-related activity was also analyzed in ERPs from − 500 to 1000 ms with a baseline during saccade occlusion (− 200 to 0 ms). The same electrode clusters as above were used. The occipital lambda response was determined between 0 and 200 ms, the occipital N1 between 20 and 200 ms, the parietal P2 between 100 and 300 ms, the occipital P2 between 100 and 400 ms, and finally the fronto-central N2 between 150 and 400 ms.

(Spatial) attention-related EEG activity
As a phasic ERL component, the fixation-related asymmetry in the N1 range was determined as the mean amplitude of asymmetry in the contra-ispi difference waves between 80 and 120 ms after fixation, in a posterior electrode cluster (P7/P8, PO9/PO10, O1/O2). The fronto-central CNV preceding saccade onset was measured as the mean amplitude between − 200 and − 50 ms in the above defined electrode cluster. As another sustained (slow) component with additional spatial aspects, the tonic ERL was measured in two time windows at posterior sites preceding the onset of the saccade (− 400 to − 200 ms and − 200 to − 50 ms). All parameters here were entered into ANOVAs with the factors task load and type of saccade.
Event related lateralizations of saccade-related EEG activity were also investigated with time-frequency analysis by calculating the event-related spectral perturbations (ERSPs) using 29 frequency traces between 2 and 30 Hz obtained from complex Morlet wavelet convolution.

Statistical analysis
Regarding the behavioral measures, the average values for driving velocity, steering acceleration, and the average frequency of blinks and saccades were entered into a repeated-measures analyses of variance (ANOVAs) with the within-subjects factor task load (low, medium, high). Additionally, the factor type of saccade (outwards vs. inwards) was analyzed for saccade frequency. The blinkrelated ERPs, that is the occipital, the parietal P2, the occipital P2, and the fronto-central N2 were analyzed using an ANOVA with the within-subjects factor task-load. Fixation-related EEG activity were analyzed using an ANOVA with the factors task-load and saccadetype (inwards versus outwards). This applies to the occipital lambda response, the occipital N1, the parietal P2, the occipital P2, and the fronto-central N2. Preceeding the saccades, the frontal CNV, as well as the posterior lateralizations were analyzed the same way.
All p-values obtained from ANOVAs were Greenhouse-Geisser corrected when indicated. As an indicator of effect size, adjusted

Fig. 2.
Histograms of driving parameters (number of segments), shown separately for the three levels of EEG-based estimation of task load. Note that all types of roads (freeway, state road, city) can appear in each level of task load. Segments with high velocity (freeway) are more often estimated as low task load, while segments with more steering acceleration are more probably estimated as high task load.
partial eta squared (adj η p 2 [30]; are presented. To keep the manuscript concise, effect sizes are only reported for significant effects. Regarding the results from the time-frequency analysis of EEG data time-locked to the saccades, the asymmetries in the EEG were investigated by calculating the ERSP for contra-and ipsilateral activity. Then, a cluster-based permutation test was performed on data averaged across all task load conditions to test the ERSP for ipsi-and contralateral activity against zero. The test used 1000 permutations and a clustering threshold of t-value corresponding to a p-value of p = .01, and a significance level for the cluster test statistic of p = .05. Since a distinct alpha suppression contralateral to saccade direction was observed (at least for inwards saccades) at PO7, alphapower for this electrode between − 800 and − 50 ms was extracted and entered in an ANOVA with the same factors as all analyses described above.
Blink frequency (see Table 2

Onset related activity
Assuming that both, the re-opening of the eye during a blink and the fixation of a new object denotes moments of a new visual impression, we analyzed ERPs time-locked to these events in a comparable fashion.

(Spatial) attention
Not only centrally evoked ERPs are of interest in the context of saccades, but also spatial components, since fixations follow either a leftward or a rightward eye movement. Based on this, event-related lateralizations of the EEG can be calculated (see Fig. 4, 1st and 3rd row and Table 4) defined by the difference between contra-and ipsilateral activity with respect to the spatial position of a stimulus.
While onset-related activity depicts the effects on input driven information processing, some effects were observed in preparation for a saccade, reflecting anticipatory mechanisms that appear to be driven by attempts to organize incoming information.
At fronto-central leads, a CNV (see Fig. 4 middle raw and Table 4) was observed that increased with task load, F(2,28) = 4.45, p =   Table 3 Means (and standard errors of means) in μV for the three levels of task load for all eye-event-related onset components derived from the ERPs.

Discussion
In the present study, we classified road sections of a naturalistic driving simulator task into three mental load categories. Based on the load classifications, we investigated the effect of cognitive load on information processing and attentional allocation using eyeactivity related EEG activity. A particular interest was to evaluate aspects of mental load not as a merely sustained state of the driver, but as a continuously changing condition that strongly depends on the driving situation and its current demands. Therefore, we conducted two types of analyses. First, we estimated situational task load based on a continuous transformation of Theta and Alpha activity in the EEG, as found in our previous studies [4,5]. Secondly, eye activity-related potentials were used to investigate specific stages of information processing and the distribution of attention within sections of a given task load.
Apparently, the categorization of different task load levels along the driving route by using EEG frequency measures in high temporal resolution revealed plausible results. Following this EEG-based task load categorization, vehicle velocity, and steering activity depicted comparable trends to a similar study with a high sample size [4]. The segments that were assessed to involve the lowest mental load by means of the EEG activity were composed of situations with little steering activity involved, most often found on freeway sections with little traffictherefore also showing high velocity values. Demanding situations, like passing a construction area or driving through a foggy passage (see also Fig. 6), were reliably detected by the algorithm and categorized as high task load situations, also comparable to our previous findings [4]. For example, the foggy road passage showed an initial increase in task load that continuously declined, accompanied by steadily increasing acceleration. Overall, these findings demonstrate the close link between driver behavior and electrophysiological measures, here particularly the ratio of Alpha and Theta activity, which is in line with previous studies. Since reduced Alpha activity is typically associated with states of increased mental workload and attentional effort e. g. [24,31], and increased Theta activity coincides with mental processing demands and task engagement[e.g [22,23], the ratio of Alpha and Theta activity is especially suitable for reliably representing driver states under natural conditions and the dynamic allocation of cognitive resources along the route see also [32]. The analysis of the blink rate and the saccades also revealed a clear dependence on the current task demand of a situation. Especially in passages classified as difficult, the blink rate decreased, while the number of saccades increased. An increase of saccades (especially outward saccades) in complex situations like city driving compared to less demanding situations like highway driving seems plausible, since in the former case much more visual stimuli at the roadside had to be attended to. Previous studies on the relationship between blink rate and task difficulty, on the other hand, have yielded partly contradictory findings: While some work found a decrease in blink rate with increasing difficulty, just as in our task [33,34], other work yielded opposite findings [35]. As Niezgoda and co-workers [36] state, this relation might strongly depend on the actual task and experimental setting. The suppression of blinks with increasing task difficulty is more likely to be observed in situations where subjects must strongly concentrate on visual stimuli, such as driving in a complex environment. Combined with the driving behavior, eye behavior results can be interpreted as a proof-of-concept that the EEG-based categorization of the driving track is valid and replicable. As we did not include time-domain EEG data nor blink activity for classification, an analysis of eye-activity related EEG activity is sensible and does not involve circular reasoning.
Regarding the analysis of event-related cognitive activity, the use of blinks as event-markers for mental load evaluation has proven beneficial recently. Previously, we have shown that eye-blink-related potentials (bERPs) of the EEG are sensitive to visual demands of natural behavior [7]. Moreover, bERPs are also found to be modulated by higher order cognitive demands like navigation [9]. In this study, modulations of bERPs showed basically the same directions as evoked by motor load, but only the N1 showed a significant decrease in amplitude with increasing task load. This effect has been interpreted as a correlate of attentional narrowing when the task at hand becomes more challenging [7]. The same interpretation may be true in the present driving task, in which a decrease in blink-related N1 has been found with increasing task demands.
Regarding the fixation-evoked ERPs, we observed a quite characteristic pattern that depended on the task load and type of saccade.

Fig. 5.
ERSPs and the outcome of the cluster-based permutation test against zero (depicted as the highlighted regions) for the contra-ipsi differences at PO7/PO8 (parietally) and O1/O2 (occipitally). While there were no significant differences for outward saccades in either the parietal or occipital region, inwards saccades demonstrated a sustained suppression of alpha activity contralateral to the direction of the saccade. These findings suggest a higher involvement of covert attentional processesmeasured as suppression in alpha powerbefore the initiation of the saccade.
In particular, N1 and N2 amplitudes decreased with increasing task load. P1, N1, and N2 amplitudes were larger with outward than inward saccades. Interestingly, P1 and P2 increased with increasing task demands, but only for inward saccades. This raises the question of how the two saccade types differ in a driving situation, ormore preciselywhat the functional difference between the two types of saccades might be. Regarding a driving task, the essential events should mainly occur in front of the driver, while less drivingrelevant events in the periphery (i.e., deviating from the driving direction). It can therefore be assumed that an outward saccade was triggered rather reflexively by a stimulus or an event in the periphery, that is, in a bottom-up fashion. In contrast, an inward saccade (i. e., back to the section of road ahead of the driver) should rather be consciously controlled, that is, planned and executed in a top-down fashion.
There is indeed evidence in our results that outward saccades are rather reflexive. None of the EEG indicators for attentional preparation (i.e., CNV, slow ERL, Alpha suppression) was pronounced for this type of eye movement, while the upcoming fixation evoked a phasic ERL, indicated that spatial signals were processed. Also, P1 and N1, both potentials associated with early, more stimulus-based processing, are more pronounced on outwards than on inwards saccades. Accordingly, a greater P1 has also been observed when regions of higher quantity and relevance of information were presented [14]. For inwards saccades, on the contrary, sustained preparation was observed both in slowly raising CNV and asymmetries that show increased negativity contralateral to the target of the saccade. Similarly, Alpha activity was continuously suppressed contralaterally to the direction of an inward saccade, indicating that the target of the saccade was attended to continuously. Although we did not record eye-tracking data, it appears plausible that inward saccades mainly returned to the center of the vehicle. Thus, inward saccades lead, in most cases, to a renewed fixation of the road. In particular, the Alpha asymmetry indicates that attention is never taken off the road.
In addition to the early P1-N1 complex, task load modulated the fronto-central N2 of the fERPs, being decreased in amplitude when Fig. 6. Examples of task load assignments in two particular driving segments (color-coded: red = high, yellow = medium, green = low task load), basic driving parameters (driving velocity and steering acceleration, upper and center plots), and z-transformed thetaalpha activity (which was the EEG basis for task load categorization, lower plots). In the left column, the entry and exit of a foggy passage are depicted. Task load is high when entering the fog and drivers decelerate. While passing the foggy part, the task load continuously decreases while drivers accelerate. In the right column is a short section of the freeway, with two construction sites without a speed limit but needing to decelerate before drivers are directed to a freeway rest area with a quite busy entry. All these distracting events locally increase the task load. When decelerating and leaving the freeway, where drivers additionally drive up a construction vehicle, thetaalpha activity shows a distinct maximum. (For interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.) the task became harder, as it was previously observed for bERPs during walking. Moreover, the N2 was larger for outward saccades. The N2 is typically associated with (often conflict-related) cognitive control processes [37,38], for example, as required in the flanker task. Here, the N2 was found to be larger the more executive control was needed to manage the conflicting situation [39]. An increased conflict potential should also be associated with outward saccades, namely when peripheral stimuli distract the driver's attention from the driving-relevant events on the road.

Limitations
The presented data are plausible, but at some points, they still show some weaknesses, which do not yet allow a final assessment. Basically, such realistic driving scenarios lack temporally precise events for the time locking of the EEG. Eye events can be used to keep the situation naturalistic and without experimental interference while being able to have a substantial number of events necessary for the computation of an ERP. Though having the reputation to be a mere workaround, a number of recent studies showed both the reliability and validity of eye-behavior and its neurophysiological correlate in event-related potentials [7,8,40]. Unfortunately, both in-depth eye movement data and lane keeping data were missing, which would allow a local assessment of driving performance. Also, the sample size of the investigated group was rather small, nonetheless high effect sizes were obtained. Adding to that point, the results of the EEG-based categorization of task load and its relation to driving speed and steering variability correspond to those of a previous study with many more subjects [4]. Finally, although the large number of neurocognitive variables considered and analyzed here provide a fairly consistent picture, the contributions of the individual measures to a robust and overarching indicator of driver states need to be further explored in future studies. Therefore, this manuscript is meant to be a conceptual take on the objective assessment of driver perception and attention without an additional task within naturalistic environments.

Conclusion
The results show that eye-event-related EEG activity is very sensitive to task load in a naturalistic driving task and represents a temporally highly resolved image of specific cognitive processes during driving. For the future, one can imagine that a combination of the parameters presented here could form the basis of a valid driver warning system that goes far beyond a purely frequency-based analysis of the EEG as it has been common for many years. By using events inherent to human behavior, EEG is a tool that enables us to look at attentional allocation in high temporal precision, even when attention is covered and without the involvement of a behavioral driving component, making this procedure very valuable for driving safety research. Blink-and fixation-related potentials might be a promising tool to evaluate (a) the overall attentional resources (fERP: lambda, bERP: N2, P3), and (b) the spatial allocation of attention (sERP: ERLs, both phasic and tonic). These measures might be used to investigate safety-critical events for drivers in different cognitive states (drowsiness, aging) and how well they manage to allocate visual attention in a manner to anticipate and avert upcoming hazards while driving. While other assessments only involved eye activity measures, using EEG can help to understand the (attentional) motivation behind eye behavior and how much attention can be used to assess the upcoming situationeven preceding eye behavior itself. Especially, the time course of processing safety-critical events can be investigated, leading to new insights into what factors lead to the occurrence of driving errors. Finally, countermeasures can be developed on the premise of considering shifts in attention detected by the means of EEG correlates and therefore increase overall driving safety.

Author contribution statement
Edmund Wascher: Conceived and designed the experiments; Analyzed and interpreted the data; Wrote the paper.