Prediction of cognitive conflict during unexpected robot behavior under different mental workload conditions in a physical human–robot collaboration

The experimental protocol was approved by the institute ethics committee (ETH21-6346). We had 24 participants (age 23 ± 4, 13 males and 11 females, all right-handed) in this study. Participants received a reimbursement of 60 AUD for their time and involvement. All participants selected for the study had no neurological or psychological disorders that might impact the experiment's results; they could wear glasses for vision correction. Additionally, the chosen participants were non-smokers and were asked to avoid drinking tea or coffee for at least 6 h before the experiment began.

The EEG data were collected using a wireless MOVE system (Brain Product GmbH, Germany) with 32 channels. The channels recorded were: Fp1, Fp2, F7, F3, Fz, F4, F8, FT9, FC5, FC1, FCz, FC2, FC6, FT10, T7, C3, Cz, C4, T8, CP5, CP1, CP2, CP6, TP9, P7, P3, Pz, P4, P8, TP10, O1 and O2 with AFz as the reference. The impedance of the electrodes was maintained below 20 kΩ, and the sampling rate was 1 kHz. The impedance of the electrodes was checked and confirmed to be below 20 kΩ approximately one hour after the start of the experiment while the participants rested after the second block of the experiment. The events from the experiment scenario were synchronized with EEG data with Lab Streaming Layer [55].

The ANBOT [53], an intelligent industrial robot for abrasive blasting featuring a UR10 robot arm by Universal Robots, was the collaborative robotic partner in this experiment. The participants were in contact with the robot arm through a handle mounted on the robot end-effector (see figure 1(A)). The interaction forces and torques are measured and controlled by the ATI mini-45 transducer, placed between the robotic arm and the end-effector.

Zoom In Zoom Out Reset image size

The game-like primary task emulated a blasting task and employed a secondary task, mental calculation of varying difficulties, to manipulate the workload of the participants. The task was presented as an interactive game called the 'Clock Game' on a 65-inch-wide LED TV. During the game, the point where the nozzle of the ANBOT pointed to was represented by a blue dot on this screen, and for this experiment, the ANBOT was limited to 2-DOF. The participants were instructed to grasp the button on the back handle of the robot end-effector to move the robot arm. They also had to grasp the button on the front handle to blast the area where the nozzle moved. The area being blasted was shown on screen in blue and this kept the participants engaged in this task, and they used both hands to move the robot arm.

Initially, the participants performed a training session where they performed 30 trials of the task to familiarize themselves with the task and move the robot arm. The experiment was divided into three blocks with 90 randomized trials (figures 1(B) and (C)). Each block started with a fixation period of 10 s, where the participants looked at a fixation cross on the screen. After this, the initial starting arithmetic equation was shown on the screen. For the first trial, the participant only performed the calculation in the allocated five seconds and had not moved the robot. This initial blasting trial was discarded in the EEG, task performance and physical performance analysis. The arithmetic equation was either adding or subtracting one-digit, two-digit, or three-digit numbers.

In the low workload condition, the arithmetic equation was a one-digit number addition or subtraction (e.g. 9 − 1). For medium workload condition, the arithmetic equation was the addition or subtraction of two-digit numbers with no carryover (e.g. 13 + 24). For the high workload condition, the arithmetic equation was either the addition or subtraction of three-digit numbers. Also, the addition involved carryover in the one's and ten's places (e.g. 135 + 168) and the subtraction of three-digit numbers with borrowing from hundred's and ten's place (e.g. 423 − 187).

In the first trial, the participants performed the calculation, after which, a large circle with a red dot at the center and four green dots along the perimeter was shown. Each of the green dots in the perimeter had a number next to it. The participants were asked to identify the current target, which was the green dot, with the answer to the arithmetic equation that they had previously solved. This 'Identify Target' phase was presented for 2 s, within which the participants had to identify the target green dot.

Afterwards, the participants were instructed to move the blue dot to the center of the big circle or the red dot. When the participants brought the robot's nozzle represented by the blue dot to the center, the participants received instructions to 'Move to Target' and 'Calculate' along with an arithmetic equation of random difficulty. The participants then moved the robot arm while performing the calculation, and they had five seconds to complete the blasting task or the trial reset. In order to maintain participants' interest in the game, we displayed a score on the screen, which was calculated based on the sprayed area and the target identification accuracy. The arithmetic equation disappeared from the screen as soon as the robot arm reached the target position ensuring that participants performed the calculation simultaneously while moving the robot arm.

The unexpected robot stopping was achieved by placing an invisible obstacle placed three-fifths on the path to any potential target or green dot. The screen also turned black when the participants reached the obstacle to corroborate the occurrence of the error. We also ensured that the participant would not be able to move around the invisible obstacle to try and reach the target, as the invisible obstacle takes the form of an annular barrier that restricts any movement of the robot arm. The robot arm remained stuck until the blasting phase resets. The probability of occurrence of the invisible obstacle in the blasting path was limited to 33% to prevent participants from generating an internal cognitive anticipating an unexpected robot stopping. Therefore, within each block, there were 90 trials of each workload condition with 60 trials of normal blasting and 30 error trials, and the trials with unexpected robot stopping were completely randomized.

After the blast phase, the participant received feedback with messages, such as 'Yay! Correct target', 'Oops. Wrong target' or 'Sorry. Time is up' in case of the normal trials. They received feedback such as 'Aborting trial. Correct Direction', 'Aborting trial. Wrong Direction' or 'Sorry. Time is up' in case of obstacle trials. We provided such feedback messages to keep the participants engaged in the task. After the feedback, which is shown for 1.5 s, the participants were asked to identify the target again based on the equation's answer before they were instructed to 'Get back to center'. During each block, the participants were randomly asked to complete the NASA-TLX questionnaire to record their subjective workload ratings. The NASA-TLX questionnaire evaluated the mental demand, physical demand, temporal demand, performance, frustration and effort measures. The participants provided a score between 0 and 100 for each of these measures every time the NASA-TLX questionnaire sought their subjective workload ratings in a trial. Following the participant's identification of the target during the 'Identify Target' phase, the experiment paused to record the participant's NASA-TLX ratings. Afterward, it resumed with the 'Calculate and Blast' phase. In our analysis of the Behavior data, we excluded the trials where NASA-TLX ratings were gathered to calculate accuracy. For each workload condition within a block, we collected NASA-TLX ratings twice from every participant.

Every participant performed three blocks and each block had randomized workload trials with Normal and Obstacle conditions. Every block was approximately 18–20 min, and participants rested for 10 min before proceeding to the subsequent block. All participants performed the clock game for approximately 1.5 h.

The control system that achieved pHRC in our experiment is outlined in [30, 31]. We employed an admittance-based controller [56] to convert the forces applied by the operator on the handles into the desired velocities of the robot in Cartesian space. To maintain the robot's admittance without introducing a singularity avoidance strategy, we restricted the task to a specific area within the robot's workspace. This ensured that the Jacobian matrix remained invertible in the designated workspace without requiring an active singularity avoidance algorithm, which could have led to unintended opposing forces. The calculation for the desired Cartesian velocities is shown below:

$$\begin{equation}{{{\dot X}}_{\text{d}}} = A\left( {{f_{\text{H}}} + \,{f_{\text{D}}} - \,{f_{\text{R}}}} \right).\end{equation} \tag{ 1 }$$

Here, ${{{\dot X}}_{\text{d}}}$ represents the desired velocities of the robot's end effector in Cartesian space. $A$ stands for the diagonal matrix holding the admittance gains, ${f_{\text{H}}}$ indicates the forces exchanged with the human operator, ${f_{\text{D}}}$ refers to a force vector that smooths out the Cartesian velocities, and ${f_{\text{R}}}$ represents the resistive force introduced to manipulate cognitive conflict. The admittance gains are specifically set at 0.01 m sN⁻¹ for the linear parts and 0.4 rad sN⁻¹ for the angular parts. To calculate ${f_{\text{D}}}$, we multiply the actual Cartesian velocity of the robot's end effector, ${{{\dot X}}_{\text{r}}}$, by a diagonal matrix of gains serving as a damper $D$.

In our experiments, we restricted the movements of the robot end effector in certain directions and orientations using a simple P controller. This controller manages the end effector's position at specific set points within those directions. Using $i$ to represent the directions and orientations in Cartesian space (where $i = 1,\,2$ for the two active directions in the planar task), we can summarize the control system for the experiment as follows:

$$\begin{equation}{{\dot X}}_{\text{d}}^i = A\left( {f_{\text{H}}^{{\,}i} + \,f_{\text{D}}^{{\,}i} - \,f_{\text{R}}^{{\,}i}} \right),\,{\text{if}}\,i = 1\,or\,2.\end{equation} \tag{ 2 }$$

In this setup, $x_{\text{R}}^i$ represents the real Cartesian position in the $i$th direction, while $x_{\text{D}}^i$ is the targeted Cartesian position. The operator solely presents ${f_{\text{H}}}$, and their perceived admittance is ${A^*} = \,\frac{{{{{{\dot X}}}_{\text{d}}}}}{{{f_{\text{H}}}}}$. The resistive force, ${f_{\text{R}}}$, acts like a compressed spring, opposing movement and influencing the admittance in our experiment. To adjust both the admittance and its rate of change, we utilize the resistive force ${f_{\text{R}}}$, which is calculated based on Hooke's law:

$$\begin{equation}{f_{\text{R}}} = \,{K_{\text{R}}}\,{x^\sim }.\end{equation} \tag{ 3 }$$

The stiffness gain, represented by ${K_{\text{R}}}$, gets multiplied by ${x^\sim }$, which stands for the difference between the cursor's position (the point aimed at by the ANBOT's nozzle), ${x_{\text{p}}}$ and the position where the resistive force begins, ${x_{\text{s}}}$. The position where the robot's movement is restricted is ${x_{\text{t}}}$, corresponding to the target in Normal trials or the hidden obstacle in Obstacle trials. As the cursor moves from its starting point, ${x_{\text{f}}}$ sits just before the target or obstacle, positioning the start of the force at ${x_s} = \,{x_t} - \,{x_f}$. Consequently,

$$\begin{equation}\left\{ {\begin{array}{*{20}{@{}l}} {{f_{\text{R}}} = 0\kern18.3pt\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,{\text{if}}\,{x_{\text{p}}} \unicode{x2A7D} \,{x_{\text{s}}}} \\ {\,{f_{\text{R}}} = \,{K_{\text{R}}}\,\,\left( {{x_{\text{p}}} - \left( {{ }{x_{\text{t}}} - \,{x_{\text{f}}}} \right)} \right)\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,{\text{if}}\,{x_{\text{s}}}\, \unicode{x2A7D} \,{x_{\text{p}}} \unicode{x2A7D} \,{x_{\text{t}}}\,\,\,} \end{array}} \right.\end{equation} \tag{ 4 }$$

Once ${x_{\text{p}}}$ reaches ${x_{\text{t}}}$, the robot's movements come to a halt. To create a sudden resistance upon reaching the obstacle or target, ${x_{\text{f}}}$ is positioned 20 pixels (equivalent to 2.3 cm) before the center of either the target or obstacle.

The NASA-TLX scores were randomly collected from the participants for all the six conditions: low, medium and high workload conditions, for both normal and obstacle trials in a block. The mean scores of each subscale were then calculated for each condition.

The performance in the obstacle trials of the blasting task was evaluated by (i) the target identification accuracy and (ii) the time to reach the obstacle. Target identification accuracy for each workload level was the ratio of the number of trials where the participant correctly identified the target to the total number of trials,

$$\begin{equation}{\text{Accuracy}} = \frac{{{\text{n}}\_{\text{correct}}}}{{{\text{n}}\_{\text{total}}}}\end{equation} \tag{ 5 }$$

where n_correct was the number of trials where participants correctly identified the target and moved in the correct direction before reaching the obstacle, and n_total was the total number of obstacle trials. The time taken to move the robot arm from the center of the circle to the invisible obstacle in the blasting path to any potential targets was also considered for the different workload conditions. Further, the mean applied force and mean end-effector velocity was extracted from the force-torque sensor (ATI Mini-45) placed between the handle and blasting nozzle.

EEG was preprocessed using EEGLAB v2022.0 toolbox [57] in MATLAB R2020b (The Mathworks, Inc., Natick, MA, USA), as shown in figure 2. The data was down-sampled to 250 Hz, before band-pass filtering between 1 and 45 Hz using eegfilt function in EEGLAB, which performs two-way least-squares finite impulse response filtering. Line noise was removed using the clean_line function of EEGLAB which estimates and removes sinusoidal noise from the data using Thompson F-statistic and multi-tapering. The channels with flat lines for over three seconds were eliminated using clean_flatline function in EEGLAB. Subsequently, using the clean_channels function, the noisy channels were also identified and removed. On average, 2 ± 0.4 channels were removed, which were later restored using the spherical spline method, which interpolated the data from neighboring channels. A copy of this dataset was made, and using pop_rejcont, continuous artifactual regions were cut off. Then, the clean_windows function was used to perform window cleaning. After these artifact removal steps, independent component analysis (ICA) [58] was performed using EEGLAB's AMICA algorithm [59, 60].

Zoom In Zoom Out Reset image size

After decomposing each participant's data by ICA, we employed ICLabel plugin [61] in EEGLAB to identify and reject all the components with less than 30% of brain activity. Subjects had 5 ± 0.76 ICs remaining after this step. Then, the ICA information (icawinv, icasphere, and icaweights) was copied back to the original dataset and component data was back-projected to the sensor level. All the blasting epochs were extracted, and three datasets corresponding to the three workload levels of blasting were created. We further extracted the obstacle trials from these three blasting datasets to form another three datasets with obstacle trials under low, medium and high workload conditions in order to study the PEN and Pe modulations across the three workload conditions.

The data from multiple participants were managed and processed using the EEGLAB STUDY structure [62]. A Study was created with a group (of 24 participants) with six conditions corresponding to the three workload levels and the presence or absence of obstacle. The components for all participants were clustered using k-means clustering algorithm [63] based on three equally weighted (weight = 3) criteria: (1) dipole locations, (2) dipole orientation, and (3) scalp maps to produce six clusters. The frontal, central and parietal clusters were identified using Talairach coordinates [64] of the fitted dipole sources of these clusters.

For the frontal, central, and parietal clusters, the power spectral density (PSD) of each component was evaluated using the spectopo function, considering each 1 s segment using a Hamming window. For each IC of the frontal, central and parietal clusters, the mean (PSD) at delta (1–4 Hz), theta (4–8 Hz), alpha (8–12 Hz) and beta (12–30 Hz) and gamma (30–45 Hz) bands were examined for the 1 s immediately after reaching the target in normal conditions and for the 1 s immediately after reaching the obstacle in the obstacle trials. The calculated PSD at each IC was normalized using the fixation period as the baseline to compare the changes in PSD upon reaching either target or obstacle in different workload conditions.

We also created a simple ERP Study, implemented in EEGLAB, with a group (of 24 participants) with three conditions corresponding to the three workload levels. The components for all participants were clustered using k-means clustering algorithm [63] based on three equally weighted (weight = 3) criteria: (1) dipole locations, (2) dipole orientation, and (3) scalp maps. The channel-based ERP data was evaluated at all the 32 channels. PEN was measured as the minimum error negativity amplitude in the 100–350 ms window after reaching the obstacle, and Pe was measured as the maximum error positivity in the 350–450 ms window after reaching the obstacle.

We employed deep learning algorithms to predict PEN and Pe amplitudes from the EEG data prior to reaching the obstacle. The two seconds of preprocessed EEG data prior to reaching the obstacle was used as the input to EEGNet, ShallowConvNet and DeepConvNet [65] to predict the PEN and Pe amplitude at the Fz channel. We used 80% of the data for training and it was in the (1026 × 500 × 29) format and 20% for validation (342 × 500 × 29) and another 20% for testing (342 × 500 × 29). Using hyperparameter optimization, we identified and employed the optimal batch size = 128, dropout rate = 0.5 and Adam optimizer.

SPSS (IBM SPSS 26.0; Chicago, IL, USA) statistical tool was used to conduct all the statistical analyses in this study. Using the Shapiro–Wilk test, it was ascertained that the data was normally distributed. The variations of the NASA-TLX scores with the workload levels and the presence and absence of obstacle was evaluated using two-way repeated measures analysis of variance (ANOVA) test with workload levels and trial type (normal or obstacle trial) as within-subjects factors. The variations of EEG parameters, task and physical performance at three workload levels of the task were investigated by conducting a one-way repeated-measures ANOVA with the workload level as the within-subjects factor. We used Mauchly's test to test for sphericity, and if sphericity was dissatisfied (p < .05), Greenhouse–Geisser correction was performed. Posthoc comparisons were conducted with Bonferroni corrections (α = 0.05) if the main effect of ANOVA was significant.

The subjective workload measures of the NASA-TLX increased with increasing difficulty levels in the arithmetic task in the clock game task with unexpected robot stopping, as shown in figure 3. The results of the two-way repeated measures ANOVA revealed that there was significant main effect of workload on participants' mental demand scores (F(2, 48) = 1499.764, p < .01, η_p ² = .980). Also, the ANOVA revealed that mental demand scores during trials with unexpected robot stopping were significant (F(1, 24) = 163.748, p < .01, η_p ² = .845). Further, there was a significant interaction between workload and unexpected robot stopping (F(2, 48) = 3.191, p = .048, η_p ² = .096) such that participants reported increasing mental demand scores with the increasing workload during conditions with unexpected robot stopping than the normal conditions. There was a significant main effect of workload on participants' physical demand scores (F(2, 48) = 403.112, p < .01, η_p ² = .931). Also, the ANOVA revealed that physical demand scores during trials with unexpected robot stopping were significantly higher compared to normal trials (F(1, 24) = 408.133, p < .01, η_p ² = .932). Further, there was a significant interaction between workload and unexpected robot stopping (F(2, 48) = 26.118, p < .01, η_p ² = .465) such that participants reported greater physical demand scores with the increasing workload during conditions with unexpected robot stopping than the normal conditions.

Zoom In Zoom Out Reset image size

The results of the two-way repeated measures ANOVA revealed that there was a significant main effect of workload on participants' temporal demand scores (F(2, 48) = 1177.726, p < .01, η_p ² = .975). Also, the ANOVA revealed that temporal demand scores during trials with unexpected robot stopping were significant (F(1, 24) = 12.462, p = .001, η_p ² = .293). Further, there was a significant interaction between workload and unexpected robot stopping (F(2, 48) = 15.432, p < .01, η_p ² = .340) such that participants reported increasing temporal demand scores with the increasing workload during conditions with unexpected robot stopping than the normal conditions. There was a significant main effect of workload on participants' performance scores (F(2, 48) = 1016.185, p < .01, η_p ² = .971). Also, the ANOVA revealed that performance scores during trials with unexpected robot stopping were significant (F(1, 24) = 925.325, p < .01, η_p ² = .969). Further, there was a significant interaction between workload and unexpected robot stopping (F(2, 48) = 100.151, p < .01, η_p ² = .769) such that participants reported higher performance scores with the increasing workload during conditions with unexpected robot stopping than the normal conditions.

There was a significant main effect of workload on participants' effort scores (F(2, 48) = 1091.580, p < .01, η_p ² = .973). Also, the ANOVA revealed that effort scores during trials with unexpected robot stopping were significant (F(1, 24) = 202.924, p = .001, η_p ² = .871). However, there was no significant interaction between workload and unexpected robot stopping as the participants reported similar increasing effort scores with the increasing workload during unexpected robot stopping and normal conditions. There was a significant main effect of workload on participants' frustration scores (F(2, 48) = 1368.842, p < .01, η_p ² = .979). Also, the ANOVA revealed that frustration scores during trials with unexpected robot stopping were significant (F(1, 24) = 334.303, p < .01, η_p ² = .918). Also, there was no significant interaction between workload and unexpected robot stopping as the participants reported similarly increasing frustration scores with the increasing workload during unexpected robot stopping and normal conditions.

The target identification accuracy deteriorated significantly with increasing workload during the unexpected robot stopping conditions, [F(2, 48) = 228.770, p < .001, η_p ² = .916], as shown in figure 4(A). Target identification accuracy significantly decreased for low and medium (p = .004), low and high (p < .01) and medium and high (p < .01) conditions. The time to reach the obstacle increased with the increasing workload in the unexpected robot stopping conditions, [F(2, 48) = 113.896, p < .01, η_p ² = .792], as shown in figure 4(B). The increase was significant for low and medium (p < .01), low and high (p < .01) and medium and high conditions (p < .01).

Zoom In Zoom Out Reset image size

The applied human force decreased significantly with the increasing workload during the unexpected robot stopping conditions, [F(2, 48) = 130.340, p < .01, η_p ² = .828], as shown in figure 4(C). Applied human force decreased significantly for low and medium (p < .01), low and high (p < .01), and medium and high conditions (p < .01). The end-effector velocity decreased with the increasing workload during the unexpected robot stopping conditions, [F(2, 48) = 86.647, p < .01, η_p ² = .762], with significant decreases for low and medium (p < .01), low and high (p < .01), and medium and high (p < .01), as shown in figure 3(D).

The frontal, central and parietal clusters were identified based on the dipole locations and the cluster centroid [50]. The Talairach coordinate of the frontal cluster centroid was at (4, 38, 39) and included 19 components from the 19 participants. The Talairach coordinate of the central cluster centroid was at (3, −5, 51) and had 18 components from 18 participants. The Talairach coordinate of the parietal cluster centroid was at (11, −49, 45), including 24 ICs from 24 participants.

The frontal cluster's ICs' PSD for the different workload conditions during blasting is shown in figure 5(A). There was significant main effect of workload on frontal theta PSD (F(2, 38) = 37.894, p < .01, η_p ² = .643) during unexpected robot stopping condition. Frontal theta PSD increased significantly in the low-medium (p < .01), medium-high (p = .003) and low-high (p < .01) conditions shortly after unexpected robot stopping. Further, the frontal theta power was significantly higher after reaching the obstacle as compared to reaching the target for the low (p < .01), medium (p = .029) and high (p < .01) workload conditions. However, there were no significant differences in the frontal cluster's delta, alpha, beta or gamma PSD.

Zoom In Zoom Out Reset image size

The central cluster's ICs' spectral power for different workload conditions soon after reaching the obstacle in the unexpected robot stopping condition and after reaching the target in the normal conditions is shown in figure 5(C). There was a significant main effect of workload on central alpha PSD (F(2, 36) = 15.274, p < .01, η_p ² = .540) during unexpected robot stopping condition. Central alpha PSD increased significantly in the medium-high (p = .038) and low-high (p < .01) conditions shortly after the unexpected robot stopping. Further, the central alpha power varied significantly only for the low workload condition (p < .01) upon reaching the obstacle as compared to reaching the target. The central beta PSD varied significantly after reaching the obstacle across the three workload levels (F(2, 36) = 18.453, p < .01, η_p ² = .552). Central beta PSD increased significantly in the medium-high (p = .002) and low-high (p = .001) conditions shortly after the unexpected robot stopping. Further, the central beta power increased significantly for the low (p < .01) and medium (p < .01) workload conditions upon reaching the obstacle as compared to reaching the target. No significant changes were noted in the other frequency bands across the workload conditions.

The parietal cluster's ICs' spectral power for the three workload levels upon reaching the target in the unexpected robot stopping condition or reaching the obstacle in the normal condition is illustrated in figure 5(B). There was a significant main effect of workload on parietal alpha PSD (F(2, 48) = 59.458, p < .01, η_p ² = .730) during unexpected robot stopping condition. Parietal alpha PSD decreased significantly in the low-medium (p < .01), medium-high (p = .003) and low-high (p < .01) conditions shortly after the unexpected robot stopping. Further, the parietal alpha power varied significantly only for the low workload condition (p < .01) upon reaching the obstacle as compared to reaching the target. No significant changes were noted in delta, theta, beta or gamma PSD at the parietal cluster.

An independent samples t-test demonstrated a significant difference in PEN value at the Fz channel for low (t (23) = 5.216, p = .002), medium (t (23) = 3.416, p = .02) and high (t (23) = 2.186, p = .048) workload conditions. The Pe values at the Fz channel were also found to be significantly different at the low (t (23) = 5.846, p = .001), medium (t (23) = 4.312, p = .008) and high (t (23) = 1.078, p = .044) workload conditions. Therefore, the error-related potential at the Fz electrode showed significant changes in the amplitudes of the PEN and Pe, as shown in figure 6. Further, the ANOVA results showed a significant increase in the PEN with increasing workload (F(2, 48) = 55.276, p < .01, η_p ² = .648). PEN increased significantly for low-medium (p < .01), medium-high (p < .01) and low-high (p < .01) workload conditions, as shown in figures 7(A) and (B). The error positivity (Pe) also varied significantly across the workload levels (F(2, 48) = 415.107, p < .01, η_p ² = .933). Pe decreased significantly for low-medium (p < .01), medium-high (p < .01) and low-high (p < .01) workload conditions, as shown in figures 7(A) and (C). Further, the normalized applied force was negatively correlated with the PEN, r(71) = −.644, p < .01 and positively correlated with Pe, r(71) = .785, p < .01.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

We employed deep learning algorithms to predict PEN and Pe amplitude from the EEG data prior to reaching the obstacle. Figure 7 shows the mean squared error (mse) and mean absolute error (mae) values for the prediction of PEN and the mse and mae values for the prediction of Pe from the 2 s EEG data prior to reaching the obstacle.

As shown in figures 7(C) and (D), EEGNet can predict the amplitudes of PEN and Pe with great resolution based on the EEG data prior to the erroneous robot behavior at a latency of 0.0008 s. Therefore, relying on a two-second window prior to the error, which holds crucial indicators of the workload experienced by the user, we can reliably predict the PEN and Pe amplitude and thereby, the error awareness of the user.