Next Article in Journal
Temporal Data Correlation Providing Enhanced Dynamic Crypto-Ransomware Pre-Encryption Boundary Delineation
Next Article in Special Issue
Coupling Analysis of Compound Continuum Robots for Surgery: Another Line of Thought
Previous Article in Journal
Q-Learning-Based Pending Zone Adjustment for Proximity Classification
Previous Article in Special Issue
Path Planning for Obstacle Avoidance of Robot Arm Based on Improved Potential Field Method
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sensors
Volume 23
Issue 9
10.3390/s23094354
sensors-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Physiological Metrics of Surgical Difficulty and Multi-Task Requirement during Robotic Surgery Skills

by
Chiho Lim
1,
Juan Antonio Barragan
1,
Jason Michael Farrow
2,
Juan P. Wachs
1,
Chandru P. Sundaram
2 and
Denny Yu
1,*
1
School of Industrial Engineering, Purdue University, West Lafayette, IN 47907, USA
2
School of Medicine, Indiana University, Indianapolis, IN 46202, USA
*
Author to whom correspondence should be addressed.
Sensors 2023, 23(9), 4354; https://doi.org/10.3390/s23094354
Submission received: 21 March 2023 / Revised: 19 April 2023 / Accepted: 20 April 2023 / Published: 28 April 2023
(This article belongs to the Special Issue Advanced Technologies in Medical and Surgical Robotics)
Browse Figures
Versions Notes

Abstract

:
Previous studies in robotic-assisted surgery (RAS) have studied cognitive workload by modulating surgical task difficulty, and many of these studies have relied on self-reported workload measurements. However, contributors to and their effects on cognitive workload are complex and may not be sufficiently summarized by changes in task difficulty alone. This study aims to understand how multi-task requirement contributes to the prediction of cognitive load in RAS under different task difficulties. Multimodal physiological signals (EEG, eye-tracking, HRV) were collected as university students performed simulated RAS tasks consisting of two types of surgical task difficulty under three different multi-task requirement levels. EEG spectral analysis was sensitive enough to distinguish the degree of cognitive workload under both surgical conditions (surgical task difficulty/multi-task requirement). In addition, eye-tracking measurements showed differences under both conditions, but significant differences of HRV were observed in only multi-task requirement conditions. Multimodal-based neural network models have achieved up to 79% accuracy for both surgical conditions.

1. Introduction

Minimally invasive surgery (MIS) has several important benefits; these include smaller incisions, better perioperative pain control, reduced scar burden, and less blood loss [1,2,3]. While the purported benefits to the patient are well-established, the physical challenges encountered by the surgeon are less well defined. MIS approaches utilizing conventional laparoscopy have induced physical pain, fatigue, and high cognitive workload for surgeons secondary to lack of depth perception and restricted movement of laparoscopic tools [4,5,6,7]. Robotic-assisted surgery (RAS) was developed to ameliorate the aforementioned challenges by providing an ergonomic interface while maintaining the patient-level benefits of laparoscopy. In contrast to conventional laparoscopy, during RAS, surgeons perform key elements of the procedure remotely within a console equipped with two master controllers. The visual magnification to support the added dexterity afforded by the robotic platform allows for complex procedures to be safely completed while overcoming some of the known limitations of conventional laparoscopy [4]. However, technical advances can result in unintended consequences to both the patient and the surgeon. Specifically, the added cognitive workload experienced by surgeons during RAS due to remote positioning from the patient, necessitating optimal communication with the bedside assistant, the coordinated use of multiple instruments simultaneously on a limited visual field, and the lack of any tactile feedback can generate significant challenges that may ultimately affect patient safety.
This added workload can become excessive when the various tasks exceed an individual’s perceived resources to cope [8,9]. Information overload has also been correlated with poorer performance due to limitations on human information processing ability [10]. These findings have been demonstrated in similarly high-stakes professions such as aviation and the military [11,12]. Surgeons continuously face demanding tasks that are inherently multifactorial and are necessarily time-sensitive. The abundance of situational data input could produce deleterious cognitive workloads that may result in performance errors [13,14].
Prior efforts at understanding this phenomenon were built upon self-reported measurements (Multiple Resources Questionnaire/NASA Task Load Index), which demonstrated a decrease in cognitive workload during RAS when compared to conventional laparoscopy [15,16,17,18,19]. However, these self-reported workload measurements are limited by significant biases and may be less reliable than objective measures [20,21]. To tackle these limitations, sensor-based measurements are being explored, and most recently, some studies have employed electroencephalography (EEG) sensors to assess cognitive states [22,23]. For example, better RAS performance in lysis of adhesions surgery was achieved when surgeons utilized more mental resources measured by EEG [23]. Multimodal-sensing approaches have also been proposed for the detection of different cognitive states between pre-and post-RAS training: EEG and eye-tracking [24] and EEG, heart rate variability (HRV), electromyography, and electrodermal activity [25]. Notably, multimodal sensing outperformed individual sensors for predicting cognitive load.
These previous RAS-related studies have focused on the surgical task difficulty to control participants’ cognitive states [15,16,17,18,19,24,25]. Specifically, as primary task difficulty increases, so do the processing resources required, ultimately exceeding the capacity available and resulting in deteriorating performance [26]. However, many errors, in particular skill-based errors (i.e., ‘slips’ and ‘lapses’), occur when there are distractions and interruptions. In addition, the cognitive workload can vary depending on the number of concurrent tasks to be processed [27,28]. In the operating room, surgeons are continuously and inevitably facing situations where multiple clinical inputs and outputs occur concurrently. These divided attention tasks in dual-task interference (or multi-task requirement) are difficult to isolate and understand based on previous studies that only defined cognitive load by task difficulty. Thus, the cognitive workload caused by processing simultaneous surgical tasks needs to be studied to examine how the multi-task requirements influence physiological responses during RAS under different levels of surgical task difficulties.
To better understand how physiological responses represent the cognitive workload encountered in various RAS conditions, we aimed to examine the synchronous effects of two factors: primary task (i.e., surgical task difficulty) and multi-task requirement. We hypothesized that physiological measurements could discriminate the degree of cognitive workloads from both study factors, but that the physiological responses induced from each study factor are different. Our secondary objective was to determine the accuracy of machine learning models for predicting the study factors using the physiological features that can be extracted from multimodal physiological sensors.

2. Materials and Methods

This research has been reviewed and approved by University’s Institutional Review Board (IRB-1906022354).

2.1. Participants

A total of 10 engineering university students (8 males/2 females) with no prior experience in surgical-related tasks were recruited for our multi-session repeated measures study. Two participants’ heart rate variability (HRV) data were removed due to disconnection of the HRV sensor during the four sessions. Thus, 10 participants’ data were used to analyze EEG and eye-tracking, and 8 participants’ data were used for the HRV analysis and development of classification models. All participants were right-hand dominant with a mean age of 25 ± 1.2 years. The participants provided written consent and were paid for their participation.

2.2. Experimental Design

To elicit different cognitive states in the users, we designed an experiment with two factors: primary task and multi-task requirement. In the primary task factor, we examined the effects of tasks requiring different degrees of cognitive processing resources; the multi-task requirement factor allowed us to study the effects of distractors and spare working memory as participants performed secondary tasks alongside primary tasks during simulated RAS. We adopted the subsidiary task paradigm for the multi-task requirement where participants were required to maintain primary task performance at the expense of the secondary task [26]. Participants carried out four-session experiments over two days. Each session consisted of a combination of two primary tasks (peg transfer/suturing) and multi-task requirement conditions (single task condition/n-back/target game) in random order. Participants performed six tasks in each session (Table 1). Thus, the participants were required to have four repeated measures for each of the six tasks, resulting in a total of 24 tasks for each participant.

2.2.1. Primary Task

The primary task contained two levels. Each level was a different task with different workload requirements. Peg transfer and suturing tasks were adapted from the Fundamentals of Laparoscopic Surgery (FLS) training program, developed by the Society of American Gastrointestinal and Endoscopic Surgeons and launched in 2004 [29]. These tasks had different levels of hand-eye coordination and bimanual dexterity as described below:
  • The peg transfer task required the participants to translocate six objects to the peg. Participants started to grasp the object with the non-dominant hand from the left side of the pegboard and transferred it to the dominant hand to place on the right side of the board (if the left hand is a dominant hand, the task starts from the right side of pegboard). Once six objects were entirely transferred, the objects on the right side were transferred to the left side of the board, starting with the dominant hand. The participants were required to repeat the peg transfer task for up to 3 min.
  • For the suturing task, a cloth material pad had a slit to guide the suturing path. The suturing task required the participants to put the needle precisely in one side of the slit and put it through on the other side of the slit using the robot’s gripper, which required them to constantly adapt the wrist position of the end effector. Along with this, the interval of stitching needed to be consistent. The participants were required to repeat the suturing task for up to 3 min.

2.2.2. Multi-Task Requirement

To assess spare working memory capacity, users were instructed to perform secondary tasks. Secondary tasks are based on the Multiple Resource Theory (MRT) [30,31] and are widely used to assess cognitive workload. From the MRT, a secondary task uses the remaining mental capacity after a primary task takes a certain amount of mental capacity. In most cases, the successful performance relies on how well the operator retrieves information from memory. In particular, the working memory has been closely related to the operator’s performance in a variety of situations [26]. In this study, auditory and visual working memory tasks that are frequently engaged with primary surgical procedures were adopted to measure sensory-related multi-task effects on physiological responses. As the user performed the primary task, three multi-task requirement conditions were studied: single task condition (only peg transfer or suturing), n-back multi-task, or a target game multi-task coded in the da Vinci Research Kit (dVRK, Intuitive Surgical Inc., Sunnyvale, CA, USA).
The n-back task has been used to investigate working memory in dual-task experiments [32]. This study adopted the auditory 2-back task [33] as an auditory memory task instead of the traditional visual n-back. The visual n-back can block or interfere with the view of the main surgical task. Pre-recorded sets of single-digit numbers (0–9) were randomly presented verbally at 26.6 wpm. Following established n-back protocol, participants recited numbers heard two positions previously throughout the task.
The target game as a visual memory task was designed by adjusting the Sternberg memory search task (item-recognition task) [34] to dVRK. A timer (mm:ss) with one-digit target numbers was displayed in the dVRK stereo viewer. Figure 1 shows the example of a stereo viewer including target game information. The users were instructed to tap the console foot pedal each time the timer’s second digit number corresponded to the target numbers. The random target numbers changed every 20 s. The target game information was displayed at the top of the stereo display to avoid blocking the view of the main task. Based on the Sternberg memory search task [34], the target number, second digit number of the timer, and pedal tapping were considered positive stimulus set, test stimulus, and positive response, respectively.

2.3. Data Analysis

2.3.1. Electroencephalogram (EEG)

A wireless electroencephalogram (EEG), g.Nautilus (Guger Technologies OG, Graz, Austria), was employed to collect EEG signals. Signals were sampled at 250 Hz on 32 channels positioned based on a 10–20 system. This device used an ear-clip electrode as a reference. The collected EEG data were preprocessed using EEGlab in MATLAB [35]. The EEG data were filtered offline using a basic finite impulse response filter of 0.1–30 Hz and the data were re-referenced to the average of the signal [36]. Cleanline plugin was used for removing sinusoidal noise, specifically, power line interference in EEG recordings [37]. To remove artifacts generated by eye movements, eye blinks, and temporal muscle activities, independent component analysis (ICA) [35] and semi-automated rejection plugin (ADJUST) were used [38].
After artifact removal, EEG signals were processed using spectral analysis to quantify band power, and EEG indices were calculated to assess cognitive workload [39,40,41,42]. Power spectral density analysis was computed using Welch’s method with a Hanning window of 0.2 s and 50% overlap for the 3-min task duration data of each channel to calculate absolute band power (theta (4–8 Hz)/alpha (8–12 Hz)/beta (12–30 Hz)). The formulas for calculating the engagement index and theta-alpha ratio (TAR) were calculated using Equations (1) and (2), respectively. In this study, we used EEG data from P3, P4, PZ, and CZ channels to calculate the engagement index as proposed by Prinzel et al. [43]. The theta band power of the FZ channel and alpha band power of the PZ channel were used to calculate TAR as proposed by Holm et al. [44].
Engagement Index = (beta band power)/(theta band power + alpha band power)
TAR = (theta band power (FZ))/(alpha band power (PZ))

2.3.2. Eye-Tracking

Wearable eye-tracker, Tobii Pro Glasses 2 (Tobii Technology AB, Danderyd, Sweden), was used to monitor participants’ gaze data during the experiments. Data were collected at 50 Hz. The Tobii Pro Lab Software (Tobii Technology AB, Danderyd, Sweden) was used to extract the fixation and saccade metrics. In addition, Saccade-Fixation Ratio (SF Ratio) [45] and gaze entropy were extracted from the eye-tracking data. SF ratio was calculated using Equation (3) [46], and gaze entropy was calculated following Equation (4) [47]:
SF   ratio   =   ( Total   Saccade   Time ) / ( Total   Fixations   Time )
H ( X )   =   − ∑ p ( x ,   y ) log 2 p ( x ,   y )
where p(x, y) is the probability of the gaze falling in the (x, y).

2.3.3. Heart Rate Variability (HRV)

This study used optical pulse ear clip (PPG) of the Shimmer3 GSR + Unit (Shimmer, Dublin, Ireland) to acquire HRV data. The data were collected at 128 Hz. Time and frequency domain metrics were extracted from the HRV data and used as indicators of workload. The Kubios HRV Premium software (Kubios Oy, Kuopio, Finland) was used to extract the time domain metrics (mean RR interval (meanRR), standard deviation of RR (stdRR), the square root of the mean squared differences between adjacent normal RR interval (RMSSD), the percentage of adjacent NN intervals that differ from each other by more than 50 ms (pNN50)) and frequency domain metrics (low frequency (LF, 0.04–0.15 Hz)/high frequency (HF, 0.15–0.4 Hz) ratio).

2.4. Procedure

Prior to the experiments, each participant was trained on the dVRK to become familiar with the primary tasks (peg transfer/suturing) and the multi-task requirement tasks (n-back/target game) for 3 h over two days. After the training sessions, we conducted a basic suturing task within a 3-min limit to screen the participants who did not have a sufficient skill level. Three participants were excluded from this task. In this study, we utilized the first generation Da Vinci robot with the proprietary software and hardware provided by the dVRK development community [48]. After the training session, participants performed the four study sessions as outlined in the experimental design. Physiological data were collected throughout the sessions with the aforementioned EEG, eye-tracker, and HRV sensors. Each sensor was set up and calibrated based on the manufacturer’s guidelines.

2.5. Statistical Analysis

Statistical analyses were performed in R Studio (version 1.3.959, R-core 4.0.1). Prior to the analyses, the normality test of the data was conducted, and outliers were removed from datasets (EEG/eye-tracking/HRV) using Cook’s distance measure [49]. The extracted features were examined using a linear mixed model [50], with the primary task (peg transfer/ suturing) and multi-task requirement (single task condition/n-back/target game) as fixed variables and subjects as random effects. Two-way interactions were explored, and Tukey’s multiple comparison test was further performed to determine differences between factor level combinations at the significance level of 0.05.

2.6. Artificial Neural Network Classification Model

Classification models were developed for a supervised binary class problem. The classification model consists of two hidden layers and one output layer. A log-sigmoid transfer function was used in hidden layers, and a softmax function was used for an output layer [51].
The extracted features from physiological sensors were used as input to develop the artificial neural network. To reduce redundancy and improve the accuracy and generalization capability of the classification model, the least absolute shrinkage and selection operator (Lasso) was used to identify the key feature subset of input variables [52].
In this study, four classification models were developed based on the study factors: (1) primary task classification model (peg transfer vs. suturing), (2) multi-task requirement classification models (single task condition vs. n-back multi-task, single task condition vs. target game multi-task, n-back multi-task vs. target game multi-task). For evaluation of model performance, 10-fold cross validation was implemented.

3. Results

Physiological data were collected from a total of 10 participants, and each participant completed four sessions (60 min/session) over two days and performed 24 tasks in total. One and three data samples were removed based on the Cook’s distance measure for the EEG and eye-tracking datasets, respectively. The results below are organized by physiological modality and describe how study factors (primary task and multi-task requirement) affected the observed physiological responses.

3.1. Electroencephalogram (EEG)

3.1.1. Effect of the Primary Task

Figure 2a shows the EEG topographic plots of the averages of the band power differences between the suturing and peg transfer tasks across all participants, with the band power of peg transfer subtracted from the band power of suturing. Theta band power was found to be significantly higher on five channels during suturing tasks compared to peg transfer tasks, F(1, 239) = [3.88–14.20], p < 0.05. A post hoc Tukey test showed that theta band power during the suturing tasks was [0.31–0.59 μV2] higher than theta band power during the peg transfer tasks.
The suturing tasks showed significant increases and decreases of alpha band power on 4 channels (F(1, 239) = [4.03–8.35], p < 0.05) compared to peg transfer tasks. From the Tukey test, alpha power during the suturing tasks was 0.16, 0.56, and 0.36 μV2 higher than during the peg transfer task on the CP1, T7, and T8 channels, respectively, but 0.38 μV2 lower on the FP2 channel.
For EEG index measures, significantly higher TAR were observed in the suturing tasks, with F(1, 239) = 15.57, p < 0.05. From the Tukey test, 0.18 higher TAR was observed during the suturing tasks than the peg transfer tasks (Figure 2b).
No significant differences were observed for the beta band power and the engagement index between primary tasks.

3.1.2. Effect of the Multi-Task Requirement

Figure 3a shows the EEG topographic plots of the averages of the band power differences between multi-task requirement and single task conditions across all participants, with the band power of the single task conditions subtracted from the band power of the multi-task requirement conditions. For the effect of the multi-task requirement, higher theta band power values were observed on 13 channels in the multi-task requirement than in the single task condition (F(2, 239) = [3.19–20.54], p < 0.05). Specifically, theta band power on P7 was 0.37 μV2 higher for the n-back multi-task than for a single task condition. Theta band power during target game multi-task was [0.31–1.92 μV2] higher on 12 channels than theta band power during a single task condition. The multi-task requirement factor had significant interactions with the primary task on the FP1 channel, F(2, 239) = 3.30, p < 0.05. The theta power on FP1 was 2.05–2.41 μV2 higher when participants performed the peg transfer task simultaneously with the target game than when performing the peg transfer task without the target game. The theta band power of the peg transfer with the target game was 2.30 μV2 higher than the theta band power during the suturing task only (Figure 3b).
The multi-task requirement both increased and decreased alpha band power on 9 channels compared to the single task condition (F(2, 239) = [3.09–10.63], p < 0.05). The Tukey test showed that when n-back was added to the primary task, [0.17–1.00 μV2] lower alpha band power was observed compared to alpha band power during a single task condition on 3 channels. With the target game, [0.18–0.86 μV2] higher alpha power was observed than during a single task condition on 6 channels.
For beta band power, multi-task requirement (F(2, 239) = [3.21–9.98], p < 0.05) had [0.28–3.67 μV2] and [0.19–0.57 μV2] higher beta band power for n-back (5 channels) and target game multi-task (7 channels), respectively, when compared to the single task condition.
Among EEG index measures, only TAR differed between task conditions, with F(2, 239) = 5.13, p < 0.05. From the Tukey test, TAR was 0.16 lower during the target game multi-task than the single task condition. In addition, the n-back multi-task had 0.16 higher TAR than the target game multi-task. No significant differences were observed for the engagement index.

3.2. Eye-Tracking

3.2.1. Effect of the Primary Task

Participants had 0.02 lower SF ratio during the suturing tasks than peg transfer tasks (F(2, 237) = 7.84, p < 0.05) (Figure 4). Gaze entropy did not differ between primary tasks.

3.2.2. Effect of the Multi-Task Requirement

Gaze entropy (F(2, 237) = 4.90, p < 0.05) and SF ratio (F(2, 237) = 5.92, p < 0.05) differed statistically among three conditions (single task condition/n-back/target game) (Figure 5). A post hoc Tukey test showed that the gaze entropy was 0.30 higher, and the SF ratio was 0.02 higher during the target game multi-task than during the single task condition (Figure 5). Gaze entropy and SF ratio during the target game were 0.33 and 0.02 higher than during the n-back multi-task, respectively (Figure 5). Eye-tracking indicators did not differ between the single task condition and n-back multi-task.

3.3. Heart Rate Variability (HRV)

3.3.1. Effect of the Primary Task

Interaction between primary and multi-tasks was significant for HRV (F(2, 192) = 3.76, p < 0.05). Suturing tasks had a 0.96 lower LF/HF ratio than peg transfer tasks for the single-task conditions but did not differ for the multi-task conditions (Figure 6). Other HRV indicators were not sensitive to differences between primary tasks.

3.3.2. Effect of the Multi-Task Requirement

Significant differences during n-back multi-task in meanRR (F(2, 192) = 4.15, p < 0.05), stdRR (F(2, 192) = 6.56, p < 0.05), and RMSSD (F(2, 192) = 4.96, p < 0.05) were observed (Figure 7) compared to the single task condition. During the n-back multi-task, meanRR was 22.44 ms lower than meanRR during the single task condition. In addition, 5.45 ms higher stdRR and 6.36 ms higher RMSSD were observed during n-back multi-task than during a single task condition. Other HRV indicators were not statistically significant in terms of a multi-task requirement effect.

3.4. Artificial Neural Network Classification Model

Eight participants’ data were used to develop classification models, and if one of the physiological data in the specific task was removed based on the Cook’s distance measure, other modality data in the task were not used in the development of the classification model. Accuracy of classification models in distinguishing differences between study factors (primary task effect and multi-task requirement effect) using physiological metrics ranged from 65 to 79%.
The Lasso feature selection algorithm identified the best subset of input variables from three physiological sensors for each classification model (Table 2). The model performances measured by accuracy and F1-score indicated that approximately 65% accuracy was achieved for the primary task effect, 79% for the single task condition vs. n-back multi-task model, 76% for the single task condition vs. target game multi-task model, and 72% for the n-back multi-task vs. target game multi-task model (Table 2).

4. Discussion

Cognitive workload varies dynamically and can come from a wide variety of work demands during RAS. Physiological measurements sensitive to these demands are needed to identify the influence of various cognitive workloads encountered during RAS. However, existing studies predicting cognitive workload during RAS have primarily focused on modulating cognitive load using task difficulty. This limits (1) the potential applicability of previously identified behaviors to RAS (where multiple sources contribute to users’ cognitive demands) and (2) our understanding of physiological response behaviors and whether they are sensitive to changes to these demands. In this study, we measured patterns of physiological responses due to changes and interactions in workload demands from the primary task and multi-task requirement study factors. Finally, multimodal physiological signal-based classification models were developed to distinguish the primary tasks and multi-task requirements. The following discussion of the experimental results is organized by physiological modalities.

4.1. Electroencephalogram (EEG)

4.1.1. Theta Band Power

Previous work found evidence that theta band power increased with more demanding tasks [53,54]. In our work, the suturing task was designed to require a higher degree of processing resources than peg transfer for the primary task factor. For the multi-task requirement factor, n-back and target game multi-tasks were designed to disperse participants’ attentional resources to a second task and increase the memory load compared to the single task condition.
Theta band power distinguished between the two study factors (primary task and multi-task requirement). Previous studies have shown that the theta band power increases with high cognitive resource demand and high task difficulty [55]. This is consistent with our findings showing higher theta band power during the suturing task than during the peg transfer task. In this study, the suturing task required higher levels of hand-eye coordination and bimanual dexterity than the peg transfer task. For the primary task effect, the increased theta band power was primarily observed over the frontal area during suturing tasks with comparison to theta band power during peg transfer tasks. This result was consistent with the claim that the frontal cortex is linked to cognitive workload for the theta band power [56].
In addition, previous studies found that theta band power increased as the number of concurrent tasks needing to be processed increased [57,58]. For example, Scharinger et al. investigated the effects of the n-back task for theta band power in three conditions (digit value, position, and form). They observed an increase in theta band power as working memory load was increased by the n-back task [59]. This was consistent with our findings with the n-back task. The n-back multi-task increased theta band power over the parietal area. In addition, a significant change in theta band power was observed for the target game multi-task over the frontal and parietal areas. These regional effects can be potentially explained with the existing literature. Working memory has been linked to the parietal area of the brain [60]. For example, in a previous study, with an increased number of visual items (up to 3–4) to memorize [61], the parietal area had more activation. In addition, higher theta band power in the fronto-parietal area reflected lower working memory capacity [62,63,64]. In our study, participants were required to memorize numerical items during the n-back and the target game multi-task requirements. This required participants to utilize their working memory during the experiments. Thus, our finding of increased theta band power over the parietal area during n-back and target game multi-tasks is in line with other findings from the literature.

4.1.2. Alpha Band Power

For the primary task, alpha band power increased over parietal and temporal areas but decreased over the frontal area during the more demanding task (suturing). Previous studies have shown that alpha desynchronization (e.g., lower alpha power during the high demanding task) was observed over the task-relevant brain area, whereas synchronization was observed over task-irrelevant brain areas [65,66,67,68]. Our study results were consistent with these studies as we observed that alpha band power decreased over the motor cortex (FP2) (desynchronization) and increased over task-irrelevant areas (CP1, T7, T8) (synchronization) during suturing tasks; these require more demanding motor controls to constantly align the participants’ wrist position with the robot’s end-effector and require fine dexterity and two-handed coordination to stitch a suture.
For the multi-task requirement, decreased alpha power over the temporal area (T7 and T8) was generally observed for the n-back multi-task compared to the single task condition. From previous studies, alpha band power over parietal areas decreased with increasing n-back levels [69,70]. In this study, desynchronization of alpha power over the temporal area could be caused by auditory stimulus from auditory n-back tasks since the temporal lobe is linked to the auditory cortex. Some studies showed the link between alpha band power over temporal areas and auditory processing [71,72]. For example, Lehtelä et al. found that the presentation of noise induced the suppression of alpha band power over temporal areas. For these reasons, the auditory n-back multi-task could have induced the suppression of the alpha power band in this study. However, studies comparing auditory with other forms of n-back tasks are needed to verify these explanations.
In contrast, increased alpha power over frontal and parietal areas was observed for the target game compared to the single task condition. The target game required working memory to memorize target numbers. Synchronization of alpha was observed over task-irrelevant brain regions, interfering with cognitive processes [66,68]. Some studies have linked this synchronization to an inhibition (interfering processes) of distractors [73,74,75,76]. The studies showed that alpha band power increased over cortical areas responsible for processing distractors. In more detail, the visual target number in the dVRK stereo viewer during target game multi-tasking might be used to visually present cues based on an instructional cueing paradigm [73]. When cues were triggered, participants were instructed to press the pedal while primary tasks (peg transfer/suturing) were considered distractors. In this case, the responsible areas for processing the distractors (primary tasks) were frontal areas (motor cortex). Thus, an increased alpha band power over frontal areas (AF3, FP1, FP2, and FC5) during the target game multi-task could reflect the inhibition process of distractors (primary tasks).

4.1.3. Beta Band Power

In this study, beta band power was not sensitive to differences in the primary tasks. However, beta band power differed between multi-task requirements. When participants were required to do concurrent tasks with the primary task, an increase in beta power was seen in response to the increased workload. Beta band power has been associated with short term memory and visual or auditory stimuli [77,78,79]. Results from this study showed that beta band power during both multi-task requirement conditions increased in comparison to the single task condition. Consistent results were observed in previous studies when additional working memory use was forced on the participants [80,81]. For example, Chen and Huang found increased beta band power with increased working memory load in the visual n-back experiment [80]. However, caution should be taken in using beta band power as a workload indicator because varying beta band powers were observed with an increase in cognitive tasks, and the role of beta band power remains unclear [79].

4.2. Eye-Tracking

Our results showed that the SF ratio decreased during the more demanding primary task (suturing). SF ratio was used to compare the time spent searching (saccade) to the time spent processing (fixation) [45]. Based on our results, the participants spent relatively more time on processing the task and less time on search activities during suturing tasks compared to peg transfer tasks. This is consistent with the results from previous studies where saccade duration was shorter during the tasks which induced higher cognitive load [82,83], and fixation duration became longer as the task workload demand increased [84,85].
However, for the multi-task requirement task effect, a higher SF ratio was observed during the target game multi-task than the single task condition. One potential reason is that the target game forced the participant’s gaze to be dispersed from the primary task to the numerical target numbers on the stereo viewer located at the corner of the field of view. This could have led to a longer saccade duration and shorter fixation duration during the task.
On the other hand, gaze entropy during target game multi-task was higher than gaze entropy during a single task condition. Gaze entropy is a measure of disordered search processes which do not follow a systematic pattern, and it has been a valid task load index in various areas. Specifically, gaze pattern tends to be more random during more complex workload tasks [86,87]. Other surgical-related studies have also shown that higher gaze entropy is related to higher surgical task load [88,89]. However, a key limitation to using eye-tracking measures in the cognitive workload estimation is that eye behavior patterns depend on the assigned tasks [90,91].

4.3. Heart Rate Variability (HRV)

Although HRV metrics were not sensitive to the different primary tasks, differences in these metrics were observed for the multi-task requirement effect (n-back task). During the n-back multi-task, a decrease of meanRR and increase of stdRR and RMSSD were observed compared to single task condition. In previous studies for multi-task environments, heart rate, which has a reciprocal relation with the RR interval, increased with task difficulty [92], and HRV decreased during more demanding tasks [85,93]. For example, Veltman et al. observed that adding secondary tasks caused decreased HRV in the flight environment. However, mental workload tasks involving speech tasks should be interpreted with caution because respiratory changes from speech might alter HRV patterns [94,95,96]. In this study, during the n-back multi-task, the participants were required to speak the memorized numbers aloud. Such speech-related respiratory patterns might influence the HRV patterns. Thus, HRV measures used in surgical environments (e.g., RAS) requiring frequent communications between surgical team members need to consider these speech-related respiratory patterns.

4.4. Artificial Neural Network Classification Model

The results of the study show that the classification models had a better performance for the multi-task requirement condition than the primary task condition. One possible explanation for this could be that the addition of the n-back task or target game task as a secondary task in the multi-task condition may have increased the cognitive load and attentional demands on the participants, resulting in more pronounced differences in the physiological signals compared to the primary task alone. Another possibility is that the specific physiological metrics in the classification models were more sensitive to changes in multi-tasking demands rather than primary task factors. In our study, HRV-related metrics were not selected by the Lasso feature selection algorithm as optimal feature sets for the primary task factor, but were included in all levels of the multi-task requirement factors. This suggests that HRV-related metrics may play a more significant role in distinguishing between different levels of multi-tasking demands, potentially contributing to the higher accuracy achieved in the multi-task requirement models.

5. Limitations

Several limitations need to be acknowledged in the present study. First, this study application is focused on robotic surgery. However, the study population consisted of university students with no surgical experience. Even though they were trained before the experiments, they do not have the same level of surgical skills as surgeons. However, physiological response patterns to task demands may be consistent across individuals as previous research has shown. Thus, the patterns identified in this study may still provide insight on how physiological responses change with task demands in RAS. Nonetheless, a future study utilizing a surgeon cohort is needed to evaluate the generalizability of these physiological patterns to individuals that perform RAS. Future work is needed to design experimental paradigms with more granular load modulation for evaluating physiological responses during RAS.

6. Conclusions

High cognitive workload experienced in the operating room theater is multifactorial and is better modeled by generating a composite exposure relying on multi-task requirements. This study identified distinguishing physiological responses for two study factors (primary task/multi-task requirement). Multiple responses for EEG band power and channel locations were characterized according to the study factors. Concurrently, we identified gaze pattern distinctions between the two study factors. While the HRV metrics were significant during a multi-task requirement (n-back), they were not sensitive at distinguishing different types of surgical skills (primary task effect).
Our findings contribute further evidence of the consistencies and differences in physiological behaviors during RAS with existing neuroergonomics work. Understanding how users respond to task difficulty and multitasking workloads can enable the development of robotic surgery interfaces that are responsive to user cognitive states.

Author Contributions

Conceptualization, C.L. and D.Y.; methodology, C.L. and J.A.B.; software, C.L.; validation, D.Y., J.P.W. and C.P.S.; formal analysis, C.L.; investigation, C.L.; resources, J.P.W. and C.P.S.; data curation, C.L.; writing—original draft preparation, C.L.; writing—review and editing, D.Y., J.P.W., J.M.F. and C.P.S.; visualization, C.L.; supervision, D.Y.; project administration, D.Y., J.P.W. and C.P.S.; funding acquisition, J.P.W., D.Y. and C.P.S. All authors have read and agreed to the published version of the manuscript.

Funding

This research was supported, in part, by a research grant from Intuitive Surgical and the National Institutes of Health (grant number R21EB026177).

Institutional Review Board Statement

This research has been reviewed and approved by the University’s Institutional Review Board (IRB-1906022354).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The data that support the findings of this study are available on request from the corresponding author, D.Y. The data are not publicly available due to restrictions containing information that could compromise the privacy of research participants.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Sathianathen, N.J.; Kalapara, A.; Frydenberg, M.; Lawrentschuk, N.; Weight, C.J.; Parekh, D.; Konety, B.R. Robotic Assisted Radical Cystectomy vs. Open Radical Cystectomy: Systematic Review and Meta-Analysis. J. Urol. 2019, 201, 715–720. [Google Scholar] [CrossRef]
  2. Cao, L.; Yang, Z.; Qi, L.; Chen, M. Robot-assisted and laparoscopic vs open radical prostatectomy in clinically localized prostate cancer: Perioperative, functional, and oncological outcomes: A Systematic review and meta-analysis. Medicine 2019, 98, e15770. [Google Scholar] [CrossRef]
  3. Cirocchi, R.; Cesare Campanile, F.; Di Saverio, S.; Popivanov, G.; Carlini, L.; Pironi, D.; Tabola, R.; Vettoretto, N. Laparoscopic versus open colectomy for obstructing right colon cancer: A systematic review and meta-analysis. J. Visc. Surg. 2017, 154, 387–399. [Google Scholar] [CrossRef] [PubMed]
  4. Hazey, J.W.; Melvin, W.S. Robot-Assisted General Surgery. Semin. Laparosc. Surg. 2004, 11, 107–112. [Google Scholar] [CrossRef] [PubMed]
  5. Park, A.; Lee, G.; Seagull, F.J.; Meenaghan, N.; Dexter, D. Patients Benefit While Surgeons Suffer: An Impending Epidemic. J. Am. Coll. Surg. 2010, 210, 306–313. [Google Scholar] [CrossRef] [PubMed]
  6. Szeto, G.P.; Ho, P.; Ting, A.C.; Poon, J.T.; Cheng, S.W.; Tsang, R.C. Work-related musculoskeletal symptoms in surgeons. J. Occup. Rehabil. 2009, 19, 175–184. [Google Scholar] [CrossRef] [PubMed]
  7. Zhang, J.; Patel, V.L.; Johnson, T.R.; Shortliffe, E.H. A cognitive taxonomy of medical errors. J. Biomed. Inform. 2004, 37, 193–204. [Google Scholar] [CrossRef]
  8. Lazarus, R.S. Psychological Stress and the Coping Process; McGraw-Hill: New York, NY, USA, 1966. [Google Scholar]
  9. Matthews, G. Levels of transaction: A cognitive science framework for operator stress. In Stress, Workload, and Fatigue; Lawrence Erlbaum Associates: Mahwah, NJ, USA, 2001; pp. 5–33. [Google Scholar]
  10. Wickens, C.D. Multiple resources and mental workload. Hum. Factors 2008, 50, 449–455. [Google Scholar] [CrossRef]
  11. Sexton, J.B.; Helmreich, R.L. Analyzing cockpit communications: The links between language, performance, error, and workload. Hum. Perf. Extrem. Environ. 2000, 5, 63–68. [Google Scholar] [CrossRef]
  12. Wallenius, C.; Larsson, G.; Johansson, C.R. Military observers’ reactions and performance when facing danger. Mil. Psychol. 2004, 16, 211–229. [Google Scholar] [CrossRef]
  13. Bogner, M.S. Human Error in Medicine; CRC Press: Boca Raton, FL, USA, 2018. [Google Scholar]
  14. Byrne, A.J.; Oliver, M.; Bodger, O.; Barnett, W.A.; Williams, D.; Jones, H.; Murphy, A. Novel method of measuring the mental workload of anaesthetists during clinical practice. Br. J. Anaesth. 2010, 105, 767–771. [Google Scholar] [CrossRef]
  15. Lee, G.I.; Lee, M.R.; Clanton, T.; Sutton, E.; Park, A.E.; Marohn, M.R. Comparative assessment of physical and cognitive ergonomics associated with robotic and traditional laparoscopic surgeries. Surg. Endosc. 2014, 28, 456–465. [Google Scholar] [CrossRef]
  16. Klein, M.I.; Warm, J.S.; Riley, M.A.; Matthews, G.; Doarn, C.; Donovan, J.F.; Gaitonde, K. Mental workload and stress perceived by novice operators in the laparoscopic and robotic minimally invasive surgical interfaces. J. Endourol. 2012, 26, 1089–1094. [Google Scholar] [CrossRef]
  17. Mouraviev, V.; Klein, M.; Schommer, E.; Thiel, D.D.; Samavedi, S.; Kumar, A.; Leveillee, R.J.; Thomas, R.; Pow-Sang, J.M.; Su, L.M.; et al. Urology residents experience comparable workload profiles when performing live porcine nephrectomies and robotic surgery virtual reality training modules. J. Robot. Surg. 2016, 10, 49–56. [Google Scholar] [CrossRef]
  18. Sethi, A.S.; Peine, W.J.; Mohammadi, Y.; Sundaram, C.P. Validation of a novel virtual reality robotic simulator. J. Endourol. 2009, 23, 503–508. [Google Scholar] [CrossRef]
  19. Stefanidis, D.; Wang, F.; Korndorffer, J.R.; Dunne, J.B.; Scott, D.J. Robotic assistance improves intracorporeal suturing performance and safety in the operating room while decreasing operator workload. Surg. Endosc. Other Interv. Tech. 2010, 24, 377–382. [Google Scholar] [CrossRef]
  20. Byers, J.C. Traditional and raw task load index (TLX) correlations: Are paired comparisons necessary? Adv. Ind. Ergon. Saf. 1989, 1, 481–485. [Google Scholar]
  21. McKendrick, R.D.; Cherry, E. Proceedings of the Human Factors and Ergonomics Society Annual Meeting. In A Deeper Look at the NASA TLX and Where it Falls Short; SAGE Publications Sage: Los Angeles, CA, USA, 2018; pp. 44–48. [Google Scholar]
  22. Guru, K.A.; Esfahani, E.T.; Raza, S.J.; Bhat, R.; Wang, K.; Hammond, Y.; Wilding, G.; Peabody, J.O.; Chowriappa, A.J. Cognitive skills assessment during robot-assisted surgery: Separating the wheat from the chaff. BJU Int. 2015, 115, 166–174. [Google Scholar] [CrossRef]
  23. Guru, K.A.; Shafiei, S.B.; Khan, A.; Hussein, A.A.; Sharif, M.; Esfahani, E.T. Understanding Cognitive Performance During Robot-Assisted Surgery. Urology 2015, 86, 751–757. [Google Scholar] [CrossRef]
  24. Wu, C.; Cha, J.; Sulek, J.; Sundaram, C.P.; Wachs, J.; Proctor, R.W.; Yu, D. Sensor-based indicators of performance changes between sessions during robotic surgery training. Appl. Ergon. 2021, 90, 103251. [Google Scholar] [CrossRef]
  25. Zhou, T.; Cha, J.S.; Gonzalez, G.; Wachs, J.P.; Sundaram, C.P.; Yu, D. Multimodal Physiological Signals for Workload Prediction in Robot-assisted Surgery. Acm Trans. Hum. Robot. Interact. 2020, 9, 1–26. [Google Scholar] [CrossRef]
  26. Proctor, R.W. Human Factors in Simple and Complex Systems; CRC Press: Boca Raton, FL, USA, 2018. [Google Scholar]
  27. Schvaneveldt, R.W. Effects of complexity in simultaneous reaction time tasks. J. Exp. Psychol. 1969, 81, 289. [Google Scholar] [CrossRef]
  28. Yeh, Y.Y.; Wickens, C.D. Dissociation of Performance and Subjective Measures of Workload. Hum. Factors 1988, 30, 111–120. [Google Scholar] [CrossRef]
  29. Peters, J.H.; Fried, G.M.; Swanstrom, L.L.; Soper, N.J.; Sillin, L.F.; Schirmer, B.; Hoffman, K.; Committee, S.F. Development and validation of a comprehensive program of education and assessment of the basic fundamentals of laparoscopic surgery. Surgery 2004, 135, 21–27. [Google Scholar] [CrossRef] [PubMed]
  30. Wickens, C.D.; Gordon, S.E.; Liu, Y.; Lee, J. An Introduction to Human Factors Engineering; Pearson Prentice Hall: Upper Saddle River, NJ, USA, 2004; Volume 2. [Google Scholar]
  31. Wickens, C.D. Multiple resources and performance prediction. Theor. Issues Ergon. Sci. 2002, 3, 159–177. [Google Scholar] [CrossRef]
  32. Kirchner, W.K. Age differences in short-term retention of rapidly changing information. J. Exp. Psychol. 1958, 55, 352–358. [Google Scholar] [CrossRef]
  33. Mehler, B.; Reimer, B.; Dusek, J.A.J.C. MIT AgeLab Delayed Digit Recall Task (N-Back); Massachusetts Institute of Technology: Cambridge, MA, USA, 2011; Volume 17. [Google Scholar]
  34. Sternberg, S. Memory-scanning: Mental processes revealed by reaction-time experiments. Am. Sci. 1969, 57, 421–457. [Google Scholar]
  35. Delorme, A.; Makeig, S. EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J. Neurosci. Methods 2004, 134, 9–21. [Google Scholar] [CrossRef]
  36. Bigdely-Shamlo, N.; Mullen, T.; Kothe, C.; Su, K.M.; Robbins, K.A. The PREP pipeline: Standardized preprocessing for large-scale EEG analysis. Front. Neuroinform. 2015, 9, 16. [Google Scholar] [CrossRef]
  37. Mullen, T. NITRC: CleanLine: Tool/Resource Info. 2012. Available online: https://www.nitrc.org/projects/cleanline (accessed on 20 March 2023).
  38. Mognon, A.; Jovicich, J.; Bruzzone, L.; Buiatti, M. ADJUST: An automatic EEG artifact detector based on the joint use of spatial and temporal features. Psychophysiology 2011, 48, 229–240. [Google Scholar] [CrossRef]
  39. Gevins, A.; Leong, H.; Du, R.; Smith, M.E.; Le, J.; DuRousseau, D.; Zhang, J.; Libove, J. Towards measurement of brain function in operational environments. Biol. Psychol. 1995, 40, 169–186. [Google Scholar] [CrossRef] [PubMed]
  40. Anguera, J.A.; Boccanfuso, J.; Rintoul, J.L.; Al-Hashimi, O.; Faraji, F.; Janowich, J.; Kong, E.; Larraburo, Y.; Rolle, C.; Johnston, E.; et al. Video game training enhances cognitive control in older adults. Nature 2013, 501, 97–101. [Google Scholar] [CrossRef] [PubMed]
  41. Coelli, S.; Sclocco, R.; Barbieri, R.; Reni, G.; Zucca, C.; Bianchi, A.M. In EEG-based index for engagement level monitoring during sustained attention. In Proceedings of the 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milan, Italy, 25–29 August 2015; pp. 1512–1515. [Google Scholar]
  42. Kakkos, I.; Dimitrakopoulos, G.N.; Gao, L.; Zhang, Y.; Qi, P.; Matsopoulos, G.K.; Thakor, N.; Bezerianos, A.; Sun, Y. Mental Workload Drives Different Reorganizations of Functional Cortical Connectivity Between 2D and 3D Simulated Flight Experiments. IEEE Trans. Neural Syst. Rehabil. Eng. 2019, 27, 1704–1713. [Google Scholar] [CrossRef] [PubMed]
  43. Prinzel, L.J.; Freeman, F.G.; Scerbo, M.W.; Mikulka, P.J.; Pope, A.T. A closed-loop system for examining psychophysiological measures for adaptive task allocation. Int. J. Aviat. Psychol. 2000, 10, 393–410. [Google Scholar] [CrossRef]
  44. Holm, A.; Lukander, K.; Korpela, J.; Sallinen, M.; Muller, K.M. Estimating brain load from the EEG. Sci. World J. 2009, 9, 639–651. [Google Scholar] [CrossRef]
  45. Goldberg, J.H.; Kotval, X.P. Computer interface evaluation using eye movements: Methods and constructs. Int. J. Ind. Ergon. 1999, 24, 631–645. [Google Scholar] [CrossRef]
  46. Sharafi, Z.; Shaffer, T.; Sharif, B.; Guéhéneuc, Y.-G. Eye-tracking metrics in software engineering. In Proceedings of the 2015 Asia-Pacific Software Engineering Conference (APSEC), New Delhi, India, 1–4 December 2015; pp. 96–103. [Google Scholar]
  47. Shannon, C.E. A mathematical theory of communication. Bell Syst. Tech. J. 1948, 27, 379–423. [Google Scholar] [CrossRef]
  48. Kazanzides, P.; Chen, Z.; Deguet, A.; Fischer, G.S.; Taylor, R.H.; DiMaio, S.P. An open-source research kit for the da Vinci® Surgical System. In Proceedings of the 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China, 31 May–7 June 2014; pp. 6434–6439. [Google Scholar]
  49. Cook, R.D. Detection of Influential Observation in Linear-Regression. Technometrics 1977, 19, 15–18. [Google Scholar]
  50. Bates, D.; Mächler, M.; Bolker, B.; Walker, S. Fitting linear mixed-effects models using lme4. arXiv 2014, arXiv:1406.5823. [Google Scholar]
  51. Hagan, M.T.; Demuth, H.B.; Beale, M. Neural Network Design; PWS Publishing, Co.: Boston, MA, USA, 1997. [Google Scholar]
  52. Tibshirani, R. Regression shrinkage and selection via the Lasso. J. R. Stat. Soc. Ser. B-Methodol. 1996, 58, 267–288. [Google Scholar] [CrossRef]
  53. Gevins, A.; Smith, M.E. Neurophysiological measures of cognitive workload during human-computer interaction. Theor. Issues Ergon. Sci. 2003, 4, 113–131. [Google Scholar] [CrossRef]
  54. Salvendy, G. Handbook of Human Factors and Ergonomics; John Wiley & Sons: New York, NY, USA, 2012. [Google Scholar]
  55. Tsang, P.S.; Vidulich, M.A. Mental workload and situation awareness. In Handbook of Human Factors and Ergonomics; Salvendy, G., Ed.; John Wiley & Sons, Inc.: New York, NY, USA, 2006. [Google Scholar]
  56. Borghini, G.; Astolfi, L.; Vecchiato, G.; Mattia, D.; Babiloni, F. Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neurosci. Biobehav. Rev. 2014, 44, 58–75. [Google Scholar] [CrossRef]
  57. Yamada, F. Frontal midline theta rhythm and eyeblinking activity during a VDT task and a video game: Useful tools for psychophysiology in ergonomics. Ergonomics 1998, 41, 678–688. [Google Scholar] [CrossRef]
  58. Fairclough, S.H.; Venables, L. Prediction of subjective states from psychophysiology: A multivariate approach. Biol. Psychol. 2006, 71, 100–110. [Google Scholar] [CrossRef]
  59. Scharinger, C.; Soutschek, A.; Schubert, T.; Gerjets, P. Comparison of the Working Memory Load in N-Back and Working Memory Span Tasks by Means of EEG Frequency Band Power and P300 Amplitude. Front. Hum. Neurosci. 2017, 11, 6. [Google Scholar] [CrossRef]
  60. Eriksson, J.; Vogel, E.K.; Lansner, A.; Bergstrom, F.; Nyberg, L. Neurocognitive Architecture of Working Memory. Neuron 2015, 88, 33–46. [Google Scholar] [CrossRef]
  61. Vogel, E.K.; Machizawa, M.G. Neural activity predicts individual differences in visual working memory capacity. Nature 2004, 428, 748–751. [Google Scholar] [CrossRef]
  62. Sauseng, P.; Klimesch, W.; Schabus, M.; Doppelmayr, M. Fronto-parietal EEG coherence in theta and upper alpha reflect central executive functions of working memory. Int. J. Psychophysiol. 2005, 57, 97–103. [Google Scholar] [CrossRef]
  63. Sauseng, P.; Griesmayr, B.; Freunberger, R.; Klimesch, W. Control mechanisms in working memory: A possible function of EEG theta oscillations. Neurosci. Biobehav. Rev. 2010, 34, 1015–1022. [Google Scholar] [CrossRef]
  64. Klimesch, W. Memory processes, brain oscillations and EEG synchronization. Int. J. Psychophysiol. 1996, 24, 61–100. [Google Scholar] [CrossRef]
  65. Pfurtscheller, G.; Klimesch, W. Topographical display and interpretation of event-related desynchronization during a visual-verbal task. Brain Topogr. 1990, 3, 85–93. [Google Scholar] [CrossRef] [PubMed]
  66. Pfurtscheller, G.; Stancak Jr, A.; Neuper, C. Event-related synchronization (ERS) in the alpha band—An electrophysiological correlate of cortical idling: A review. Int. J. Psychophysiol. 1996, 24, 39–46. [Google Scholar] [CrossRef] [PubMed]
  67. Fukuda, K.; Mance, I.; Vogel, E.K. α power modulation and event-related slow wave provide dissociable correlates of visual working memory. J. Neurosci. 2015, 35, 14009–14016. [Google Scholar] [CrossRef] [PubMed]
  68. Klimesch, W. Alpha-band oscillations, attention, and controlled access to stored information. Trends Cogn. Sci. 2012, 16, 606–617. [Google Scholar] [CrossRef] [PubMed]
  69. Gevins, A.; Smith, M.E. Neurophysiological measures of working memory and individual differences in cognitive ability and cognitive style. Cereb. Cortex 2000, 10, 829–839. [Google Scholar] [CrossRef] [PubMed]
  70. Gevins, A.; Smith, M.E.; McEvoy, L.; Yu, D. High-resolution EEG mapping of cortical activation related to working memory: Effects of task difficulty, type of processing, and practice. Cereb. Cortex 1997, 7, 374–385. [Google Scholar] [CrossRef]
  71. Lehtelä, L.; Salmelin, R.; Hari, R. Evidence for reactive magnetic 10-Hz rhythm in the human auditory cortex. Neurosci. Lett. 1997, 222, 111–114. [Google Scholar] [CrossRef]
  72. Becker, R.; Pefkou, M.; Michel, C.M.; Hervais-Adelman, A.G. Left temporal alpha-band activity reflects single word intelligibility. Front. Syst. Neurosci. 2013, 7, 121. [Google Scholar] [CrossRef]
  73. Foxe, J.J.; Snyder, A.C. The role of alpha-band brain oscillations as a sensory suppression mechanism during selective attention. Front. Psychol. 2011, 2, 154. [Google Scholar] [CrossRef]
  74. Kelly, S.P.; Lalor, E.C.; Reilly, R.B.; Foxe, J.J. Increases in alpha oscillatory power reflect an active retinotopic mechanism for distracter suppression during sustained visuospatial attention. J. Neurophysiol. 2006, 95, 3844–3851. [Google Scholar] [CrossRef]
  75. Payne, L.; Guillory, S.; Sekuler, R. Attention-modulated alpha-band oscillations protect against intrusion of irrelevant information. J. Cogn. Neurosci. 2013, 25, 1463–1476. [Google Scholar] [CrossRef] [PubMed]
  76. Sauseng, P.; Klimesch, W.; Heise, K.F.; Gruber, W.R.; Holz, E.; Karim, A.A.; Glennon, M.; Gerloff, C.; Birbaumer, N.; Hummel, F.C. Brain oscillatory substrates of visual short-term memory capacity. Curr. Biol. 2009, 19, 1846–1852. [Google Scholar] [CrossRef] [PubMed]
  77. Haenschel, C.; Baldeweg, T.; Croft, R.J.; Whittington, M.; Gruzelier, J. Gamma and beta frequency oscillations in response to novel auditory stimuli: A comparison of human electroencephalogram (EEG) data with in vitro models. Proc. Natl. Acad. Sci. USA 2000, 97, 7645–7650. [Google Scholar] [CrossRef] [PubMed]
  78. Huster, R.J.; Enriquez-Geppert, S.; Lavallee, C.F.; Falkenstein, M.; Herrmann, C.S. Electroencephalography of response inhibition tasks: Functional networks and cognitive contributions. Int. J. Psychophysiol. 2013, 87, 217–233. [Google Scholar] [CrossRef]
  79. Spitzer, B.; Haegens, S. Beyond the Status Quo: A Role for Beta Oscillations in Endogenous Content (Re)Activation. eNeuro 2017, 4. [Google Scholar] [CrossRef]
  80. Chen, Y.; Huang, X. Modulation of Alpha and Beta Oscillations during an n-back Task with Varying Temporal Memory Load. Front. Psychol. 2015, 6, 2031. [Google Scholar] [CrossRef]
  81. Wang, S.; Gwizdka, J.; Chaovalitwongse, W.A. Using wireless EEG signals to assess memory workload in the n-back task. IEEE Trans. Hum. -Mach. Syst. 2015, 46, 424–435. [Google Scholar] [CrossRef]
  82. Marchitto, M.; Benedetto, S.; Baccino, T.; Canas, J.J. Air traffic control: Ocular metrics reflect cognitive complexity. Int. J. Ind. Ergon. 2016, 54, 120–130. [Google Scholar] [CrossRef]
  83. Tao, D.; Tan, H.; Wang, H.; Zhang, X.; Qu, X.; Zhang, T. A Systematic Review of Physiological Measures of Mental Workload. Int. J. Environ. Res. Public Health 2019, 16, 2716. [Google Scholar] [CrossRef]
  84. Brookings, J.B.; Wilson, G.F.; Swain, C.R. Psychophysiological responses to changes in workload during simulated air traffic control. Biol. Psychol. 1996, 42, 361–377. [Google Scholar] [CrossRef]
  85. Veltman, J.A.; Gaillard, A.W. Physiological workload reactions to increasing levels of task difficulty. Ergonomics 1998, 41, 656–669. [Google Scholar] [CrossRef] [PubMed]
  86. Tole, J.R.; Stephens, A.T.; Harris, R.L.; Ephrath, A.R. Visual scanning behavior and mental workload in aircraft pilots. Aviat. Space Environ. Med. 1982, 53, 54–61. [Google Scholar] [PubMed]
  87. Di Nocera, F.; Camilli, M.; Terenzi, M. A random glance at the flight deck: Pilots’ scanning strategies and the real-time assessment of mental workload. J. Cogn. Eng. Decis. Mak. 2007, 1, 271–285. [Google Scholar] [CrossRef]
  88. Di Stasi, L.L.; Diaz-Piedra, C.; Rieiro, H.; Sanchez Carrion, J.M.; Martin Berrido, M.; Olivares, G.; Catena, A. Gaze entropy reflects surgical task load. Surg. Endosc. 2016, 30, 5034–5043. [Google Scholar] [CrossRef] [PubMed]
  89. Diaz-Piedra, C.; Sanchez-Carrion, J.M.; Rieiro, H.; Di Stasi, L.L. Gaze-based Technology as a Tool for Surgical Skills Assessment and Training in Urology. Urology 2017, 107, 26–30. [Google Scholar] [CrossRef]
  90. Lohmeyer, Q.; Matthiesen, S.; Meboldt, M. Task-dependent Visual Behaviour of Engineering Designers-an Eye Tracking Experiment, DS 77. In Proceedings of the DESIGN 2014 13th International Design Conference, Dubrovnik, Croatia, 19–22 May 2014; pp. 549–558. [Google Scholar]
  91. Van Orden, K.F.; Limbert, W.; Makeig, S.; Jung, T.P. Eye activity correlates of workload during a visuospatial memory task. Hum. Factors 2001, 43, 111–121. [Google Scholar] [CrossRef]
  92. Fournier, L.R.; Wilson, G.F.; Swain, C.R. Electrophysiological, behavioral, and subjective indexes of workload when performing multiple tasks: Manipulations of task difficulty and training. Int. J. Psychophysiol. 1999, 31, 129–145. [Google Scholar] [CrossRef]
  93. Luque-Casado, A.; Perales, J.C.; Cardenas, D.; Sanabria, D. Heart rate variability and cognitive processing: The autonomic response to task demands. Biol. Psychol. 2016, 113, 83–90. [Google Scholar] [CrossRef]
  94. Bernardi, L.; Wdowczyk-Szulc, J.; Valenti, C.; Castoldi, S.; Passino, C.; Spadacini, G.; Sleight, P. Effects of controlled breathing, mental activity and mental stress with or without verbalization on heart rate variability. J. Am. Coll. Cardiol. 2000, 35, 1462–1469. [Google Scholar] [CrossRef]
  95. Hoshikawa, Y.; Yamamoto, Y. Effects of Stroop color-word conflict test on the autonomic nervous system responses. Am. J. Physiol. 1997, 272, H1113–H1121. [Google Scholar] [CrossRef]
  96. Beda, A.; Jandre, F.C.; Phillips, D.I.; Giannella-Neto, A.; Simpson, D.M. Heart-rate and blood-pressure variability during psychophysiological tasks involving speech: Influence of respiration. Psychophysiology 2007, 44, 767–778. [Google Scholar] [CrossRef]
Sensors 23 04354 g001 550
Figure 1. An example of a stereo viewer which has target game information during a peg transfer task. There are two target numbers (3 and 2). The participants were instructed to tap the foot pedal each time the timer’s second digit number corresponded to the target numbers. (In the example, the timer displays 00:32. The second digit is 2.) Once the participants play the game successfully, a green square in the upper corner of the stereo display appears and the score is increased by 1.
Figure 1. An example of a stereo viewer which has target game information during a peg transfer task. There are two target numbers (3 and 2). The participants were instructed to tap the foot pedal each time the timer’s second digit number corresponded to the target numbers. (In the example, the timer displays 00:32. The second digit is 2.) Once the participants play the game successfully, a green square in the upper corner of the stereo display appears and the score is increased by 1.
Sensors 23 04354 g001
Sensors 23 04354 g002 550
Figure 2. (a) EEG topographic plots in accordance with each band power. Red and blue color areas indicate higher and lower band power of the suturing task compared to the peg transfer task, respectively; (b) difference of TAR for the effect of the primary task (error bars show standard errors). Asterisk indicates a statistically significant difference with a significance level of 0.05.
Figure 2. (a) EEG topographic plots in accordance with each band power. Red and blue color areas indicate higher and lower band power of the suturing task compared to the peg transfer task, respectively; (b) difference of TAR for the effect of the primary task (error bars show standard errors). Asterisk indicates a statistically significant difference with a significance level of 0.05.
Sensors 23 04354 g002
Sensors 23 04354 g003 550
Figure 3. (a) EEG topographic plots in accordance with each band power and multi-task requirement (n-back/target game). Red and blue color areas indicate higher and lower band power of multi-task requirement tasks compared to single task conditions, respectively; (b) interaction plot of theta (FP1) indicating significant differences between peg transfer with the target game and other levels of peg transfer without the target game and suturing task (error bars show standard errors); (c) difference of TAR for the effect of the multi-task requirement (error bars show standard errors). Asterisk indicates a statistically significant difference with a significance level of 0.05.
Figure 3. (a) EEG topographic plots in accordance with each band power and multi-task requirement (n-back/target game). Red and blue color areas indicate higher and lower band power of multi-task requirement tasks compared to single task conditions, respectively; (b) interaction plot of theta (FP1) indicating significant differences between peg transfer with the target game and other levels of peg transfer without the target game and suturing task (error bars show standard errors); (c) difference of TAR for the effect of the multi-task requirement (error bars show standard errors). Asterisk indicates a statistically significant difference with a significance level of 0.05.
Sensors 23 04354 g003
Sensors 23 04354 g004 550
Figure 4. Bar plot of SF ratio for the effect of primary task (error bars show standard errors). Asterisk indicates a statistically significant difference with a significance level of 0.05.
Figure 4. Bar plot of SF ratio for the effect of primary task (error bars show standard errors). Asterisk indicates a statistically significant difference with a significance level of 0.05.
Sensors 23 04354 g004
Sensors 23 04354 g005 550
Figure 5. (a) Difference of gaze entropy and (b) SF ratio across multi-task requirements (error bars show standard errors). Asterisk indicates a statistically significant difference with a significance level of 0.05.
Figure 5. (a) Difference of gaze entropy and (b) SF ratio across multi-task requirements (error bars show standard errors). Asterisk indicates a statistically significant difference with a significance level of 0.05.
Sensors 23 04354 g005
Sensors 23 04354 g006 550
Figure 6. Interaction plot of LF/HF ratio indicating a significant difference between peg transfer and suturing tasks for a single task condition. Error bars show standard errors.
Figure 6. Interaction plot of LF/HF ratio indicating a significant difference between peg transfer and suturing tasks for a single task condition. Error bars show standard errors.
Sensors 23 04354 g006
Sensors 23 04354 g007 550
Figure 7. (a) Difference of meanRR for effect of multi-task requirement (n-back); (b) difference of stdRR for effect of multi-task requirement (n-back); (c) difference of RMSSD for effect of multi-task requirement (n-back). Error bars show standard errors. Asterisk indicates a statistically significant difference with a significance level of 0.05.
Figure 7. (a) Difference of meanRR for effect of multi-task requirement (n-back); (b) difference of stdRR for effect of multi-task requirement (n-back); (c) difference of RMSSD for effect of multi-task requirement (n-back). Error bars show standard errors. Asterisk indicates a statistically significant difference with a significance level of 0.05.
Sensors 23 04354 g007
Table
Table 1. Simulated RAS related surgical tasks.
Table 1. Simulated RAS related surgical tasks.
Single Taskn-Back TaskTarget Game Task
Peg transferPeg transferPeg transfer with n-backPeg transfer with target game
SuturingSuturingSuturing with n-backSuturing with target game
Table
Table 2. Classification model results with identified subset of input variables.
Table 2. Classification model results with identified subset of input variables.
Classification ModelIdentified Features (Lasso)AccuracyF-1 Score
EEGEye-trackingHRV
ThetaAlphaBetaIndex
Primary taskPeg transfer vs. SuturingF3, FC6CP1, FZF8TARSF ratio 65.1866.27
Multi-task requirementSingle task vs. n-backAF4, P7, T7CZ, P3, P7, T7AF3, CP2, CP5, F3, FC6, FP2, FZ, P4, P8, PO3, T7 Gaze entropyMeanRR, StdRR, LF/HF78.6179.40
Single task vs. Target gameAF4, F4, F7, FC1, FP2, P7, PZ, T7AF4, CP5, FC6, FP2AF3, CP2, CZ, FC6,FP1, P7, PO3TARGaze entropy, SF ratioMeanRR75.6576.44
n-back vs. Target gameF3, FP2, PZCp5, FP2, T7AF3, FP1, FP2, T7 Gaze entropyLF/HF72.3973.21
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Lim, C.; Barragan, J.A.; Farrow, J.M.; Wachs, J.P.; Sundaram, C.P.; Yu, D. Physiological Metrics of Surgical Difficulty and Multi-Task Requirement during Robotic Surgery Skills. Sensors 2023, 23, 4354. https://doi.org/10.3390/s23094354

AMA Style

Lim C, Barragan JA, Farrow JM, Wachs JP, Sundaram CP, Yu D. Physiological Metrics of Surgical Difficulty and Multi-Task Requirement during Robotic Surgery Skills. Sensors. 2023; 23(9):4354. https://doi.org/10.3390/s23094354

Chicago/Turabian Style

Lim, Chiho, Juan Antonio Barragan, Jason Michael Farrow, Juan P. Wachs, Chandru P. Sundaram, and Denny Yu. 2023. "Physiological Metrics of Surgical Difficulty and Multi-Task Requirement during Robotic Surgery Skills" Sensors 23, no. 9: 4354. https://doi.org/10.3390/s23094354

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop