Effects on cognitive load of tutoring in virtual reality simulation training

Aims: According to the guidance hypothesis, tutoring during technical skills training can result in tutoring overreliance, reflected in a negative effect on performance when tutoring is discontinued. In this study, we wanted to explore if similar effects would be found for cognitive load. Methods: Two cohorts of novice medical students were recruited for distributed virtual simulation training (five practice blocks of three procedures): 16 participants received intermittent simulator-integrated tutoring and 14 participants served as a reference cohort and did not receive simulator-integrated tutoring. Cognitive load during simulation was estimated using secondary task reaction time. Linear mixed models were used to account for repeated measurements. Results: Overall, the tutored cohort had a significantly higher cognitive load than the reference cohort (mean difference = 7 %, p=0.006). Simulator-integrated tutoring did seem to lower cognitive load when active but also caused the tutored cohort to have a substantially higher cognitive load in subsequent performances where it was turned off (mean difference = 7 %, respectively, p<<0.001). Conclusions: Concurrent feedback by simulator-integrated tutoring causes tutoring over-reliance and modifies cognitive load. This suggests that tutoring, in addition to degrading motor skills learning also affects the cognitive processes involved.


Introduction
Virtual reality (VR) simulation training of surgical procedures is often self-directed-especially when training requires prolonged repeated practice. Self-directed practice allows the learner to practice according to individual needs and schedule but also requires clear training goals along with supports to scaffold the learning experience (Brydges et al., 2009). Consequently, feedback is an important learning support in simulation-based education . VR simulation training offers unique possibilities for automated feedback that can be used both for summative assessment but also for real-time feedback and tutoring.
A review of feedback in simulation-based procedural skills training found that concurrent feedback can result in tutoring over-reliance, and that terminal feedback results in better long term learning (Hatala et al., 2013). Tutoring over-reliance can be understood from a motor skills learning perspective by the guidance hypothesis: continuous feedback during skills acquisition degrades learning (Schmidt and Wulf, 1997). Distributing practice seems to partially counteract this negative effect on performance in simulation-based training (Andersen et al., 2015). This moderate protective effect of distributed practice could be a result of spacing the practice: distributed practice allows memory consolidation to occur in the interim periods between training sessions, which is not possible under massed practice conditions (Shea et al., 2000).
Tutoring over-reliance can also be understood in the context of cognitive load (CL) theory: feedback during skills training can potentially modify CL, resulting in either an increased CL due to information overload, or a decreased CL due to improved structuring of for example learning instructions for easier processing (Hatala et al., 2013). The effects of tutoring on CL can therefore be mixed and also hypothesized to vary when it comes to immediate effects (performance) and longer term effects (learning). However, almost no research has directly investigated the role of CL during different feedback conditions (Hatala et al., 2013).
In a study on the effects of intermittent tutoring in a distributed VR simulation training program of temporal bone surgery, we aimed to explore the effects of tutoring on CL.

Study design and participants
This was a prospective study of a distributed, temporal bone virtual reality simulation training program in the period Oct. 2015-Jan. 2017 as previously reported (Andersen, Mikkelsen and Sørensen, 2020). One cohort of 16 participants (tutored) and another cohort of 14 participants were recruited among medical students at the University of Copenhagen. The participants were recruited similarly for the two cohorts and represent complete novices without previous exposure to any temporal bone procedure in real-life or simulated. During the training program, two participants dropped out of the tutored cohort and one participant out of the reference cohort because they could not find time to complete the training program.

Training program
Participants trained in the Visible Ear Simulator, an academic freeware virtual reality temporal bone surgical simulator (Sørensen, Mosegaard and Trier, 2009). The training program consisted of five blocks of three identical procedures in the VR simulator (mastoidectomy-a common but complex procedure with sequential drilling of the temporal bone near vital anatomical structures such as the facial nerve and dura) (Andersen, Konge and Sørensen, 2018). Each practice block lasted between 2 and 3.5 hours and was separated by at least a week. Both cohorts were Andersen S, Frendø M, Sørensen M MedEdPublish https://doi.org/10.15694/mep.2020.000051.1 Page | 3 self-directed and had access to a written, on-screen, step-by-step guide of the procedure for instructions.
In addition to this, the tutored cohort received simulator-integrated tutoring by green lighting of the target volume (Andersen et al., 2015) for the first of the three consecutive procedures in each block. The reference cohort did not receive this tutoring.

Outcomes and data analysis
To estimate cognitive load, secondary task reaction time (hundredths/s) was measured manually using a reaction timing device (American Educational Products LLC, USA) as previously reported (Andersen, Konge and Sørensen, 2018). This task consisted of the participant pressing a foot switch in response to an auditory stimulus (a beep) at baseline (immediately before and after the training block) and at two times during simulation (t=5 min and t=15 min). At each measurement time, reaction time was measured in a series of four consecutive stimulus/response times. The individual relative change in reaction time (unitless) during simulation compared to baseline mean was calculated. Data were analyzed in SPSS (SPSS Inc., IL, USA) version 23 for MacOS X using linear mixed model due to repeated measurements as outlined by Leppink (Leppink, 2015). Procedure number (repetition 1-15), measurement time (t=5/15 min), and measurement number (1-4) constituted the repeated effects. The final model further included cohort (tutored/reference) and tutoring during the procedure (yes/no) as fixed effects. Estimated marginal means and 95 % CI intervals are reported. P-values <0.05 are considered statistically significant.

Ethics
The regional ethics committee for the Capital Region of Denmark deemed this study to be exempt (H-15011780). All participants volunteered for the study, signed informed consent, and participation was not part of their study programme.

Discussion
In this study, we investigate the effects on cognitive load of using a simulator-integrated tutor-function for intermittent guidance during VR temporal bone surgical simulation. In agreement with the positive immediate effects and negative learning effects on different performance outcomes (i.e. tutoring over-reliance) previously reported (Andersen, Mikkelsen and Sørensen, 2020), we find a similar pattern for CL: concurrent feedback using a simulator-integrated tutor-function results in less CL being induced in the learner when active but when the tutorfunction is discontinued in subsequent procedures, CL is increased above the level of never tutored novices.
The immediate effect of tutoring during the procedure-a reduced CL-could potentially be explained by a Andersen S, Frendø M, Sørensen M MedEdPublish https://doi.org/10.15694/mep.2020.000051.1 Page | 4 reduction of the extraneous component of CL by the spatial split attention principle (sources of information being physically separated, requiring the learner's attention to shift between these by a change of gaze), and redundancy principle (two self-contained sources of information being replaced with a single source of information) (van Merriënboer and Sweller, 2010). This could potentially have been further substantiated if it had been possible to measure the learners' use of the written on-screen instructions in our setup.
The negative effects on CL in the post-tutored performances and the overall effect on the tutored cohort suggest that tutoring might also interfere on other mechanisms involved in learning. For example it can be hypothesized that tutoring reduces the learners cognitive engagement in the learning task and that the reduction in CL is due to less germane load (used for schema formation and actual learning) being induced. This could potentially explain tutoring over-reliance from a CL theory point of view. However, a major limitation in using secondary task reaction time to estimate CL is that the different components of CL cannot be separated. Measuring germane load (Klepsch, Schmitz and Seufert, 2017) and separating it from intrinsic load is difficult and still debated (Schnotz and Kürschner, 2007), so whether tutoring causes a reduction in the extraneous load component and/or the germane load component remains to be established.

Conclusion
Concurrent feedback by simulator-integrated tutoring during distributed practice of a surgical procedure causes tutoring over-reliance even when tutoring is only used intermittently. This affects not only performance but also modifies cognitive load, adding to the current literature that tutoring not only degrades motor skills but also affects the cognitive processes involved. The problem of tutoring over-reliance in simulation-based training can therefore be viewed from both a motor skills learning perspective (guidance hypothesis) and a cognitive load perspective: tutoring might degrade motor skills learning and at the same time modify CL for example by causing less cognitive engagement, thereby reducing the germane load. Future investigations on the relative contribution of different components of CL might shed further light on the exact mechanisms.