Towards Virtual VATS, Face, and Construct Evaluation for Peg Transfer Training of Box, VR, AR, and MR Trainer

The aim of this study is to develop and assess the peg transfer training module face, content and construct validation use of the box, virtual reality (VR), cognitive virtual reality (CVR), augmented reality (AR), and mixed reality (MR) trainer, thereby to compare advantages and disadvantages of these simulators. Training system (VatsSim-XR) design includes customized haptic-enabled thoracoscopic instruments, virtual reality helmet set, endoscope kit with navigation, and the patient-specific corresponding training environment. A cohort of 32 trainees comprising 24 novices and 8 experts underwent the real and virtual simulators that were conducted in the department of thoracic surgery of Yunnan First People's Hospital. Both subjective and objective evaluations have been developed to explore the visual and haptic potential promotions in peg transfer education. Experiments and evaluation results conducted by both professional and novice thoracic surgeons show that the surgery skills from experts are better than novices overall, AR trainer is able to provide a more balanced training environments on visuohaptic fidelity and accuracy, box trainer and MR trainer demonstrated the best realism 3D perception and surgical immersive performance, respectively, and CVR trainer shows a better clinic effect that the traditional VR trainer. Combining these in a systematic approach, tuned with specific fidelity requirements, medical simulation systems would be able to provide a more immersive and effective training environment.


Introduction
Video-assisted thoracoscopic surgery (VATS), which is the most common minimal invasive surgery (MIS) therapy for lung carcinomas [1][2][3], is the most widespread cancer in the world with only approximately 16% five-year survival rate. Furthermore, hospitalization outcomes show the patients' quality of life (QoL) and follow-up adjuvant chemotherapy endurance that are significantly promoted compared with traditional thoracotomy, without interfering with survival outcomes [4,5]. Peg transfer training, as one of the essential modules of the fundamental thoracoscopic surgery curriculum, is the compulsory test requirement before the surgeries take the American Board of Surgery examination [6,7]. With two Maryland clamps, trainees need to pick up six tiny blocks with nondominate hand, transfer to the other hand, and place them stably on the other side peg, respectively, whereby to achieve the bimanual dexterity and eye-hand coordinative skills training [7,8].
e surgical simulators can be divided into box trainer, VR trainer, AR trainer, and MR trainer, and each class has advantages and disadvantages.
e Box trainer is the traditional surgical training framework based on a real physical model that is portable and easy to operate; however, the disadvantage is this kind of trainer cannot be reused for multitimes. VRbased surgical simulations have attracted many researchers' attentions over the years and gradually turned into a real-life medical training simulation solution, providing repeatable training experience, without ethical or hygienic issues [9][10][11], and the disadvantage of this simulator is lack of high immersive visual and haptic rendering algorithms. Due to the recent advances in the field of AR and MR, cognitive sense has brought into the next level of surgical simulator with enhanced immersion and interactivity [12][13][14]; however, there are also several defects of these simulators, such as the high price, visual uncomfortable, and lack of the real surgical environment fidelity. Maciel et al. developed a virtual reality laparoscopic skill trainer named VBLaST, with real-time evaluation function for the peg transfer [15], and the Lap Mentor ™ simulator [16,17]. Loukas et al. compared the AR-based peg transfer with the box simulator and VR simulator [6], and Huber et al. added a highly immersive 360°real operating room environment to construct an MR training environments [18,19]. Nevertheless, there is no comparative study that has focused box, AR, VR, CVR, and MR simulators on the peg transfer training, in another word, which one is the most effective simulator to shorten the surgical learning curve [20,21]. e aim of this research is to determine which simulator is better by the advantages and disadvantages of the five simulators that were compared through the evaluation results of the face and content and construct. ere are three main innovative contributions in this study, and the overarching one is we compared the experimental data and simulation results among five different kinds of trainers of peg transfer training and summarized the conclusion which may benefit for the advanced research of the virtual surgery. e second one is we addressed a full immersive and accurate scenario to peg transfer training with a detailed assessment tool, and the last one is we addressed the detailed design of the first commercial available VATS VR simulator in both hardware and the training implementation. e structure and content of this article are organized as follows: we briefly make an introduction on peg transfer training and challenges of virtual medical training in the abovementioned part, after that, we review the previous related works on the virtual peg transfer simulator. irdly, the simulator design, with virtual training evaluation experiments, is designed in the methodology part. Fourthly, the experimental data and evaluation results are demonstrated and discussed. Finally, we summarize the results and the potential contributions this paper makes.  [1,15], and after observed considerable VATS lobectomy operations in the thoracic surgery department of Yunnan First People's Hospital, we chose the three-port standardized anterior VATS surgery as the simulated content. As the commonest surgical approach in lung carcinomas lobectomy, the three-port VATS are described as opened 3 single diameters of 1.5 cm incisions in the side of the patient's thorax: one is for endoscopic camera recording the operation field and the other two are the operative tunnels for surgical instruments stretching in and out to dissect and staple the lesion [2,16]. e surgical simulator is called VatsSim-XR, as shown in Figure 1, based on the procedures need to be simulated in the abovementioned, and hardware of VatsSim-XR (VR, AR, and MR) is 60 × 67 × 160 cm and principally composed of a 24-inch naked eye 3D display, a footswitch pedal, two VATS surgical devices connected with dual haptic devices with one held as the drag instrument and the another mimics the stapling device, an endoscope connected with a 3D mouse (3Dconnexion, Germany), an HMD, and a workstation. To improve the surgical immersion, we proposed a solution to incorporate multiple higher-fidelity factors towards a surgeon's sensations (vision, touch, and hearing) during practical surgery to achieve total immersion. We developed a versatile simulation platform VatsSim-XR that is able to implement 5 different simulation modes, specifically, the AR and MR modes. To ensure the same training block between the real and virtual peg transfer platform, the .STL file is exported from the blender firstly, and then, the 3D printer is employed to print all the training pegs for the box training simulator. For the AR peg transfer simulation, we utilized a web camera Logitech CC2900ep HD1080p as the detected sensor for the display, and for the CVR training and the MR training, HTC VIVE (HTC Corporation, Taiwan, China) is the first virtual reality helmet, and it is used to show the surgical environments [22]. Especially, for the real surgical environments rendering, we employed the Samsung Gear 360 (Samsung, South Korea) to record the 360°video in the operation room (OR) of an upper-right lobe VATS in Yunnan First People's Hospital [18,19,23,24].

Software.
We designed a framework for the implementation of peg transfer training simulation, as shown in Figure 2. e virtual simulator mainly includes two parts: visual (physics and graphic) and haptic (PHANTOM Omni hardware and OpenHaptic software). e corresponding OpenHaptic function will be invoked when pressing the physical button of PHANTOM Omni (Geomagic, USA) or there is contact between the virtual surgical instrument and the model with a rigid body, and then, there will have a corresponding force feedback. ere are physics with rigid or deform and graphics with a shader in the visual aspect. We combine the visual plugin, such as Bullet, AR kit and SteamVR, and the haptic plugin OpenHaptic into the Unity3D (version 2017.3.1) for simulation training.

Participants.
e evaluation cohort of 32 trainees comprising 24 novices and 8 experts underwent the peg transfer procedures on the five simulators conducted in the department of thoracic surgery of Yunnan First People's Hospital. Both expert and novice trainers firstly receive a didactic teaching from a developer of these simulators with the tasks and techniques of peg transfer. e medical experience of the novice group is 3 to 6 years and the expert group is 11 to 30 years. To evaluate if the virtual game experience may affect the training result, the experience of using both VR and HMD had been recored during the experiments.
e demographic details of the trainees are shown in Table 1.

Simulator Tasks.
Both novice and expert groups after didactic training session are towards to evaluate the simulator validity. e order of training is the box, VR, CVR, AR, and MR, and the 30-minute break should be interspersed between each peg transfer simulator test. Firstly, the trainee adjusted the virtual endoscope to reach the proper position and angle to obtain the best viewing perspective of the operating scene; after that, two clamps are used to grasp and transfer the block model on the peg base. e left-hand clamp grabs the block on the left side and passes it to the right clamp; then, the right clamp places the block on the right side, which is demonstrated in Figure 3.
After the aforementioned experiments, all 32 trainees' performance is recorded by the objective questionnaires to evaluate the detailed surgical skills and be compared with each other. Subjective questionnaires also need to be filled to evaluate the face and content validity of five simulators, and the detailed evaluation flow chart is demonstrated in Figure 4.

Evaluation.
A questionnaire consisting of 13 questions about the visual and haptic aspects of the simulators was created for face and content assessment validity. Due to the lack of experience of the novice, subjective judgment may have a large error in the understanding of the simulator.   then compare the performance of multiple simulators under six parameters.

Data Analysis.
In terms of face and content validity, descriptive statistics were used to analyze the questionnaire data by calculating the mean value of subjective question and their standard errors. For the construct validity, assessing the test results of novice and expert on six parameters of five simulators with Shapiro-Wilk test, p > 0.05, is considered to be subject to normal distribution. Box plots are used for analysis of objective parameters. Each group of T of novices and experts on the five simulators follows a normal distribution and was compared between groups using an independent-sample t-test, and each group of T of five simulators on novices and experts was compared between groups using the one-way ANOVA test. But, most of the CL, EL, CA, EA, and ND were not a normal distribution, each group of those data of novices and experts on the five simulators was compared between groups using the twotailed Mann-Whitney U test, and each group of those data of five simulators on novices and experts was compared between groups using the Kruskal-Wallis test. A p value < 0.05 was considered significant. Analyses were performed using the software package, IBM SPSS Statistics, version 20.0.

Face and Content Validation.
Aiming to the perceptions of visual and tactile sensations during the use of the five simulators, we set up a subjective questionnaire with 13 questions using the scoring method of the 5-point Likerttype scale. After the experiment of the simulation operation, the questionnaire was filled by 8 experts for the experience during the operation. rough the Shapiro-Wilk test, each set of score data is an approximately normal distribution. e thirteen subjective questionnaires were collated, and the score data of each simulator were assessed by the Shapiro-Wilk test. e average and standard deviation of each set of data are shown in Table 3, and the distribution of the average score is shown in Figure 5. It can be found that the box and MR scores demonstrated a higher score than the AR and VR scores in a visual sense. In terms of haptic sense, the box scores are higher than the AR, VR, CVR, and MR scores, which demonstrated the best haptic immersion among these virtual simulators.

Construct Validation.
During the operation of 32 trainees, the VR, CVR, AR, and MR simulators automatically recorded the trajectory length and rotation angle of the surgical instruments and endoscopes in real time, as well as the total time spent and the numbers of block drop on each operation. However, the box simulator cannot record the length and angle of the motion track and the numbers of the block drop, and the total time of the operation can only be roughly recorded by the timer. Six box plots for the experimental data of six parameters T, CL, EL, CA, EA, and ND were made by SPSS, and those box plots which include mediums and means are shown in Figure 6. Similarly, after calculating these p values of the parameters of the different simulators for the novice and expert groups, the results are shown in Table 4. Most of the evaluation parameters of the expert group are lower than that of the novice group in each simulator. p < 0.05, which is only for the EL of VR simulator. EL and EA of CVR simulator data of the expert group demonstrated a higher score than that of the novice. Comparing VR, CVR, and MR scores of the expert group, those scores of the VR group demonstrate the best performance. e clamp moving trajectory of AR, VR, CVR, and MR groups during the simulation is demonstrated in Figure 7, and it shows that the constructive difference between these four training groups and the AR group demonstrated a betterconcentrated trajectory than other groups.

Discussion
ere is an increasing use of simulators to learn, improve, and rehearse surgical skills in medical training. e VatsSim-XR is a versatile simulator that is able to perform multiple surgical training scenarios and implement 4 different simulation modes (VR, CVR, AR, and MR) on a single device. e Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) and the American College of Surgeons (ACS) have established a standard for fundamentals of laparoscopic surgery (FLS) training and surgical skills assessment [25][26][27].
e FLS consists of five basic foundations: peg transfer, pattern cutting, ligation loop, and suturing with either intracorporal or extracorporeal knot tying, and these tasks are designed to train the operational skills and assessment of medical interns and residents. Peg transfer is the basic training task to train the dual-hands coordination and the hand-eye coordination for the medical interns and residents [8]. FLS simulation training is mostly based on box and VR simulators in the market.
In terms of haptic, the box simulator is superior to the other four simulators, but an operation object is an object that is easily destroyed after repeated use. In terms of visual, the box and MR simulators are better. e AR simulator is the best in terms of interactive, and in the interactive environment part, where the simulator is more immersive: MR simulator > CVR simulator; however, the box, VR, and AR simulators have almost no surroundings and are less immersive.
Compared with novice groups of six parameters on five simulators, most of the expert groups show short operation time, short track length, small angle accumulation, and small drop number, but only these three groups, the EL of the VR simulator, EL, and EA of CVR the simulator, show the opposite result, and the main reason is that when adjusting the endoscope, the novice does not move and rotate the endoscope to an optimal viewing angle in strict accordance with the surgical standard, resulting in greater difference. For the VR, CVR, and MR simulators of six parameters, the data size of expert groups basically follows a rule VR < CVR < MR, but only the MR simulator data of the CA parameter are smaller than VR and MR simulators, because the experts have adapted to the 360°real operating room scene in the MR simulator in advance. Although the MR simulator is not perfect in terms of T, CL, EL, CA, EA, and ND, it provides a highly realistic operating room for hospital  interns and residents. e environment, closer to the real surgery site, has great potential for development. Experiments and evaluation results conducted by both professional and novice thoracic surgeons show that the AR trainer is able to provide a more balanced training environment on visuohaptic fidelity and accuracy, the box trainer and MR trainer demonstrated the best realism 3D perception and surgical immersive performance, respectively, and the CVR trainer shows a better clinic effect that the traditional VR trainer. e advantages and disadvantages of the five simulators are shown in Table 5. e haptic, visual, and surrounding     Journal of Healthcare Engineering environments are compared separately by levels (III means the highest and I means the lowest). e complex design of our integrated system will provide a more immense and effective training environment for the medical surgery simulation.

Conclusions
In this paper, we have detailed a virtual surgical educative simulator with realistic performance in both visual and haptic sensations for the peg transfer procedures, build the face, content and construct validation on the peg transfer training by use of the box, VR, AR, and MR trainer, thereby to compare advantages and disadvantages of these simulators. However, during the interaction, the sense of touch is not immersive enough; furthermore, there is a cross between the gripper and the interacting virtual object. What is more, in the comparison experiments design, the box simulator is not able to automatically record the trajectory and rotation angle parameters of the instrument and endoscope, which means the comparison of the six parameters is not comprehensive enough in this manuscript. In the future works, we will improve the visual and haptic experiment design parts to achieve a high immersive and realistic simulation environment both visual and haptic perception.

Data Availability
e data used to support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest
e authors declare that there are no conflicts of interest regarding the publication of this paper.