A holographic telementoring system depicting surgical instrument movements for real-time guidance in open surgeries

Background and Objective: During open surgeries, telementoring serves as a valuable tool for transferring surgical knowledge from a specialist surgeon (mentor) to an operating surgeon (mentee). Depicting the intended movements of the surgical instruments over the operative field improves the understanding of the required tool-tissue interaction. The objective of this work is to develop a telementoring system tailored for open surgeries, enabling the mentor to remotely demonstrate the necessary motions of surgical instruments to the mentee. Methods: A remote telementoring system for open surgery was implemented. The system generates visual cues in the form of virtual surgical instrument motion augmented onto the live view of the operative field. These cues can be rendered on both conventional screens in the operating room and as dynamic holograms on a head mounted display device worn by the mentee. The technical performance of the system was evaluated, where the operating room and remote location were geographically separated and connected via the Internet. Additionally, user studies were conducted to assess the effectiveness of the system as a mentoring tool. Results: The system took 307 ± 12 ms to transmit an operative field view of 1920 × 1080 resolution, along with depth information spanning 36 cm, from the operating room to the remote location. Conversely, it took 145 ± 14 ms to receive the motion of virtual surgical instruments from the remote location back to the operating room. Furthermore, the user studies demonstrated: (a) mentor ’ s capability to annotate the operative field with an accuracy of 3.92 ± 2.1 mm, (b) mentee ’ s ability to comprehend and replicate the motion of surgical instruments in real-time with an average deviation of 12.8 ± 3 mm, (c) efficacy of the rendered dynamic holograms in conveying information intended for surgical instrument motion. Conclusions: The study demonstrates the feasibility of transmitting information over the Internet from the mentor to the mentee in the form of virtual surgical instruments ’ motion and projecting it as holograms onto the live view of the operative field. This holds potential to enhance real-time collaborative capabilities between the mentor and the mentee during an open surgery.


Introduction
Telemedicine involves the use of information and telecommunications technologies to provide and support health care services over a distance [1].It facilitates exchange of medical knowledge for diagnoses, treatment, and prevention of diseases.It has also emerged as a teaching tool for remote education of health care providers across geographic boundaries [2].Consequently, due to the availability of prompt expert medical care, telemedicine has addressed staffing shortages in diverse locations, such as rural health centers, ambulance vehicles, and home monitoring.The applications of telemedicine span across telecardiology, teleradiology, telepathology, teledermatology, teleophthalmology, teleoncology, and telepsychiatry [3].
Telemedicine when applied to surgical domain takes the form of teleconsultation, telesurgery, and telementoring.Tele-consultation is used for preoperative assessment and diagnosis before the surgery, and for post-operative consultations and monitoring following the surgical procedure [4].Telesurgery is defined as the procedure performed on a patient, in which the surgeon is not at the immediate site of the operation.Any visualizations or manipulations of the tissues and equipment are done remotely through robotic systems.While these systems have been successfully demonstrated on animate models, their clinical translation has been limited by factors such as networking latency and response time.Additionally, the cost associated with deployment of these systems in rural settings also hinders its widespread adoption.To overcome these limitations, low-cost surgical telementoring technologies have been developed that enables remote transfer of surgical skills.
When telemedicine is used for training and education, it is referred to as 'Telementoring'.More precisely, SAGES defines telementoring as the real-time interactive teaching of surgical techniques by an expert surgeon to a trainee at a different site [5].The first use of telementoring was reported in 1965 by Dr. DeBakey, who transmitted guidance on open heart surgery from the USA to surgeons in Europe [6].Within surgery, telementoring facilitates the distribution of surgical knowledge by means of both technical assistance and real-time guidance between an experienced surgeon (mentor) and a novice (mentee) in different geographical locations [7].The expertise of national or international surgeons can be brought to the patient, leading to reduced travel expenses for both patients and specialists [8].With the recent occurrence of COVID-19 pandemic, telementoring has been gaining wide acceptance as an alternative to on-site mentoring in terms of clinical and educational outcomes [9][10][11].
Given the significance of telemedicine, this work introduces a telementoring system tailored for open surgery.The system facilitates realtime guidance to a mentee inside the operating room from a geographically distant mentor.Visual cues in the form of virtual surgical instrument motion are generated by the mentor in real-time and rendered as dynamic holograms onto the operative field.These cues assist the mentor to demonstrate to the mentee the required surgical instrument motion to operate on the tissue during the surgery.The subsequent sections present the advantages of the proposed system over existing communication modes used in surgical telementoring, the surgical setup and workflow adopted by the mentor and the mentee, and indepth technical details related to system's implementation.Finally, the experimental setup, and results that assess the performance of the system, are presented.

Visual cues used for telementoring
Different modes of communication have been developed and studied to facilitate telementoring during open surgeries.The basic mode includes two-way audio exchange and one-way live video transmission of the operative field from the operating room to the remote location [12].Such audio-visual exchange ensures communication is as effective as face-to-face encounters during intraoperative surgical guidance [13,14].To further assist the mentee, written information from the mentor is also displayed.Texts in the form of sequential steps, safety guidelines, or textual labels aid the mentor in ensuring clarity of surgical instructions and minimizing the misinterpretations during the procedure [14,15].In certain modes, the mentor provides additional support to the mentee by sending screenshots/videos of pre-recorded surgical procedures [16], and preoperative image data (such as CT scans, MRI scans, or three-dimensional (3D) reconstructions of anatomical structures) [17,18].These visuals assist the mentor to explain complex surgical techniques as well as to discuss and/or reference relevant anatomy during the procedure.
To further improve precision and clarity, some modes incorporate laser beam overlays onto the operative field [19].These overlays assist the mentor to precisely point out anatomical structures, identify landmarks, or mark regions of interest.Furthermore, in some modes, the mentor can also augment the acquired view of the operative field using static annotations in the form of geometric primitives (such as labels, points, incision lines, and arrows) [20,21] or virtual models of surgical instruments (such as scalpel, scissors, and retractor) [22,23].These static annotations are displayed on a 2D screen [24] in the operating room, and aids the mentee during the surgical procedure [25].
To enhance communication, it becomes beneficial if the visual cues from the mentor can be rendered directly in the vicinity of the operative field.This would avoid the mentee (operative surgeon) to divert the focus from the operative field towards the 2D screens during the surgical procedure.Recently, holographic rendering using a Head Mounted Display (HMD) device has been gaining popularity in medical applications.This is due to its realistic 3D rendering of the virtual models, ability to perceive depth and spatial relationship, and touch-free interfaces usable in sterile environments [26].Static annotations (including geometric primitives and surgical instruments) rendered as holograms has shown improvement during surgical telementoring (e.g., leg fasciotomy [27,28], cricothyrotomy [29], or suturing tasks [30]).
While static annotations are useful, dynamic holograms can effectively convey complex surgical steps.It can enhance the understanding as it enables the mentee to grasp the temporal aspects of the information conveyed by the mentor.Studies conducted by Nagayo et al. [31,32], although not focused on telementoring, demonstrated the use of pre-recorded hand gestures and surgical instrument motion rendered as dynamic holograms in learning open surgery suturing techniques.In a similar study, Liu et al. [33] demonstrated the use of dynamic holograms for teaching scalpel placement during fasciotomy.However, the telementoring setup required the same tissue models at both the mentor and the mentee sites, limiting it to training scenarios only.In this work, we incorporate the usage of dynamic holograms as visual cues within the proposed surgical telementoring system.These visual cues can be generated in real-time by the mentor, directly overlaid onto the operative field, and doesn't require identical setup at the mentor's and the mentee's sites.

Telementoring systems
The telementoring systems used in open surgeries can be broadly categorized into two groups: (a) Augmented Reality (AR) based systems and (b) Mixed Reality (MR) based systems.The system introduced in this work belongs to latter group.In following paragraphs, we compare the benefits of using our proposed system over existing AR and MR-based systems.
AR-based telementoring system, such as Rods & Cones [34], Proximie [35][36][37][38], and STAR [23,27,[39][40][41][42], project visual cues from the mentor onto a 2D screen.This lacks depth perception and necessitates the mentee to shift the focus away from the operative field towards the screen.The proposed MR-based telementoring system addresses this limitation by rendering visual cues within the vicinity of the operative field.This assists the mentee to maintain focus and improve depth perception.
Examples of MR-based telementoring systems include RISP [43] and ARTEMIS [44].The RISP [43] system is confined to displaying static holographic cues.Whereas the proposed MR-based telementoring system enables utilization of dynamic holographic cues, which enhances the learning experience of the mentee.ARTEMIS [44], although capable of displaying dynamic visual cues, requires a complex setup involving optical tracking cameras within the operating room for continuous hardware tracking.This is because the system was primarily designed for trauma surgeries with ample space.However, this might not be practical in traditional open surgery settings where the operative field is surrounded by surgeons, potentially causing line-of-sight issues.In contrast, the proposed system can be easily set up in a traditional operating room and does not require intricate calibration steps.Furthermore, while the ARTEMIS [44] system was only evaluated through a qualitative observational study (lacking detailed technical performance or system accuracy), our work includes a comprehensive study, including quantitative performance of the proposed system in various settings.
While the existing AR or MR-based systems work in a single setting, the proposed system offers flexibility in configurations, allowing adjustments based on (a) the desired depth of the operative field that the mentee needs to visualize, (b) the necessary resolution of the operative field image required by the mentee, and (c) the need for a synchronization between the depth and the image data, while visualizing the operative field at the mentee's end.This adaptability is vital for accommodating different types of open surgeries and optimizing the performance of the telementoring system based on available network connectivity.Thus, the novelty of the proposed system, in addition to rendering dynamic holographic cues, lies in its capacity to be customized to suit the specific needs of open surgeries, and to be integrated in the operating room.Aligned with the aforementioned features, this work introduces a configurable telementoring system that enables the mentor to demonstrate in real-time the required motion of surgical instruments as dynamic holograms overlaid onto the operative field.The contributions of this work include: (a) introducing an architecture of a configurable telementoring system and corresponding algorithms for processing in realtime the information exchanged between the operating room and the remote site, (b) evaluating the performance of the architecture to operate under various network conditions and configurations tailored for different types of open surgeries, (c) demonstrating the ability of the proposed telementoring system to annotate an operative field and illustrate the movements of surgical instruments in comparison to existing systems, (d) showcasing the effectiveness of holographic visual cues rendered by the proposed telementoring system as compared to 2D augmented visual cues.

Materials and methods
The setup of the telementoring system in the operating room and at the remote location is outlined in Section 3.1.In Section 3.2, an overview of the surgical workflow followed by the mentee and the mentor is presented.Further insights into the system architecture, as well as the hardware and software employed for implementation, are provided in Sections 3.3 and 3.4, respectively.

Surgical setup
The proposed telementoring system comprises of two setups, one in the operating room for the mentee (Fig. 1) and the other at the remote location for the mentor (Fig. 2).The operating room setup (shown in Fig. 1a) consists of display screens, an HMD device (HoloLens 2 -Microsoft) worn by the mentee, a speaker/microphone, and an RGB-D camera (Azure Kinect -Microsoft) over the operative field to acquire color (RGB) and depth (D) data streams, connected to a workstation (2.4 GHz processor, 64 GB RAM, Intel UHD Graphics 630 GPU, Windows Operating System).Similarly, the remote location setup (shown in Fig. 2a) consists of a display screen, user interfaces (Touch™ haptic devices -3D Systems), and a speaker/microphone connected to a workstation (2.4 GHz processor, 64 GB RAM, Intel UHD Graphics 630 GPU, Windows Operating System).To facilitate communication between the two setups, the workstations are connected over a network.The view of the operative field in the operating room is acquired using the RGB-D camera (Fig. 1a).This visual information is transferred over the network and rendered to the mentor on the display screen.The mentor visualizes view of the operative field as two-dimensional image (Fig. 2b) as well as a three-dimensional point cloud (Fig. 2c).The view of the point cloud can be rotated, zoomed, and panned in threedimensional space.This aids the mentor to assess the depth of the space surrounding the operative field.Along with the operative field view, virtual surgical instruments are also rendered, allowing the mentor to control their movements using the user interfaces.These movements are sent back to the operating room over the network.The movements act as a visual cue to the mentee and is rendered on both the display screens as well as the HMD device.Inside the HMD device, movements of the virtual surgical instruments are rendered as dynamic holograms.These holograms can be visualized by the mentee from different perspectives (Fig. 1b and Fig. 1c) and assist in comprehending the spatial dynamics essential for the interaction between the surgical instrument and the tissue within the operative field.

Surgical workflow
The telementoring system for open surgery operates via two concurrent surgical workflows; one governed by the mentee in the operating room and the other by the mentor at the remote location (Fig. 3).At the start of the open surgery, the workstation at the remote location establishes a network connection with the operating room workstation.As the surgery is performed by the mentee in the operating room, the mentor observes the procedure in real-time on a display screen at the remote location.If guidance is required during the surgery, the mentee requests the mentor's involvement using audio communication.In response, the mentor selects appropriate virtual surgical instruments and positions them over the live view of the operative field using the depth map.Then, the mentor demonstrates the required motion for the surgical instruments to operate on the tissue.This demonstration is carried out by manipulating virtual surgical instruments through user interfaces.The motion of these virtual surgical instruments is then transferred back over the network to the mentee and rendered on both the display screen and the HMD device.It acts as a visual cue and assists the mentee to perform the necessary surgical sub-steps during the open surgery using the telementoring system.Following the completion of the surgery, the network connection is disconnected, and the workstations in both the operating room and remote location are shut-down.

System architecture
Fig. 4 depicts the architecture of the telementoring system at the operating room and the remote location.A multi-threaded architecture was implemented with different threads responsible for interfacing with the hardware units, processing the acquired data, and transferring it over the network.The data exchanged by the threads is presented in Table 1 and the role of each thread is described in Table 2.The core processing thread acts as a central core for processing data received from different threads.The algorithm running on the core processing thread at the operating room and the remote location is presented in Fig. 5.
The selection of a raw mode versus an augmented mode would be dependent upon the type of open surgery performed.When the operative field is relatively stable (for example, in regions where motion caused by breathing and/or a beating heart in absent), the raw mode would be preferred over the augmented mode.The Raw mode has lower latencies and does not necessitate the synchronization between depth and video frames.Both compressed depth and video frames are transmitted separately and received asynchronously.Examples would  include suturing [10,31], leg fasciotomy [16], and craniotomy [20].Conversely, when the operative field is highly dynamic, the augmented mode would be preferred to accurately portray the changes to the mentor.Compressed depth frame is integrated to metadata of video frame during transmission over network, ensuring synchronization upon reception at the remote location.This includes scenarios such as chest thoracotomy [45] and laparotomy [12].Similarly, the selection of size of the depth map (8 bits versus 16 bits) would determine the depth of the operative field to be acquired during the open surgery.For example, in some open surgeries, including cleft lip repair [18], cricothyroidotomy [44], and skin grafting [33], a depth map of size 8 bits would be sufficient, whereas for other open surgeries, such as cholecystectomy [13] and myomectomy [46], a depth map of size 16 bits would be required.

Implementation details
The modules of the telementoring system were implemented as taskdedicated threads in C++.Qt (Qt Company) was used for graphical user interface, whereas VTK library (Kitware) was used for graphical renderings on the display screens.The holographic rendering on HMD device was performed using Unity 3D (Unity Technologies).The pose of the RGB-D camera with respect to the HMD device was detected using Vuforia SDK (Vuforia).Encoding/decoding of the video was performed using FFMPEG.The data from the user interfaces at the remote location were processed using OpenHaptics library (3DSystems).Integration and calibration of the RGB-D camera were achieved using the Azure Kinect Sensor SDK (Microsoft).
The network data module, network video module, and audio module were implemented using Web Real Time Communication (WebRTC) protocol.WebRTC enforces end-to-end encryption between the workstations and is in-built at the protocol layer.A direct peer-to-peer connection was established between the workstations to initiate communication.A Simple Traversal of User Datagram Protocol through Network Address Translators (STUN) server at stun1.google.com was used to discover the public IPs (Internet Protocol) addresses, and then a signaling server hosted on DigitalOcean.comwas used to exchange the public IPs along with audio-video formats used by the network modules.To avoid eavesdropping, a signaling layer was implemented over Hypertext Transfer Protocol Secure (HTTPS) protocol.

Evaluation of telementoring system
The developed open surgery telementoring system was evaluated for both technical performance and usability.Four distinct experimental studies were conducted to measure (i) network performance (described in Section 4.1), (ii) accuracy in demonstrating the mentor's annotations on the operative field (described in Section 4.2), (iii) ability of the mentee to follow the surgical instrument motion demonstrated by the mentor (described in Section 4.3), and (iv) visual guidance rendered to the mentee (described in Section 4.4).The latter three experimental studies involved subjects selected as mentor-mentee pairs (n = 6 for the second study, n = 6 for the third study, and n = 8 for the fourth study).The subjects were researchers from the Department of Surgery, Hamad General Hospital, Qatar with exposure to basic surgical skills.In addition, the subjects were familiar with using Touch haptic device (by 3D Systems, USA) to maneuver a 3D cursor, and HoloLens (Microsoft, USA) to visualize and interact with the holograms in mixed reality environment.Ethical approvals (approval number MRC-01-20-087) were obtained from institutional review board (Medical Research Center, Doha,  Qatar).

Network performance
The experimental setup consisted of two workstations, replicating the operating room and the remote location setup, connected over the Internet.The clocks on both the workstations were synchronized using Network Time Protocol (NTP) server 216.239.35.4 (time2.google.com).The time difference between the NTP server and the workstations (due to asymmetric routes and network congestion) was also measured and added as a clock drift to the recorded timestamps of the data packets.The workstations were connected using a wireless network with an average network throughput of 900 Mbps.Data packets sent and received between the two workstations were recorded and used to measure: (i) the average delay in transferring data (video frame, depth frame, surgical instrument poses) from the sender to the receiver over the network, (ii) distortion in the quality of video frames caused due to encoding at the sender and decoding at the receiver end, (iii) the percentage of dropped video frames at the receiver end, and (iv) video and depth frame transfer rates.The sizes of each video frame, depth frame, and surgical instrument pose are shown in Table 3.In raw mode, video frames and depth frames are transmitted through separate channels, with each video frame and 12 bytes of metadata in the depth frames (out of 28 bytes) sharing common 12 bytes of metadata.
The system was evaluated by configuring four different parameters: (i) resolution of the video frame (HD / Full HD), (ii) size of the depth map (8 bit / 16 bit), (iii) technique used for transfer of depth data (raw mode / augmented mode), and (iv) location of the operating room and the remote location (inter-country / intra-country).As each parameter has two discrete states, the experiment resulted in sixteen different configurations.Performance of the system under each configuration was evaluated for multiple trials (n = 3), where each trial lasted for a duration of 10 min.At the start of the trial, a network connection was established between the two workstations and recording of the data packets was initiated.The inter-country setup consisted of the operating room workstation situated in Doha, Qatar and the remote location workstation situated in Houston, Texas, USA.Whereas in intra-country setup, both the operating room and the remote location workstations were situated in Doha, Qatar.During the trial, the mentee at the operating room workstation streamed the view of operative field using video and depth frames, whereas the mentor at the remote workstation streamed the motion of the virtual surgical instruments.As the experimental setup emphasized on measuring the network performance, no specific surgical task was selected.However, the mentor was asked to follow the boundaries of rectangular sheet of paper (210 mm × 297 mm) every 10 s to generate the data packets for surgical instrument motion.The mentee followed the motion using a real surgical instrument to ensure the view of the operative field changes.In addition, audio communication was also enabled.The trial ended after 10 min and recording of the data packets was stopped.

Annotation of operative field
A user study was conducted between mentor-mentee pairs (n = 6) to evaluate the accuracy of the system in demonstrating the mentor's  instructions onto the operative field.The experimental setup for the study is shown in Fig. 6a.A rectangular grid (150 mm × 220 mm) was used as an operative field (Fig. 6b) and comprised of ten color-coded targets (10 mm × 10 mm).The study involved the task of annotating the ten targets sequentially.During the task, the mentor was asked to place the virtual surgical instrument tooltip at the center of each target (Fig. 6c).The mentee at the operating room then used a pencil to mark the tip of the holographic surgical instrument as viewed through the HoloLens (Fig. 6d).After each mentor-mentee pair performed the task, the distance between the pencil mark and the center of each target was measured to determine the accuracy of the system for placing the mentor's instrument onto the operative field.

Demonstration of surgical instrument motion
The user study was conducted to assess the usage of the proposed open surgery telementoring system for the mentor to generate instructions in the form of virtual surgical instrument motion, and for the Fig. 5. Core processing thread algorithms running at operating room and remote location.mentee to understand and replicate the instructions provided by the mentor.The experimental setup at the operating room is shown in Fig. 7a.In the study, both the mentor and the mentee were asked to perform a task simultaneously.At the remote location workstation, a predefined guidance path in 3D space was rendered along with the view of the operative field view.The mentor was asked to follow the guidance path from start to end using the tooltip of the virtual surgical instrument augmented onto the operative field (Fig. 7b).The motion of the virtual surgical instrument, excluding the guidance path, was transmitted to the operating room workstation, and displayed as a dynamic hologram to the mentee.At the same time when the mentor was moving the virtual surgical instrument, the mentee was asked to follow the tooltip motion of virtual surgical instrument using a real surgical instrument (Fig. 7c).A retroreflective spherical marker was attached to the tooltip of the real surgical instrument to track its motion using an optical tracking system (V120: Trio OptiTrack -NaturalPoint).Optical markers, as per our prior research [47], were affixed to the HoloLens to facilitate its tracking relative to the optical tracking system.This process enables the registration of the HoloLens, and consequently the RGB-D camera, within the coordinate system of the optical tracking system.Each mentor-mentee pair performed the task for three different guidance paths (Fig. 7d).
During the task, the trajectories of the paths followed by both the mentor and the mentee were recorded and processed to compute following parameters: (i) the average distance between two paths measured using Dynamic Time Warping (DTW) distance divided by the number of sample points on the paths in 2D and 3D space, and (ii) the average duration taken by the mentor-mentee pair to follow the entire path.Paths in 2D space were created by projecting the 3D paths onto a 2D viewing plane parallel to the HMD device's camera frustum.DTW was used to enhance the robustness of similarity computation between the two paths [47].The method enables comparison of time series with varying lengths.Instead of Euclidean distance, the method substitutes the one-to-one point comparison with a many-to-one (and vice versa) comparison.These parameters had been utilized in previous studies for assessing surgical skills [48] and demonstrating surgical tool motion [49,50].

Visual guidance
During telementoring, the visual guidance provided by the mentor to the mentee is rendered using two visualization modes: (a) '2D Augmented Display Mode' utilizing a 2D display screen to render the augmented video stream to the mentee, and (b) 'Holographic Display Mode' utilizing a HMD device to render the holographic data stream to the mentee.A user study was conducted to subjectively assess the acceptance of the two visualization modes.The subjects were randomly assigned to two groups, one for each visualization mode.During the study, the mentor demonstrated movements of virtual surgical instruments required to traverse different landmarks (Fig. 8a) and throw a knot (Fig. 8b).A questionnaire (based on previous work [51][52][53]) on a 5-point Likert scale was used to record the mentee's perception of the visualization mode.The used questionnaire is shown in Fig. 9.The study was then repeated by switching each subject to the other visualization mode.

Network performance
The network performances of the developed open surgery telementoring system for intra-country and inter-country communication are presented in Tables 4 and Table 5, respectively.The latency of transferring video frames over the network increased as the resolution of the video frames progressed from HD to Full HD.Furthermore, in the augmented mode, where depth frames were combined with video frames, the latency was higher compared to the raw mode.Additionally, when the two workstations were located in different countries (intercountry) as opposed to the same country (intra-country), there was a further increase in latency.The latency ranged from 67 ± 3 ms for intracountry HD video streams in the raw mode to 307 ± 12 ms for intercountry Full HD video streams in the augmented mode.However, all latencies remained within the recommended limit of 450 ms set by Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) for live telementoring [54].This indicates that the open surgery telementoring system was suitable for real-time usage.
Similar to video frame transfers, the latencies for depth frames also increased when transitioning from HD to Full HD resolution in the raw mode.However, when it came to 16-bit depth maps in the raw mode, the latencies for inter-country transfer of depth frames surpassed the recommended limit of 450 ms.This occurred due to insufficient depth data compression, leading to larger packet sizes and subsequent network The average delays in transmitting 16-bit depth frames in the raw mode were higher than the augmented mode.It is noteworthy that in the augmented mode, video and depth data are combined and transmitted using video channel Real-time Transport Protocol (RTP).RTP is optimized for low latency and real time delivery of data without The mentor demonstrates the mentee the placement of a suture (time instant t 1 to t 2 ) and then the mentee follows the mentor's instruction (time instant t 3 to t 4 ).guaranteeing the reliable delivery of data packet at the other end.This makes it faster.In contrast, in the case of the raw mode, the video and depth data are sent separately.While video still uses RTP, the depth uses Stream Control Transmission Protocol (SCTP).SCTP provides reliable, ordered, and flow-controlled delivery of data streams.In this case, if the packet is loss, it will try to retransmit the data.Thus, SCTP have higher overhead as compared to RTP due to its additional features for reliability and error recovery.This adds more overhead when the depth data size is doubled in the case of 16-bit.
In the case of latencies for transferring tool motion, they were independent of video frame resolutions and the sizes of depth maps.These latencies ranged from 10 ± 1 ms to 20 ± 7 ms for intra-country and from 135 ± 14 ms to 149 ± 1 ms for inter-country.These latencies remained within the range of 200 ms recommended by Xu et al. [55] to demonstrate surgical tool motion.
To evaluate the degradation in the quality of video frames caused by encoding and decoding over the network, standard video quality metrics were used [50,56].The metrics included Mean Square Error (MSE), Peak Signal-to-Noise Ratio (PSNR), and Structure Similarity Index Measure (SSIM).No significant variations in distortion values were observed across different modes, video frame resolutions, and sizes of depth maps.This consistency was evident in both intra-country and inter-country trials.Among the metrics, SSIM proved to be more closely related to the perceptual quality of the human visual system, as it compares luminance, contrast, and structure to model image distortion [57].In contrast, PSNR and MSE lack the ability to discern structural content in the video frames.The average SSIM score of 0.80 indicated that the mentor could effectively comprehend the operative field throughout the experiment, with no discernible loss of video frame quality during compression and transmission.
It should be noted that the occurrence of dropped data packets remained minimal throughout the experiments, with rates ranging from 0.003 % to 0.7 % for both intra-country and inter-country trials.This low level of packet drops ensured consistent and uninterrupted communication between the operating room and the remote location.

Annotation of operative field
The study recorded an average annotation error of 3.92 ± 2.1 mm, which demonstrated an improvement over the accuracy errors reported by existing surgical telementoring systems utilized in open surgeries [28][29][30] (summarized in Table 6).Thus, it is feasible to perform static annotations using the telementoring system.These static annotations can be leveraged by the mentor to identify areas of interest in the surgical field, such as anatomical structures, incisions points, suturing lines, and dissecting boundaries.

Demonstration of surgical instrument motion
The results from the user study are summarized in Table 7.The mentor followed the predefined guidance path with an average distance of 3.9 ± 1.8 mm in 2D space and 13.12 ± 11 mm in 3D space.The mentee successfully followed the mentor with an average distance of 8.6 ± 2.1 mm in 2D space and 12.8 ± 3 mm in 3D space.Kruskal-Wallis test was performed to determine whether the recorded distances (both in 2D and 3D) and duration were statistically different among the three paths.The null hypothesis was accepted, showing no statistical difference.Thus, no significant differences were observed among the recorded parameters for the three paths.This shows irrespective of the trajectory of the surgical instrument motion (to be demonstrated for a surgical substep), the telementoring system enables: (a) the mentor to demonstrate the required motion of the surgical instrument over the operative field, and (b) the mentee to understand the motion and replicate it in real-time.The paths traversed in 2D were derived by projecting the paths travelled in 3D space onto a 2D viewing plane parallel to the camera's frustrum.In this process, the recorded average distances decreased in 2D space because the depth along the observer's viewing direction was not factored into the calculations.The improved average distances observed in 2D compared to 3D reflect the effect of depth perception while following surgical instrument motion using the proposed system.

Visual guidance
The distribution of the responses collected from the mentees on a Likert scale is shown in Fig. 9.In general, feedback from the subjects on using the holographic display mode was positive, particularly due to the ability to observe the visual guidance overlaid onto the operative field from different viewpoints, and the realism of the learning experience.The ability to perceive depth was highly appreciated by subjects, enabling them to easily understand and replicate surgical tool motion.The subjects were able to accurately align the real and holographic tools in the 3D space over the tissue phantoms simulating the operative field.Most of the subjects agreed that the system has the potential to replace traditional in-person guidance.They highlighted the system's capability to depict tool motion as though the mentor was demonstrating right beside them.However, some subjects also reported challenges while using the holographic display mode.This included limited field-of-view, disturbances from reflections on the HMD device's visor, and discomfort due to the weight of the device.
Under 2D augmented display mode, subjects reported no visual or physical discomfort, but rated unfavorably on the ease of understanding surgical instrument motion and user-friendliness of the system.As the visual guidance was limited to the view acquired by an RGB-D camera situated above the operative field, it hindered the understanding of the surgical instrument motion in 3D space.Depth perception was only intuitive when the size of the tool changed as it moved closer or farther from the RGB-D camera.The surgical instrument movements along a plane parallel to the RGB-D camera's viewing frustrum were not easily perceptible.Most of the subjects noted difficulty in having to constantly switch gaze between the operative field and the 2D display screen, resulting in higher cognitive workload.Due to these limitations, subjects reported difficultly with using 2D augmented display mode in place of traditional in-person guidance.

Limitations and future work
The proposed study encounters several limitations.First, the telementoring system was evaluated within a controlled environment using anthropomorphic phantoms.Although these studies demonstrated the technical feasibility of the tele-mentoring system, the subsequent phase would entail assessing its clinical utility in operating room settings through animal studies.In addition to transmitting digital information, it is essential to establish standardized lexicons and protocols [58,59] to ensure effective and seamless communication between the mentor and the mentee [60][61][62][63].Second, clinical studies would be needed to assess the performance on the tele-mentoring system in facilitating the transfer of surgical skills across various open surgery scenarios.These potential scenarios would include (a) training basic surgical skills in a simulation lab, where an experienced instructor showcases the surgical techniques to a trainee, (b) assessing surgical skills of trainees in a controlled educational environment, where the instructor controls virtual surgical instruments and virtually demonstrates the intended steps over the operative field, and (c) transferring skills to execute a new or intricate surgical procedure, where a specialist surgeon illustrates the surgical steps for the operating surgeon to follow in the operating room.In such studies, the performance would be measured based on task completion time, surgical proficiency metric (such as OSATS) and human factors (such as NASA-TLX) [64].
In its current implementation, the telementoring system has two limitations.First, the system is confined to the field-of-view of the RGB-D camera and a depth range of up to 36 cm supported by the depth map.These limitations necessitate careful consideration and may affect the system's applicability in surgical scenarios.It was observed that the usage of a single RGB-D camera was insufficient to acquire surgical instruments.The limited thickness and reflective metallic surfaces of the surgical instruments cause interference with the infrared rays from the depth camera, distorting the acquired point cloud, especially near instruments' distal part.That is, the surgical instrument's distal part appears on the operative field instead of being over the operative field along with surgeon's hand holding them.This discrepancy gets reflected when the point cloud is rendered in a Virtual Reality (VR) environment.As a results, in the current implementation of the system, a 2D display was used to render the operative field to the mentor instead of VR.To address this limitation, multiple RGB-D cameras can be deployed around the operative field to acquire point cloud data from different viewpoints [65,66].By stitching these point clouds together, a high-resolution representation of the 3D space in form of a dynamic mesh (without occlusions) can be generated [67].When integrated into VR-based HMD device at the remote location, the 3D dynamic mesh would provide an immersive experience to the mentor.It will enhance the rendering of the operative field and provide the mentor with a better understanding of the depth to place and control the virtual surgical instruments on the operative field [68].Second, the accuracy of the proposed tele-mentoring system heavily relies on the precision of the Vuforia engine used to synchronize the coordinate system between the RGB-D camera and the HMD device.To enhance accuracy, employing multiple ArUco markers around the operative field could be considered.These markers would remain continuously visible under both the RGB-D camera and the HMD device, creating a common reference frame for continuous registration of these devices throughout the procedure.

Conclusion
The developed telementoring system enables a mentor (who could be remotely located) to demonstrate the requisite movements of the surgical instrument over the operative field.In parallel, it enables a mentee to grasp these demonstrated motions and replicate them.This fosters an effective real-time communication during an open surgery.The holographic visual cues were appreciated by the mentees for their ability to be observed from different viewpoints, facilitating depth perception, and enhancing the realism of the learning experience.The ability of the telementoring system to be configured for different types of open surgeries and optimized based on available network connectivity, further enhances its usability.

Declaration of competing interest
The authors of this submission have no affiliations with or involvement in any organization or entity with any financial interest (such as honoraria; educational grants; participation in speakers' bureaus; membership, employment, consultancies, stock ownership, or other equity interest; and expert testimony or patent-licensing arrangements), or non-financial interest (such as personal or professional relationships, affiliations, knowledge or beliefs) in the subject matter or materials discussed in this manuscript.

Fig. 1 .
Fig. 1.(a) Operating room setup of the telementoring system for open surgery.(b) The mentee wears an HMD device and visualizes the operative field from different perspectives.(c) The holograms of the virtual surgical instruments seen by the mentee using the HMD device from different perspectives.

Fig. 2 .
Fig. 2. (a) Remote location setup of the telementoring system for open surgery.(b) Two-dimensional view of the live operative field with virtual surgical instruments augmented onto it.(c) Three-dimensional depth map of the live operative field along with virtual surgical instruments.

Fig. 3 .
Fig.3.The workflow followed by both the mentee and the mentor while utilizing the telementoring system for open surgery.The workstations are connected over a network and enable communication between mentor and mentee.

Fig. 4 .
Fig. 4. Architecture of the proposed telementoring system for open surgeries.

Fig. 6 .
Fig. 6.(a) Experiment setup used at the operating room for annotation task.(b) Rectangular grid used as an operative field for annotation task.(c) View of the operative field seen by the mentor on the display screen at remote location.(d) View of the operative field seen by the mentee through the HMD device.

Fig. 7 .
Fig. 7. (a) Experiment setup used at the operating room for task related to demonstration of surgical instrument motion.Panel A shows the arrangement of optical markers on HoloLens.(b) View of the operative field seen by the mentor on the display screen at remote location.(c) View of the operative field seen by the mentee through the HMD device.(d) The three guidance paths used in the study.The guidance paths were drawn on a plane.

Fig. 8 .
Fig. 8. (a) The mentee continuously follows the mentor's instructions (time instant t 1 to t 4 ) to traverse different landmarks.(b)The mentor demonstrates the mentee the placement of a suture (time instant t 1 to t 2 ) and then the mentee follows the mentor's instruction (time instant t 3 to t 4 ).

Fig. 9 .
Fig. 9. Percentage distribution of the responses from the mentees to the questionnaire on visualization modes.

Table 1
Data processed and shared by telementoring system.
DataDescription of the processed data F Depth (t) A depth frame representing of the operating field in the form of a point cloud at time instance 't'.F ʹ Depth (t) A compressed depth frame at time instant 't' for easier transfer over the network.F Video (t) An RGB video frame of the operating field at time instance 't'.M Instuments (t) A set containing 4 × 4 affine transformation matrices of surgical instrument poses at time instance 't'.M Interfaces (t) A set containing left and right 4 × 4 affine transformation matrices of user interface poses and clutch status.M. Anabtawi et al.Computer Methods and Programs in Biomedicine 256 108396

Table 2
Threads used in the architecture and their functioning.
Depth (t) over the network.At operating room, the thread sends F Depth (t) and receives M Instuments (t) from the network.At remote

Table 3
Video frame, depth frame, surgical instrument pose data sizes.

Table 4
Intra-country network performance of the proposed open surgery telementoring system.

Table 5
Inter-country network performance of the proposed open surgery telementoring system.

Table 6
Comparison of annotation accuracy of open surgery telementoring systems.

Table 7
Comparison of the paths generated by the motion of surgical instrument's tip.