Solène Lambert
Ignacio Avellino
Abstract
Surgery is primarily taught through mentoring, where an expert mentor supervises a mentee performing surgery, taking over when necessary. Telementoring systems aim to provide mentees with access to remote mentors, but the physical distance between mentors and mentees poses unique challenges to surgical training. We investigate the underlying needs leading to takeovers in onsite mentoring and assess mentors’ ability to fulfill address these needs remotely using existing telestration tools, namely pointers and drawings on shared views. Through interviews and workshops with expert surgeons, we find that (1) mentors take over to convey gestures related to instrument placement, tissue displacement, force, and movement, (2) mentors gather information about location of tissue, equipment, and instruments, as well as gesture constraints, and (3) surgeons judge telestration insufficient for these needs. Based on this gap between onsite mentoring practices and telementoring tools, we discuss novel tools to address these needs and their evaluation.
1 INTRODUCTION
Surgeons learn new surgical techniques predominantly through mentoring, where a mentee performs surgery while a mentor supervises actions and takes over when necessary to instruct, for instance showing how to physically perform a gesture [5, 7]. When seeking to learn a new technique, surgeons can have difficulties accessing experts as these often reside in a different hospital, city or country. Therefore, telementoring emerges to bridge the distance between mentees and remote mentors. Current telementoring systems present mentors with the surgical view—a streamed view of the endoscopic video—along with telestration tools on the view, namely the ability to point and annotate. Telestration has been shown beneficial to help train the professional vision [15], reduce time spent on instructions, and reduce miscommunications [4]. Still, telementoring is limited because it lacks physicality [2], thus mentors cannot act on the body nor shape what is captured on the surgical view [38]. Today, it remains unclear how the lack of physicality limits remote mentoring, even when having access to telestration. Answering this question requires first understanding the intentions behind takeovers in collocated settings—that is, for what purposes mentors take over the instruments of the mentee.
Previous work studying onsite mentoring have categorized the information conveyed through verbal directional instructions [18], as well as mentees’ learning goals and outcomes [10, 43], although without focusing on the role of takeovers in the process. Literature has also highlighted the importance of physical action, for instance when building process common ground [17], with one piece of work proposing a categorization of physical guidance [51] although without distinguishing taking over as a way to demonstrate, from deictic and figurative gestures to explain, or from body manipulations to assist. Nonetheless, studying takeovers can elicit the underlying needs during mentoring, by shedding light on communication challenges during surgical mentoring. This is because takeovers incur an additional cost of handing over instruments to another person, which in some cases can be lower than the cost of using other means of communication (e.g., speech or gestures), as grounding conversations is usually done with as little combined effort as possible [9].
We investigate (1) the informational needs mentors fulfill through taking over in onsite mentoring, and, (2) to what extent mentors can address these needs remotely through current telestration tools. We narrow our study to laparoscopic surgery, considering that telementoring systems would be predominantly employed in this type of surgery, where the surgical view is captured on video, and can thus be easily streamed. In total, 6 expert surgeons and 1 resident participated in this study, taking part in interviews and workshops. We show that telestration and speech remain limited in conveying placement when there is no visible reference (both for instruments and tissues), simultaneous location, force and movement. Also, we show that that telestration and speech could partially support gathering information, such as seeking location information, but remain highly limited in gathering information on textures, camera orientation, gestures constraints and adequacy. Finally, we discuss tools that extend beyond telestration, presenting participants’ vision of how future tools could support their mentoring practices remotely, including novel pointers, virtual instruments, additional videos for the mentee and for the mentor, simulations, anatomical charts, physical proxies, tools to measure and represent force and improved surgical view.
Given these results, we discuss the design and evaluation of current remote collaboration systems, highlighting that telementoring tools are rarely evaluated on their ability to support mentors in conveying a wide variety of information. We discuss the potential and feasibility of tools proposed by participants, revealing that many of them exist for individual learning (i.e., instructional video) or support to the operating surgeon (i.e., global vision, force feedback) but their potential for mentoring remains unexplored.
2 SURGICAL CONTEXT
Performing surgery necessitates four hands, where one surgeon takes the role of the main operator, while another assists. Surgery thus involved parallel use of multiple tools with high coordination. During mentoring, mentors can take the role of the assistant to let mentees gain experience, taking over when necessary. This is a classic configuration in Minimally-Invasive Surgery (MIS), the type of surgery we focus on in this paper, where instruments and a camera are inserted into the patient through trocars (i.e., tubes placed through small incisions on the body). The camera, called endoscope, captures live video from inside the patient in what constitutes the surgical view, observable through a monitor in the Operating Room (OR). When MIS is performed in the abdominal and pelvic areas, in particular for urology, digestive or gynecologic surgery, it is referred to as laparoscopic surgery.
A common mentoring scenario involves the mentor performing exposure while the mentee performs dissection. To perform exposure, the mentor holds an organ with one hand, for instance using pliers, while driving the endoscope with the other, to show an unobstructed view of the surgical site—i.e., the center of the intervention. It is important to hold the endoscope horizontally, so that the anatomy shows as expected on the screen, given that the camera shaft can be rotated freely about its axis. To perform dissection, the mentee holds scissors with one hand and bipolar pliers with the other, to loosen tissue around an organ and “present the organs in the isolating style of an anatomic atlas [to] show neatly separated organs” [27]. Tissues are dissected to open planes, creating spaces between tissues (for instance bladder tissues and vagina tissues) that previously did not exist because they were “glued” by connective tissue. Dissection can detach organs, making them mobile organs, where only a few attachments are left (e.g., vessels or ligaments); it is a delicate task because it involves cutting tissue around noble structures, including arteries and structures such as the ureter, which, if wounded, can lead to an emergency and post-operative complications. While most gestures are performed with instruments inserted through trocar, the assistant can insert instruments (i.e., valve, dilator) through natural orifices (i.e., vagina, rectum) for instance to expose those tissues, that is, make them visible to the main operator.
3 RELATED WORK
We first review previous research on surgical instruction, followed by technological tools designed to support such instructions.
3.1 Building Expertise With the Help of a Mentor
Prior research has identified some of the components of expertise, as well as the process through which the mentor helps the mentee acquire expertise.
What are the components of expertise that surgeons need to acquire? Many of the skills needed to become a surgeon are acquired in the OR, including factual knowledge, as well as non-technical and technical skills. Factual knowledge includes knowledge of instruments and anatomy, especially important for junior learners [10]. Non-technical skills include situational awareness, communication, teamwork and managerial skills [10, 43], whereas technical skills, involve understanding the flow of an intervention, dealing with anatomical variants and complications, making sense of visual and haptic cues, as well as motor skills (e.g., handling instrument, exposure, and responding to different tissues) [10, 43]. Different metrics can be used to understand when a surgeon has achieved proficiency. The Global Operative Assessment of Laparoscopic Skills (GOALS) uses 5 criteria: depth perception, bi-manual dexterity, efficiency, tissues handling and autonomy [54]. Oropesa et al. [41] give an overview of metrics used in the literature, both concerning efficiency and quality. Efficiency metrics include time, path length, economy of movements, economy of diathermy (burn time), speed, motion smoothness, instrument orientation, depth, angular path, angular area, volume and force. Quality metrics consist of outcome, errors, idle state, tasks repetition, and collision/tissues damage. Other works have proposed to objectively assess complex but critical aspects of surgery, such as quality of exposure using machine learning methods [11]. Although these works focus on what skills and knowledge surgeons learn, they do not fully draw a picture of how mentors transmit them.
How does the mentor’s physical intervention support the mentee in acquiring expertise? To acquire expertise, mentees rely on mentors. Numerous studies give insights on the information transmitted through physical interventions, even though it is not their primary focus of study. Feng and Mentis [17] observed that physical actions tend to develop process common ground, whereas speech tends to develop content common ground. Chen et al. [7] classify mentor support into three categories: directing (i.e., providing guidance as the mentee operates), assisting (i.e., performing secondary tasks for the mentee) and teaching (i.e., physically showing how to do an action) which is similar to the concept of modeling in the theory of cognitive apprenticeship [5], and corresponds to what we refer as takeovers. Regarding how mentors instruct, Mondada’s [39] vast observations of surgical learning show that verbal instruction rarely occurs without accompanying embodied instructions, which involve reorienting the hand of the assistant, grasping organs, or pointing with an instrument. Feng et al. [18] also noted the use of deictic instructions when mentors gesture on endoscopic video to indicate where to look at—i.e., gaze guidance. They also show how verbal directional instructions can direct instruments towards the correct spot, guide camera maneuvering, or coordinate the use of multiple instruments. When studying how mentors teach mentees to interpret visual cues, Mentis et al. [37] identified that mentors intervened physically through tracting tissue, pointing with instruments, finger pointing on the view, or moving the camera.
These works shed light on physical guidance, but, to our knowledge, only Sutkin et al. [51] intentionally studied this practice. The authors propose categories that describe how mentors execute physical guidance: performing an action to suggest another action (transactional), pointing (deictic), exposing tissues (retracting), physically repositioning instruments (instrument specific), touching hands (hands in hands), figurative mid-air gesture, or, moving the anatomy (complex anatomy). What is interesting in this categorization is that we observe hints toward the purpose of guidance, such as providing detailed information about the upcoming steps, facilitating mentees’ gesture, describing anatomy, or describing instrument motion required in subsequent steps. However, this work does not show specifically the information that mentors convey through takeovers, and how difficult it would be to do so without physical access to the body.
In summary, previous research on surgical instructions has outlined several facets of expertise, and the ways mentors support acquiring expertise. Takeovers serve not only to impart information but also to assist in mentees’ tasks. Nevertheless, when the goal is to convey information, the precise information mentors are able to convey given they are physically present remains unclear. This understanding is crucial for the design of instructive technologies when the mentor is remote and not physically present.
3.2 Technologies for Instruction in Surgery
Both CHI and CSCW have studied how technology can support mentors during instruction in the OR, both in collocated and remote settings. We review literature on tools designed to transmit instruction during physical tasks, tasks “in which one person directly manipulates objects with the guidance of one or more other people, who frequently have greater expertise about the task” [33].
3.2.1 Telestration (Pointers and Annotations) on a Shared View.
The remote gesturing tools that dominate in remote instruction are virtual pointers and annotations on shared views, aimed at supporting object-focused discourse. Studies have demonstrated advantages for communication, as these tools support the production and understanding of pointing and representational gestures during instruction in physical tasks [22]. This leads to a gain in performance, as they support the development of Common Ground (establishing mutual knowledge, belief, attitudes and expectations [9]), restructuring talk with regard to turn-taking and number of spoken words [32].
Much HCI literature on instructional tools on shared views has focused specifically on surgery, both in collocated and remote settings. In collocated settings, where the mentor works side-by-side with the mentee, studies have demonstrated that virtual pointers on imaging systems support mentors in fostering the adoption of a professional vision [16], and can help guide the mentee’s gaze [14]. In remote settings, telestration was shown to improve attention and quality of instructions compared to telestration in onsite settings [47]. It has also been shown to reduce the time needed for instructing [4] compared to instruction without telestration. Lastly, speech and telestration have been shown to support remote mentors in shaping the view, the process through which surgeons modify the view to co-construct knowledge and make decisions [38].
While pointers benefit mentoring, they may be interpreted as the main source of information and lead to misinterpreting instruction by associating a verbal instruction to an incorrect location [46]. Besides, the asymmetrical nature of telementoring, in particular as remote mentors cannot manipulate the video themselves, still limits their ability to shape the view and hinders collaborative work when building a mutual understanding [38]. Nonetheless, a recent systematic literature review concludes that, so far, there are no studies providing evidence that surgical telementoring worsens postoperative outcomes [13].
3.2.2 Beyond Telestration: Overlays and Avatars.
To go beyond telestration, previous works explored the technical feasibility of overlaying either the remote mentor’s hand or their tools on the surgical view. Overlaying mentor’s hands on live video of the surgery is promising when describing complex surgical maneuvers, especially those involving anatomy, trajectory, rotational gestures (e.g., “turn the flap downward with this motion”), and microtechniques [50]. Overlaying instruments controlled by the remote mentor supports instructing instrument rotation in all three dimensions [29] and can help mentees follow with precision a 2D trajectory shown by a mentor [48]. Lastly, the ARTEMIS system showed the feasibility of having an avatar representation of the remote expert displayed in augmented reality in the OR, enabling the mentor to show hand movement, point, and annotate directly on the patient [23].
All in all, the benefits of tools such as telestration and overlays are often evaluated on performance, including completion time [8, 25, 31, 32], errors [8, 25], number of words, frequencies of deixis [22], turn taking [32] or self reported measures on overall experience [22, 25, 31, 47]. However, these studies do not unveil the information mentors can convey through these tools. Only Fussell et al. [22] identified the information conveyed with pointers and drawings on a robot assembly task—referring to objects, indicating location, showing motion, angle of insertion and orientation.
4 METHOD
We address the lack of understanding regarding the limits of telestration during surgical mentoring. We conduct three workshops with surgeons, each involving two participants. In the first workshop, we studied the act of mentors taking over the control of instruments from the mentee, to convey information (i.e., explain). We prepared this first workshop through an online anonymous questionnaire. For the second and third workshop, we expanded our investigation to scenarios where mentors takeover to gather information (i.e., understand the situation). We prepared these workshops by conducting 4 interviews, to create an initial categorization of information that remote mentors convey and gather through takeovers. We then discussed this categorization during the workshops and refined it during the data analysis.
Throughout all workshops, we explored the extent to which telestration can be used to address the needs met by takeovers, and brainstormed other tools for remotely addressing those needs remotely. The use of workshops and interviews enabled us to collect rich detail on the tacit needs leading to takeovers during situated action. As we aimed at covering a wide diversity of cases, including rare needs that lead to takeovers, we opted for methods where participants can narrate their experiences in their own words, as opposed to relying solely on observations. We focus on laparoscopic surgery for various reasons. First, to narrow our focus, as each type of surgery (open surgery, robotic surgery and laparoscopic surgery) may involve different types of instructions. Second, because the surgical view corresponds to the endoscopic video, providing the remote surgeon access to the same surgical view as the mentee. Lastly, we excluded robotic surgery given their high cost and low availability. Although the use of telestration could have also been studied through observations of real use, the reality is that telementoring systems remain rarely adopted, and thus are too scarce for an observational study. Nevertheless, we recruited surgeons that have used telestration tools to instruct in robotic surgery. In this setup, they are effectively distant, operating from separate consoles, even if physically in the same room. Throughout our analysis, we regularly perform member checking. Following our commitment of transparency research, we decided to register this study to establish the motivation, research questions and planned analysis. The pre-registrations of this study are available on the platform OSF (Open Science Framework) following these links: https://osf.io/v76z9, https://osf.io/fxmbw, https://osf.io/guers, https://osf.io/26acw.
4.1 Participants
Six attending surgeons and one senior resident participated in our study, with varying specialties, experiences, and affiliated to two different institutions (Table 1). We focused on specialties that deal with soft-organ surgery (i.e., gynecology, digestive, and urology), as they face similar challenges (e.g., organ deformation), and similar tasks (e.g., cutting tissue, identifying anatomy, or feeling the stiffness of tissue). We recruited experienced surgeons as we focus on telementoring scenarios that can take place within institutions, where mentees are sufficiently independent to pause surgery and wait for a colleague to arrive, and cases of telementoring between institutions, where mentees can complete the surgery autonomously. For the workshops, we recruited participants that had experience with telestration tools, virtual pointers and sometimes drawings, accrued through robotic surgery. None of the participants had used telestration tools before for telementoring purposes in classic MIS. P1 and P7 work closely with the authors and were aware of parallel work we conducted on gesturing tools for remote mentors.
Surgical | Surgical | Participation | Role | If mentor, | |
| P1 | Gynecology | Senior | Questionnaire | Mentee | - |
| P2 | Gynecology | Junior | Questionnaire | Mentee & | Resident |
| P3 | Gynecology | Junior | Questionnaire | Mentee & | Resident |
| P4 | Digestive | Senior | Interview | Mentor | Junior |
| P5 | Urology | Senior | Interview | Mentor | Junior |
| P6 | Gynecology | Senior | Interview | Mentor | Junior |
| P7 | Gynecology | Senior | Interview | Mentor | Junior |
Table 1: Participant demographics.
4.2 Procedure
We detail the procedure of workshops and interviews. For each workshop, there were two surgeons as participants and two researchers as facilitators—the first author led the workshops, and the last author was a second facilitator. We note that participants read an information letter and signed consent forms, that the online questionnaire was anonymous, and that throughout workshops and interviews we did not ask nor record patient-related data.
Figure 1: Summary of method, highlighting the changes in protocol from W1 to W2 & W3.
4.2.1 Workshop 1 (W1).
Preparation: Questionnaire. We prepared W1 through an online questionnaire, gathering events where mentors took over to discuss during the workshops. We sent the questionnaire to 16 surgeons and received 9 answers, including two that we had already recruited as participants for W1 and one resident who participated in W2. The surgeons contacted were part of our network, we encouraged them to share the questionnaire among colleagues. As the questionnaire was anonymous, we could not use it to recruit for workshops.
Procedure. This workshop took place in person. We first introduced the workshop goal and presented a case where a mentor took over to show tool placement, as an example of a takeover. Then, we carried out 3 phases. In phase 1, we asked participants to recall events in MIS where a mentor took over for the purpose of explaining, and then to categorize those events according to goals. In phase 2, participants identified for each event whether it was possible to provide the instruction using telestration tools, rather than taking over. Phase 3 was subdivided into three parts. First, participants brainstormed separately, with the help of the researcher, for new tools that may address the goals achieved through taking over, based on events from phase 2 where telestration failed, sharing and discussing with the rest of the group. Then, the researchers presented their own propositions of tools: videos of other surgeries and instruments overlays. Here, the remote mentor would control a digital representation of instruments that is overlaid on the surgical view. The overlays can be 3D models of instruments or real instruments captured on video and segmented from the background. The operating surgeon therefore sees in their video feed a set of tools that the mentor manipulates. In the last part, participants chose, for each takeover event, the technological proposition that would best fulfill the underlying goals.
Modifications for W2 and W3. After W1, we realized that it was difficult for participants to recall many events in a short brainstorming period (phase 1). We felt we needed to spend time with each participant separately to gather sufficient details to understand the complexity of the information conveyed through takeovers. We therefore decided to replace the first phase of the workshop with interviews, as illustrated in Figure 1. Based on the interviews, we created an initial categorization of information to use during W2 and W3. Moreover, W1 participants emphasized that mentors also took over to perform risky gestures, or to deal with unusual situations they may not fully understand, something that resonated with our past field work with surgeons. We therefore decided to ask interviewees to share not only takeovers aiming to explain and understand, but also to perform risky gestures. Our rationale was that, during telementoring, mentors may need to compensate for their inability to take over with additional instructions to face risky situations. While we were aware from our past field work that takeovers occur also to speedup the surgery, we decided not to explore this case as it is not related to instruction. We also conducted other minor modifications, for instance, when asked how telestration could help fulfill takeover goals, we used a three-level scale (possible, difficult, not possible) instead of two-level scale (possible, not possible). Lastly, we decided not to present the concept of instrument overlays.
4.2.2 Workshops 2 and 3 (W2 & W3).
Preparation: Interview. We conducted 4 interviews to prepare W2 and W3, either remotely through audio-video conferencing or in person. In both cases, we used a collaborative Miro board as support. After selecting three minimally-invasive surgeries where the interviewee judged that the attending surgeon would benefit from additional training, interviewees recalled instances of the three types of takeover events (explaining, understanding, taking risk) for each of those surgeries. At the end of the interview, we asked interviewees if they have experienced other instances where takeovers occur, apart from speeding up the surgery. The interview guide with the specific questions is available in section A. All interviewees participated in the subsequent workshops, except for P6 who was replaced by P7, although we did use their interview to prepare W2 and W3.
Procedure. W2 took place in person, while W3 took place remotely over audio-video videoconferencing. We used Miro as a discussion board for both workshops, following two phases. We started by an introduction, debriefing on the preparation interviews, highlighting the different type of takeovers and why these are of interest to the design of telementoring systems. In phase 1, we presented the categorization of takeover events we produced from the preparation interviews, for participants to add, merge, put aside, or split categories. As we presented the categories, we asked participants to express to what extent the goal could be achieved through telestration using a three point scale (possible, difficult, not possible) and noting the participants’ justification of their ratings. In phase 2, we let participants brainstorm on tools that fulfill the unmet needs of telestration, traditionally met through takeovers, for each need in the categorization. In W2, each participant brainstormed with a facilitator (researcher), and in W3, each participant brainstormed individually. During this brainstorming, most ideas came from surgeons but researchers also suggested ideas that surgeons rejected or expanded. We only covered a generative phase, without a phase where ideas are triaged and removed according to their technical feasibility, usefulness, or overlap with other ideas.
4.3 Data Collection and Analysis
Regarding the questionnaires, we did not perform an analysis because the answers were not detailed enough. Regarding interviews, we recorded the audio and took notes while conducting them. Using the transcripts, the first author coded, using open coding, the information mentors convey or gather through taking over. Then, we grouped codes into categories and sub categories to form an initial categorization, that we crystallized in a mindmap. Regarding workshops, we recorded the audio and took notes on paper. For W2 and W3, we recorded the screen for the entire duration of the workshop to capture the Miro board interaction. For W1, we took pictures of the categorization participants produced and the written notes synthesizing the group discussion. We analyzed primarily written notes, using the recordings only to check for ambiguities.
Following the workshops, we integrated participants’ feedback to our categorization. We explored the contextual elements (e.g., information conveyed during dissection or incision) to explained the different opinions concerning the feasibility of telestration. We then refined the categories to include those differences, and stated for each category of information whether it was possible to convey or gather it with telestration. We grouped categories that seemed different because of their context, but were similar in terms of challenges to convey or gather information. Throughout this process, we regularly performed member checking with a surgeon (coauthor) to verify our interpretations, in particular homogeneity and heterogeneity of our categories, telestration limitations, and elements specifics to surgery. Through interpreting propositions on tools beyond telestration, we elucidated underlying needs and refined our categorization. Lastly, we summarized the information participants thought each tool could help convey or gather.
4.4 Positionality Statement
There are five authors in this paper. The first and last authors conducted the workshops, and the bulk of the analysis and writing of this paper. The first author has a background in cognitive engineering and has been working with surgeons for over one year prior to the study, developing an understanding of surgical mentoring dynamics in previous work through multiple interviews of surgeons and observations of surgery. While participants understood that the first author had this understanding, it was also clear that they only had basic knowledge concerning technical aspects of surgery. The last author is an HCI researcher having performed more than 150 hours of observation of surgery and 40 interviews. Both first and last author are western, based in a European capital, holding a constructionist epistemological stance in research.
5 RESULTS
Our results shed light on the information mentors convey and gather through takeovers, the limits of telestration in doing so remotely, and reflections on future tools to overcome these limits.
5.1 Information Conveyed Through Takeovers
Our first set of findings show the information conveyed through taking over. Takeovers may occur due to breakdowns when instructing action involving communicational resources that mentors have at hand, mostly verbal utterances but also using surgical tools to point. We detail the aspects of gestures mentors convey in takeovers: instrument placement, tissue displacement, force, and movement.
5.1.1 Instrument Placement.
Instrument Tip Position. Mentors instruct instrument position according to two cases: instruments inserted via trocars and thus visible on the surgical view, and instrument inserted through natural orifices, such as the vagina or the rectum, thus not visible as they reside behind tissues. When instruments are visible, mentors can instruct the location where to perform actions such as incision, grasping, coagulation, blood vacuuming, or dissection. When instruments are not visible (including fingers), the challenge lies in accurately locating them through blind probing of the tissue to observe its deformation, frequently performed by an assistant.
Instrument Orientation. Precise instrument orientation is crucial to create safe and efficient gestures, including roll and direction (yaw and pitch). Regarding roll, when instruments have a sharp tip, they need to be positioned as far away as possible from noble structures to avoid cutting them inadvertently. When activating the ultracision harmonic scalpel [12], the fixed edge of the heat-producing jaw should remain opposite to noble structures, which should not be dissected, and are not clearly visible. Mentors, having a better intuition about their location, know precisely how to rotate instruments to keep the heating edge away. Beyond roll, mentors need to convey instrument direction relative to tissues, both for exposure and dissection. For example, positioning the instrument perpendicular to the bladder is a way to keep it parallel to the vagina during exposure, as the bladder is perpendicular to the vagina. As the instrument orientation is evaluated comparatively to tissues, surgeons can directly move the instruments or adjust the tissue position with their second hand to present them with the right angle with respect to the other instrument.
5.1.2 Tissue Displacement.
In addition to conveying where to perform an action, surgeons also take over to show where to move tissues to create appropriate exposure and enable the execution of other gestures such as dissection. Possible movements include tracting, pushing, turning, adjusting the angle of a tracted tissue or adjusting the applied tension.
5.1.3 Force.
Force is a critical dimension of gestures that complements the spatio-visual dimension. For instance, holding a mobile organ in mid-air equilibrium involves adjusting the placement of an instrument, while applying the right amount of force. Moreover, effective dissection requires following the right path to attain a target and applying the right amount of force to avoid inadvertently injuring noble structures such as blood vessels. Force is characterized by both its direction and its intensity. Mentors take over to demonstrate the acceptable intensity in a given situation.
5.1.4 Movement.
Surgical gestures cannot be described only using static information such as position, roll, direction and amount of force, they also involve a temporal dimension. Mentors take over to demonstrate movements such as jaw opening, coordination of instruments or multifaceted movements. Jaw opening involves adapting the speed and amplitude with which surgeons open jaws toward tissues during dissection. For instance, in areas containing noble structures, performing motion slowly only stretches the tissues, enabling the surgeon to stop before it breaks. Fast motion, on the other hand, would break the tissues before the surgeon realizes the presence of a noble structure in the path. Depending on context, mentors may advise performing small but quick gestures or ample but slow gestures. Coordination of instruments involves synchronizing instruments held by the operating surgeon and the assistant to perform a gesture. One example is divergent traction, where both successively pull on two sides of an open cyst to tear it apart. Here, mentors may need to instruct the rhythm when alternating traction. Multifaceted movement combines features such as direction and force with a specific timing. For example, sinking an instrument deep into tissues before pushing them is a movement characterized by several directions as well as instrument synchronization.
We presented thus far the difficulties of surgery that lead mentors to take over. We now present the information surgeons deemed possible to convey through telestration, highlighting situations where the difficulty persists, and taking over is necessary.
5.2 Using Telestration and Speech to Convey Information when Replacing Takeovers
In this section, we elaborate on why telestration is not always sufficient to transmit the information normally conveyed through takeovers, summarized in Figure 2.
Figure 2: Diagram summarizing the limits of telestration when conveying information usually conveyed through taking over (possible in green, not possible in red)
5.2.1 Instrument Placement.
Instrument Tip Position. Surgeons judged it possible to convey instrument position using speech and telestration, but only for tasks involving visible targets. This includes incisions, grasping and coagulating, as the target structure is already exposed. Here, the projection of the pointer onto a structure clearly indicates the location for the action. Similarly, blood that needs vacuuming is visible in the foreground, leaving little room for ambiguous interpretations of location with a pointer. However, dissection involves reaching non-visible targets, because tools are used to open a path through tissues. In such cases, mentors can only point toward a structure that is not yet visible to instruct the mentee on the general direction for dissection. For instance, P4 reported that mentees sometimes “do not see the plane”, which deters them from moving in the correct direction. Participants voiced that 2D pointers can lead to ambiguous interpretations of instruction when referencing positions inside tissues. Using the pointer, they can only indicate the surface point where dissection should start and need to update it constantly as the gesture evolves. In the case of non-visible instruments, inserted through the vagina or rectum, the difficulty arises because the gestures are performed blindly. It is not only the target that is not visible, but the instrument itself. In the absence of visual cues, the mentors lack a frame of reference to understand the mentees’ gesture and correct it.
Instrument Orientation. Participants agreed that telestration and speech is limited in conveying instrument orientation, particularly when there are no visible landmarks for reference. When there is a visible target structure, mentors can use it as a resource, for example asking to turn the scissors toward or away from the target, to avoid wounding it inadvertently, or during dissection, instructing to rotate the opening axis parallel or orthogonal to the axis of the limits between the planes. One participant thought that telestration could help convey roll more efficiently than sole speech, as a mentor can draw the jaws during dissection, to instruct in which direction to open pliers with respect to a plane (parallel or orthogonal). However, a visible target is not always available to ground instructions, such as when adjusting roll to the nearest degree for the dissection of ureter with ultracision. Also, mentors often instruct “to be perpendicular” (P1) for instance to a vessel or the vagina. However, this verbal instruction can be ambiguous and, therefore, difficult to execute. For instance, being perpendicular to a vessel can be achieved in different directions. It indicates how to readjust direction in one axis (e.g., yaw) but it fails to help orient the instrument in the other axis (e.g., pitch). Additionally, the instruction can require the mentee to have an accurate representation of the anatomy, for example when positioning an instrument perpendicular to the bladder which is not dissected, requires a correct mental representation of this structure to ensure the instrument is not too tangential to the surface of the organ. During dissection, mentees may not always visualize the surgical plane, and mentors cannot point to them as they are, by definition, inside tissues. Instructing instrument orientation is therefore closely linked to showing tissue placement. In addition to the challenges of conveying information, mentors stated a preference for taking over instead of using telestration or speech, because it is a more efficient way to communicate. For non-visible targets, specifically instruments inserted through natural orifices, indicating direction poses problems similar to when indicating position. The gesture is performed blindly, making it difficult for the mentor to understand and instruct it, with or without telestration.
5.2.2 Tissue Displacement.
Telestration can support explaining where to tract or tense tissues, but explaining where to push tissues is more difficult. Regarding traction, participants suggested first pointing toward where to grasp the organ, and then drawing an arrow toward the direction of movement. To express tenseness, one participant shared that they have already used telestration to draw the desired position of tissues after they have been tensed. Instructing to push tissues, however, was considered more difficult because the targeted area is hidden behind the tissue itself. P7 shared that they would not know how to draw an arrow to convey pushing direction, although they would for traction.
5.2.3 Force.
Participants agreed that telestration, even combined with speech, does not provide adequate support to convey intensity of force. The difficulty lies in the mentor’s inability to accurately interpret the force that the mentee applies. Still, some cues such as tissue deformation can help guess the current applied force, an understanding that mentors can use to instruct adjustments in the applied force, whether to increase or decrease. However, if these instructions are not sufficient for mentees to succeed after a few trials, mentors can simply take over. Regarding the direction of force, it involves instructing tissue placement and instrument direction, evoked in previous sections.
5.2.4 Movement.
Participants found that telestration can be used to convey simple movement such as opening of jaws, but less so for instrument coordination or multifaceted movements. Regarding jaw opening, directives about the opening degree can be given verbally, instructing to increase or decrease the opening. P7 reflected that telestration could be used to draw the desired opening. While participants acknowledged that speed could also be adjusted through indicating verbally, either to go faster or slower, they generally considered it more challenging than simply conveying the opening degree. Regarding coordination of instruments, participants expressed that telestration and speech do not provide adequate support. Lastly, instructions that combine several movements are too complex to instruct with telestration and verbal utterances, these were even hard for participants to articulate during workshops, for example specific jaw opening movements during the dissection of the lymphatic canaliculi.
5.3 Technological Support to Convey Information Remotely
Participants proposed tools that go beyond single 2D pointers and drawings, to convey information traditionally conveyed through takeovers, summarized in Table 2. We present these tools according to where they reside, inside or outside the surgical view, separating the case of force as it involves haptic rather than visual information.
| Remote instruction tool | Information to convey | |
Tools on the | Dual pointers | Instrument tip position |
| 3D pointers | Instrument direction | |
| Virtual instruments | Instrument direction, roll | |
Virtual anatomies / | Movement | |
Tools outside | Live video of | Movement |
Video recording of | Instrument Direction | |
Anatomical chart / | Instrument Direction (valve) | |
Anatomical | Instrument Direction | |
| Physical proxy | Tissue placement | |
Tools to | Force sensors | Force Intensity |
| Color scales | Force Intensity | |
Discrete force level | Force Intensity | |
| Physical hook | Force Intensity | |
Table 2: Correspondence between remote instruction tools and needs for conveying information.
Tools Inside the Surgical View. First, participants proposed new types of pointers: 3D pointers or dual pointers. 3D pointers consist of a cone or an arrow showing a direction in 3D dimensions, which could be used to convey dissection direction/instrument direction, in addition to tip/dissection location. A dual pointer tool consists of two cursors, each with a different shape, useful when referring to two different instruments held by the operating surgeon or the assistant. Dual pointers can support multiple deictic instructions, for example in bleeding management, two pointers support the distinguishing between the location where to coagulate from the location where to vacuum. The second proposed tool was virtual instruments, referred to as holograms, or instruments in augmented reality, that mentors can control and position within the surgical view. Participants thought these could support conveying orientation, jaw opening direction, opening speed, dissection direction, coordinate movement of pliers and other complex gestures. P5 emphasized the need to convey 3D information with virtual instruments using 3D representation of tissues. A third tool participants suggested is virtual anatomies, encompassing 3D models of tissues. These models are placed on the surgical view, and can be patient-specific derived from pre-operative images, or, generic. Participants warned that the shape of structures can diverge from the pre-operative scan, and, therefore, they put forward the ability to modify the shape to fit the real-time conditions. In this way, they would better communicate tissue placement when hidden behind fat, especially when guiding dissection and exposure.
Tools Outside the Surgical View. Tools to support remote instruction can also reside outside the body. One proposition was to capture live video of mentor’s hands and arms, to convey gestures during instruction. For instance, to mimic jaw opening motion, mimic the vein with an arm and the pliers with the other hand to show the angle when coagulating a vein, or if they are holding tools, demonstrating the instructed motion or explaining tool usage. Another proposition was using video recordings of previous surgeries to convey different instructions such as orientation, opening speed, valve tension, coordinate movement of two pliers for exposure and dissection, movement, exposure gestures, or showing tool usage. Video length was a concern, one participant mentioned videos could last “a few minutes” to show an ideal surgical step, while another emphasized the video needs to be as short as possible to avoid disrupting the flow, even just a few seconds—enough to show the movement of pliers. Moreover, one participant mentioned video can be used to show both bad and good practices, which helps understand the limits of an acceptable gesture. Anatomical charts or photos of surgery were suggested to show where and in which direction to push, particularly for valve positioning. Anatomical simulations were proposed to convey tissue displacement and instrument placement relative to anatomy, especially to demonstrate how to tense a target structure, how to expose with a set of pliers and incise with another, how to position valves or convey dissection direction, and how to perform complex gestures, including in which direction to dissect. Lastly, physical proxy were proposed to demonstrate gestures. The remote mentor would hold and move remote proxy instruments to demonstrate a gesture, while the mentee would hold and feel through a local proxy in the operating room the gestures performed by the mentor. This approach would enable mentors to convey movement in three directions and the intensity of applied force. This idea was proposed specifically to convey the blind gesture of pushing the valve in the vagina.
Tools for Conveying Force. Participants emphasized the importance of providing feedback separately for each instrument, as the required force for exposure may differ from that needed for dissection with another instrument. As describing force verbally can be particularly challenging, participants proposed equipping mentors with force sensors where they can apply forces. This could take the form of a box a mentor can push or a hook they can pull, enabling them to specify force, which would in turn be presented to the mentee through various means, such as using visual scales with colors, numerical values or a physical replication of force. Color scales can be applied to instruments, whether on a demonstration video or on overlays (e.g., virtual instruments). Participants discussed both absolute scales and scales relative to the current force applied by the mentee. When reflecting on an absolute scale, one participant also proposed using discrete names for levels of force (e.g., F1, F2, F3, F4). Participants highlighted that for relative scales to be effective across various gestures, the scale needs to instantly adapt to the mentees’ gesture, as manual adjustments by the mentor may be too slow and inefficient. In addition to color feedback or verbal references to force, participants proposed the use of a physical tracting hook to demonstrate pulling force. To illustrate this concept, the participant held the researcher’s hand (representing the hook), pulled the hand with a certain force and asked them to resist. With the hook, the mentee would physically sense the resistance instead of relying on a visual or audio scale. Participants were reluctant to the idea of having a bracelet that would pressure the skin, as they were skeptical that surgeons could easily relate the pressing force sensation to a pulling force.
5.4 Information Gathered Through Takeovers
In addition to conveying information, mentors engage in information gathering to achieve the required level of understanding to guide the mentee. Mentors observe the surgical view and take over when the view is insufficient to gather the needed information. We distinguish between information related to locations (e.g., body structures and surgical instruments) and gesture execution.
5.4.1 Understanding the Location of Tissues, Equipment and Instruments.
Concealed Tissues and Spaces. Mentors may face doubts regarding the location of tissues or spaces between tissues (planes). To gain understanding on these locations, mentors take over and seek anatomical landmarks. As P4 puts it, they dissect and expose tissues (move aside, tract or push) in different ways until uncovering a space or a structure that they did not observe previously. Mentors may also orchestrate the camera movement performed by the assistant with their actions to explore efficiently.
Misplaced Surgical Equipment. In rare yet important cases, surgeons may need to take over to locate a misplaced object within the patient, such as a needle or a compress. The challenge is similar to finding a structure or a space, requiring surgeons to manipulate tissues and the camera to pinpoint the target. However, in these scenario, there are no dissection gestures involved, meaning there is no alteration of the anatomy.
Orientation of Structures. When mentors need to identify structures or or determine their locations inside the view, they first need to understand how the camera is oriented. For example, surgeons rely on direction to identify veins; the gonadal veins travel horizontally, whereas the renal veins travel vertically (P5). However, if the camera is oriented sideways, surgeons may mistakenly misidentify one vein for another. When the mentee fails to provide this understanding, mentors take over to adjust the view. P5 explains that taking over the endoscope to perform a slight rotation already lets them understand orientation, e.g., if the view is horizontal relative to the patient body.
Identifying Pathological Tissues and Instruments through Palpation. To diagnose the location of a pathology on a tissue, mentors interact with the tissue by pulling and poking, because pathological tissues, such as tumors or endometriosis, exhibit distinctive stiffness. Similar to pathological tissues, mentors use tactile feedback to find valves or dilators located behind tissues, as their contour can seem blurry just by looking at the surgical view video. In contrast to the placement of structures or instruments inserted through trocars, mentors are unable to gauge the stiffness of tissues solely by looking at the surgical view. Therefore, they take over to sense for themselves.
5.4.2 Execution of Gestures.
We observe two specific needs related to understanding the gesture’s execution: understanding the constraints, and understanding the adequacy of the gesture. Regarding the constraints, mentors need to understand the reasons why mentees face difficulties, ensuring that the gesture they envision is hindered by external constraints rather than the mentee’s lack of dexterity or misunderstanding. A common case is the trocar preventing the mentee from reaching a distant structure with their instruments when a mentor instructs to manipulate it, but other constraints occur directly on the body, for example tissues that are heavier or stiffer than expected. Mentors also need to determine the adequacy of the gesture to perform. As some gestures can have irreversible consequences, mentors aim at making sure that all aspects of the gesture are under control. Through taking over, they can realize early if the gesture is inadequate and correct it before it exacerbates the situation. This is the case when surgeons open a folded bag inside the patient. The process of unfolding the bag requires methodical movement, only continuing the gesture if the bag unfolds as expected to avoid tangling it. In this case, mentors usually assist the gesture either through using the assistant pliers to perform parts of the gesture by themselves, or by driving the endoscope to point and direct the gesture.
5.5 Using Telestration and Speech to Gather Information When Replacing Takeovers
Our results highlight that, with few exceptions, it is intricate for remote mentors to understand the mentee, patient and context, when using speech and telestration. We now elaborate on the different cases following the same structure as the previous section, summarized in Table 3.
| Actions mentors perform during takeover | Telestration support | |||||||
| Dissecting | Moving | Moving | Feeling | Experiencing | ||||
| Information to gather | Location | Concealed spaces | x | x | x | Laborious | ||
| Misplaced equipment | x | x | Laborious | |||||
Structure/camera | x | Not possible | ||||||
Pathological tissues | x | Not possible | ||||||
Gesture | Bodily constraints | x | x | Not possible | ||||
| Trocar constraints | x | x | Not possible | |||||
| Adequacy | x | x | Not possible | |||||
Table 3: Correspondence between understanding needs to gather information, actions mentors perform during takeovers, and difficulty to gather information with telestration
5.5.1 Understanding Location of Tissues, Equipment and Instruments.
Regarding concealed tissues and spaces, participants agreed that instruct in this context without taking over is only partially possible. Mentors would need to give low-level instructions on how to move the camera, tissues and where to dissect, given that mentees cannot interpret location references, as they got lost in the first place. The process would be time-consuming and tiring, as it would require many instructions: “zoom out a bit, grasp here, release, grasp again” (P7). Also, mentors would face the difficulties in conveying information already detailed in section 5.2 such as giving directions when the target is not visible. Participants found that it may be slightly easier to look for misplaced surgical equipment as it would not require instructing dissection gestures. Nonetheless, they agreed that telestration and speech could not help them gather information on orientation of structures and the camera, as mentors cannot sense that the camera is well positioned, and have to rely on the assistant feedback. Similarly, participants judged that existing tools do not help to gather the necessary haptic cues to identify pathological tissues and instruments through palpation.
5.5.2 Execution of Gestures.
For both bodily and trocar constraints, participants expressed that speech and telestration were insufficient to understand the constraints impeding a mentee’s gesture, mentors need to experience them to be convinced that their instruction is not feasible. Specifically for stiff tissues, while the mentor could rely on the mentee’s verbal description, one participant reported it would be difficult for mentors to fully trust the mentee’s perception as they lack expertise to accurately interpret stiffness. In other words, judging whether the force applied is standard or not necessitates expertise to understand the acceptable force in a given context.
When it comes to adequacy, even if telestration could be used to transmit an acceptable level of information for testing a gesture before full completion, P7 raised a limitation: if the mentor provides an instruction that turns out to be incorrect, the mentee may perform it too extensively before the mentor intervenes, therefore making the outcome worse than the initial state. Here, the problem lies not in that telestration fails to convey the needed information, but in that the communication loop is too long to adjust the gesture quickly.
5.6 Technological Support to Gather Information Remotely
To address the needs related to understanding and overcome the limitations of telestration, participants proposed different alternatives to takeovers, as summarized in Table 4.
Tools Inside the Surgical View. Participants suggested enhancing the surgical view in different ways. One participant mentioned a global vision system which consists of having a camera where the trocar meets the abdominal wall, to see a zoomed-out view of the body from the inside, in addition to endoscope view. They expressed that this system could help understand the location of the horizon through landmarks, as well as help localize structures otherwise outside the view. P7 proposed increasing video quality to see in detail the deformation of tissues, which facilitates understanding in two ways. First, it can help identify structures and planes, and, second, it can answer the need for mentors to understand textures of tissues (for instance when diagnosing stiff pathological tissues). This idea was inspired by the high-definition view in robotic surgery to compensate for the lack of haptic feedback. Participants also proposed to add a level to the surgical view to help the mentee correct camera rotation, as well as help the mentor understand how the camera is rotated. Beyond augmenting the view, participants also proposed that mentors control the camera independently of the mentee. They also mentioned having a 360° camera for mentors to see in all directions inside the body. They compared it to existing 30° endoscopes that capture the body from a different angle, a feature they find particularly useful when looking for a misplaced needle.
Tools Outside the Surgical View. Mentors also proposed to use tools outside the surgical view, such as additional video streams in the operation room. These can have benefits when understanding constraints, for example being able to observe that the mentee is constrained by the trocar, thus blocked from moving further inside. One participant also proposed for mentors to have an x-ray-like vision of the surgical site in addition to the endoscopic video. In this enhanced vision, mentors would have an external view of the patient showing how instruments are located in the body relative to tissues. Participants specified that a downgraded version of this tool, showing only instrument insertion in the body without the tissues, would not be as useful. In addition to streaming new points of view of the operating room, participants proposed using an anatomical simulation to anticipate the adequacy of a gesture through testing them in the simulation before giving the instruction.
Tools for Understanding Force. Participants proposed to directly use force sensors to measure the force that mentees apply to tissues. If measuring force directly during the gesture is not possible, participants proposed using sensors outside the body where mentees reproduce the force they apply on the body, similarly to the propositions for conveying force (section 5.3). The representation of force would therefore be the same as when mentors convey force.
| Remote instruction tool | Information to gather | |
Tools on the | Global vision | Misplaced equipment |
| Increasing video quality (e.g., 3D) | Concealed spaces and tissues | |
| Level | tissues /camera orientation | |
| Control the camera | Misplaced equipment | |
| 360° camera | Misplaced equipment | |
Tools outside | Videos of trocars / | tissues / camera orientation |
| X ray-like vision | Trocar constraints | |
Anatomical simulations | Gesture adequacy | |
Tools to | Force sensor inside | Pathologic tissues |
| Color scales | Pathologic tissues | |
Naming discretely | Pathologic tissues | |
| Physical hook | Pathologic tissues | |
Table 4: Correspondence between remote instruction tools and needs for gathering information.
6 DISCUSSION
Our study contributes to understanding the limitations of telestration and speech in satisfying mentors’ needs typically met through takeovers. We highlight the difficulties to convey instructions when there is no visible reference point whether it be for instrument orientation (roll and direction), to indicate the location where to push tissues, or specify the direction of dissection. We also illustrate the challenges in conveying simultaneous tip positions of instruments, force, and movements including speed when opening instrument jaws, coordination of instruments, and complex multifaceted movements. In these cases, telestration is limiting not only in terms of the types of information it can effectively support in conveying, but also in the additional time and effort it demands to do so. Concerning the need to gather information, telestration can partially support indicating where to dissect and move organs (to find a concealed space, structure or equipment). Still, it falls short in supporting mentors when gathering texture information (to determine placement of instrument inserted through natural orifices, pathological aspect of tissues), camera and structure orientation, and gesture execution (including its constraints and adequacy). Based on these findings, we discuss (1) the design and evaluation of remote collaboration systems within and outside surgery, and (2) future telementoring tools that go beyond telestration.
6.1 Design and Evaluation of Tools to Increase the Diversity of Information Conveyed Remotely
While previous work described components of surgical expertise [41, 54] or mentor’s physical guidance [51], our categorization sheds light on the information conveyed and gathered through takeovers during mentoring of experienced mentees, and the extent to which this information can be conveyed with telestration. These results contribute not only to the design of novel interactive tools but, more importantly, to their evaluation.
Regarding tools for collocated and remote mentoring in surgery, these are often evaluated independently of the specific type of information that mentors need to convey during surgery. For instance, Shabir et al. [48] measured accuracy when mentees followed a 2D trajectory set by the mentor using virtual instruments overlaid on the surgical view. Chetwood et al. [8] evaluated the effect of the mentor’s gaze on the mentee’s ability to identify objects. While those tools might outperform telestration combined with speech, the evaluations do not show if they can support mentors in conveying new types of information. When examining the type of tasks in traditional evaluations, they often require transmitting information that our participants judged it possible to convey with speech and telestration such as where to incise, where to clip a visible target. Regarding evaluations relying on the dissection of real tissues, which necessitates more complex instructions than lab experiments, these evaluations have so far focused on technical aspects, such as delay [50] or interface usability [29]. Although these studies describe the different surgical steps undertaken, none provides a comprehensive description of how the evaluated tool supports mentors in conveying information. Consequently, our categorization of needs contributes to the design of tasks that require conveying information challenging to convey with telestration, providing a resource for analyzing experiments in real surgery.
Our work also challenges researchers outside surgery, to create tools that support conveying a wide diversity of instructions. Currently, remote collaboration in HCI is often studied using assembly tasks, including blocks [25], Legos of diverse sizes [31, 32], tangram pieces [31], robot construction kits [22] or even origami [31]. Assembly involves a variety of needs, including object identification and manipulation [25], search and select, rotation and alignment, attaching, detaching, and pattern matching to ensure the final model follows a plan [32]. It is often taken for granted that assembly tasks are a good proxy for general physical tasks. However, these tasks simplify the use of language, for example, pieces are colored in ways that ease reference unlike indistinguishable colors of tissues in surgery. Their orientation is limited to two options, as they can be laid either horizontally or vertically, unlike instruments or tissues in surgery, which can be laid in arbitrary angles. The building process can be sequenced, leading to instructions to manipulate one piece after another, in contrast to the need to communicate simultaneous information. Similarly, force and movement are often not components of the tasks. This difference in task complexity prevents studies from understanding the limitations of telestration. For instance, Fussell et al. [22] observed that pointers can help show motion or indicate angle of insertion and orientation, however, their task did not involve the use of non-visible structures as references. Other tasks such as bike repair [20] are perhaps more complex, although, this study used a shared manual which removes some of the difficulty during instruction. Our investigation of surgical mentoring therefore reveals new challenges for remote collaboration that are not traditionally present in evaluations.
Lastly, our findings show that tools need to go beyond sharing visual information to support conversational grounding and situation awareness. While previous work demonstrated the importance of sharing visual information [24], we highlight the potential benefits of sharing other information, such as tissue textures, body and trocar constraints, or camera orientation. Our results also emphasize that shaping the view [38] can occur in contexts where the mentor is unsure of how to handle a particular situation; the importance of exploring mechanism that give flexibility to mentors to quickly correct their instruction when realizing it is inadequate, as well as the importance in supporting mentors in distinguishing between constraints and incorrect execution of the instruction by the mentee.
In summary, by emphasizing the richness of information conveyed to teach surgery and the critical importance of gathering information to understand, our work encourages designers to explore and evaluate systems that support the conveying and gathering of diverse information necessary for complex physical tasks.
6.2 Future Tools for Surgical Telementoring
We now discuss future tools beyond telestration that are worthy of study in their contribution to surgical telementoring.
6.2.1 Tools to Convey Information Inside the View.
Novel Pointers. While participants suggested that the use of two pointers could help convey location of simultaneous actions, their instructional benefits have not been studied to our knowledge. However, dual pointers involve the need for two hands to control independent elements. Hinckley et al. [26] studied bi-manual interaction for controlling 3D models, concluding that hands are better at working cooperatively toward a single goal, as opposed to controlling separate input streams. Therefore, future work studying dual pointers should carefully consider input. Concerning 3D pointers, we did not find studies demonstrating their ability to address needs in mentoring, specifically conveying direction. The 3D pointer developed in Jarc et al. [28] did not show better results than 2D pointers compared to other means for pointing, such as hand or virtual instruments for robotic surgery mentoring.
Virtual Instruments. Virtual instruments were shown to be technically feasible in robotic surgery [29], open surgery [23] and laparoscopic surgery [48]. In terms of benefits for instructions, they led to a similar error count than onsite mentoring with hand gestures during suturing tasks [49], and were preferred to instruct certain aspects of the gesture. While these results are encouraging, participants mentioned that surgeons would not turn to telementoring for basic skills such as suturing, they expected benefits of virtual instruments for orientation, opening motion, direction of action and coordination of instruments. We therefore identify a need to explore further virtual instruments for these goals.
Virtual Anatomies. Participants suggested virtual anatomies could communicate tissue placement and instrument orientation. Standard 3D models may not be useful because the anatomy varies a lot between patients, as mentioned by one participant in the ARTEMIS study [23]. Conversely, patient-specific anatomical models have been shown to be beneficial for navigation [42], and, to relieve stress before performing the gesture through doing a last minute check [44]. However, those models, which are based on preoperative imagery, are also limited because the anatomy changes drastically as dissection is performed (see section 5.3), and reconstruction of models is time-consuming, often necessitating manual work. To our knowledge, the possibility for mentors to adjust 3D models to reflect their understanding of anatomy has not been explored, nor has its use for instruction when indicating how to orient instruments.
6.2.2 Tools to Convey Information Outside the View.
Video. While videos of prior surgery were mentioned repeatedly as means to demonstrate a specific technique, they are rarely studied as a tool for mentors to create instruction. Instead, they are often opposed to mentoring [3, 49, 56], leading to mixed results. Prerecorded instructional videos underperformed live telementoring with virtual instruments, in terms of errors and duration in Shabir et al.’s [49] study. However, Xeroulis et al. [56] found that instructional videos led to similar retention compared to expert feedback after the task and higher retention compared to expert feedback during the task. This hints toward a potential of videos for knowledge retention compared to live mentoring. All in all, there is space for exploring the benefits of videos as an instructional resource for mentors, as well as the challenges of having on-demand videos that mentors can choose from. Participants also mention using video of the mentor’s hand and arms to reify organs and instruments. This echoed how mentors in other studies used their hands to mimic the curvature of the needle [49] or movements of organs [50]. Beyond conveying information, Scavo et al. [45] showed that ghost hands can improve the sense of connectedness. We note that those works overlaid hands on the surgical view, instead of showing separate videos.
Simulation. Similarly to videos, simulations have been studied as an alternative to learning in the OR [35] and have not been explored as a potential resource for telementoring during surgery. It is still to be determined if the simulator could be useful for remote instruction, and if the parallel between simulated tissues and real tissues can easily be made.
Anatomical Charts. Participants suggested using anatomical charts to instruct the assistant when positioning a valve. While this is not a major problem as valve positioning is often performed correctly by expert surgeons, the use of other instruments inserted in natural orifices may require mentoring. To our knowledge, the use of such charts in the OR remains unexplored and in particular in how they could support the execution of blind gestures.
Physical Proxy. Beyond showing a gesture in a visual format, participants also proposed using physical proxies where a mentor can model a gesture, so that the mentee can feel it with their own hand. Haptic physical proxies were explored for remote training in a simulation [6] where the person (mentor or mentee) not exercising the control could follow the gesture of the other person. For Chebbi et al. [6], those features enable telementoring when mentors show the gesture, and tele-evaluation when mentors follow mentees’ gesture. However, performing a gesture is not just performing a motor movement, but also achieving a specific outcome and learning about the constraints of the environment in which the gesture is executed [53]. It is not evident that showing the “perfect” gesture on a simulator would provide a clear instruction, as it will focus only on the motor aspect, and not the outcome of the gesture.
6.2.3 Tools To Convey and Gather Force Information.
Our study emphasized that the need to feel and explain force in surgery is paramount, challenging telementoring tools. Outside surgery, Kim et al. [30] showed the benefits of sensing and sharing on the local worker’s hand force, to improve the expert’s awareness of the worker’s tasks. Participants proposed different tools for recording force information and representing it for surgical telementoring.
Measure. Concerning the use of sensors on the patient to record forces applied on tissues, their large size is not yet suitable for real surgical settings [19]. For physical tasks, Kim et al. [30] used electromyography sensors to measure muscular contraction which could be explored in the context of surgery. Besides, participants proposed having sensors outside the body, which has not been explored to our knowledge. It is still unclear if a simple force measure, contextualized from the direction of force, would be informative enough to fulfill the need for understanding force when guiding the mentee.
Representation. Regarding force representation, tactile and visual feedback has been explored [19]. This feedback is intended for surgeons operating in laparoscopic surgery to compensate for the deterioration of haptic feedback caused by trocars. However, for telementoring, it is unclear if visual or tactile feedback would provide the same benefits; here, the person applying the force would be different from the main recipient of the force feedback. Additionally, those research focused on presenting force to sterile surgeons. However, for force information sent from mentees to mentors, other approaches that are not limited by sterility can be explored.
6.2.4 Tools to Gather Information Inside the View.
Improving the Surgical View. Participants suggested that enhancing the vision of the surgical view could fill the need for mentor’s situational understanding, compensating for their inability to seek for this information through takeovers. In particular, a global vision could help mentors have a broader view of the body. This concept has been explored by adding two additional cameras on the trocar, at the back of the endoscope [52], capturing contextual view around the surgical site. While this system was developed to decrease blind spots when operating, it could also be used for telementoring. Moreover, this could serve as a new resource to instruct on plane position or tissue handling. Concerning 360° vision on endoscopes, the device size is currently too large to operate inside the body [40]. Other work evoked the possibility for mentors to control the camera remotely through robotics [34]. While this is possible, it requires heavy infrastructure to maintain safety, as even hand control of the endoscope can damage vessels and create complications. Lastly, the use of a level to measure the angle of the camera with respect to the horizon, something that is already integrated in robotic surgery but not in classic MIS.
6.2.5 Tools to Gather Information Outside the View.
Video. Participants proposed providing new views of the operating room to the mentors, in particular to help them understand information on trocar constraints. This could be done through the use of a robot controlled by the mentor, that could move in the OR as proposed by Agarwal et al. [1]. The authors mentioned how mentors used it to discuss where to introduce the trocars in the body, although there are no studies on how filming the trocars could help mentors become aware of gesture constraints during the operation. Moreover, adding videos should be done carefully as multiplying viewpoints has been shown to decrease efficiency [21, 36]. Another approach that could be explored would be to leverage the surgical view, to display how far instruments are inserted and thus the available margin of maneuver. Regarding the idea of an x-ray-like vision, this would necessitate both to locate organs and instruments positions relative to trocars. Locating instruments relative to trocars could be achieved through statistical and geometrical modeling approaches [55], but it would not display the key information of how instruments relate to tissues, to see specifically that a tissue is not reachable. Concerning simulations to test gesture adequacy, they are different from the ones proposed to give instruction as their goal is not expression but prediction. While some research aims at building predictive models such as for brain tumor extraction [57], it is still not fully accurate and may not be generalized to softer tissues.
6.3 Limitations
We acknowledge that our method has limitations. First, although participants have used virtual pointers and drawings for instruction in robotic surgery, they provided a projection of how they would use telestration in a remote context. We agree that further investigation is needed to confirm the ability to perform those actions with telestration in a way compatible with surgery (time and safety). Second, as surgeons are collocated during robotic surgery, this highly impacts telestration use, as they can quickly turn to other resources, such as a face-to-face conversation, when running into breakdowns. This format has constraints (immersion in a console, drawing only when the mentor is outside the console) and features (3D vision) that standard MIS does not have, which also influence how they used telestration. Lastly, the results are bound to the participants’ specialties, institutions, practices and preferences. We acknowledge the need of further exploring this topic in other populations with different cultures to complement and contrast our findings. We acknowledge the limitation of using questionnaires that resulted in less events to discuss during the first workshop than in the other ones. This can be explained because we may have failed to convey our expectations in the introduction and structure of the questionnaire. Providing examples of detailed responses could have improved the quality of our answers. Second, we sent the questionnaire to senior surgeons with limited time. It may be easier for them to take time during a meeting with a blocked duration, engaging with an interviewer than to fill a questionnaire where this task is competing with numerous other demands.
7 CONCLUSION
In this work, we present the results of interviewing and conducting workshops with surgeons, inquiring into two types of needs that cause takeovers: conveying and gathering information for mentoring. We present a categorization of these needs, and limits to fulfill them when using telementoring systems that rely solely on telestration and speech. We open research directions into the future tools that telementoring systems could study to support mentoring practices.
ACKNOWLEDGMENTS
This research was supported by the Labex CAMI (project ANR-11-LABX-0004) operated by the ANR as part of the program “Investissement d’Avenir”, and the Institut Universitaire d’Ingénierie en Santé of Sorbonne Université. We thank all the participants of this study for their valuable time as well as members of HCI Sorbonne research group that provided valuable feedback while writing this paper.
A STRUCTURED INTERVIEW SCRIPT
Do you agree that I record the interview?
[Participant signs consent form]
The objective of the interview is to identify tasks where takeover is needed during mentoring. The final objective is to guide in the design of a telementoring system, i.e. when the expert surgeon is at a distance. In this project, I focus only on non-robotic laparoscopic tasks. From the tasks we will identify, my goal is then to dig into the communicative needs of the mentor and the mentee that already exist during the mentoring or the new needs that would appear during the telementoring. Before we start, I will ask you for some general information. [Fill in the info sheet (specialty, frequency of mentoring as a mentor, frequency of mentoring as a mentee)]
To begin, could you pick three operations where you think there is the greatest need for training for practicing surgeons? Why or why not? If not, just pick three operations that you think are complex or important. [Fill in the first row of Table 5]
Now, for each operation, I’m going to ask you to identify if you have ever experienced 3 different types of handoffs as a mentor or mentee. I will introduce you to the three types. The first type is when the mentor takes over to explain because it is easier to express yourself with the tools in hand. The second type is when the mentor doesn’t know what to advise and takes over to better understand the situation, to build their intuition. The third type of handover is when the action to be taken is risky and the mentor prefers to take responsibility for it rather than letting the mentee do it. We will fill in the matrix box by box but if you think of examples of takeovers for other operations that we have not listed, do not hesitate to tell me. [Fill in Table 5]
Questions used to probe interviewees when completing Table 5:
• | Takeover to explain to the mentee: What information is the mentor conveying? | ||||
• | Takeover to understand the situation if the mentor has doubts or needs to build intuition: How is the mentor’s action different from the mentee’s action? If the mentor were to explain, what would be the most difficult information? | ||||
• | Takeover to prevent the mentee from taking a risk: How is the mentor’s gesture different from the mentee’s gesture? If the mentor were to explain, what information would be most difficult to explain? | ||||
Supplemental Material
Available for Download
- Rahul Agarwal, Adam W. Levinson, Mohamad Allaf, Danil V. Makarov, Alex Nason, and Li-Ming Su. 2007. The RoboConsultant: Telementoring and Remote Presence in the Operating Room During Minimally Invasive Urologic Surgeries Using a Novel Mobile Robotic Interface. Urology 70, 5 (Nov. 2007), 970–974. https://doi.org/10.1016/j.urology.2007.09.053Google Scholar
Cross Ref
- Mehran Anvari. 2007. Telesurgery: Remote Knowledge Translation in Clinical Surgery. World Journal of Surgery 31, 8 (Aug. 2007), 1545–1550. https://doi.org/10.1007/s00268-007-9076-5Google ScholarCross Ref
- Ignacio Avellino, Sheida Nozari, Geoffroy Canlorbe, and Yvonne Jansen. 2021. Surgical Video Summarization: Multifarious Uses, Summarization Process and Ad-Hoc Coordination. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (April 2021), 140:1–140:23. https://doi.org/10.1145/3449214Google ScholarDigital Library
- Andrius Budrionis, Per Hasvold, Gunnar Hartvigsen, and Johan Gustav Bellika. 2016. Assessing the impact of telestration on surgical telementoring: A randomized controlled trial. Journal of Telemedicine and Telecare 22, 1 (Jan. 2016), 12–17. https://doi.org/10.1177/1357633X15585071 Publisher: SAGE Publications.Google ScholarCross Ref
- Bennet A. Butler, Cameron M. Butler, and Terrance D. Peabody. 2019. Cognitive Apprenticeship in Orthopaedic Surgery: Updating a Classic Educational Model. Journal of Surgical Education 76, 4 (July 2019), 931–935. https://doi.org/10.1016/j.jsurg.2019.01.009Google ScholarCross Ref
- B. Chebbi, D. Lazaro, and P.X. Liu. 2007. A COLLABORATIVE VIRTUAL HAPTIC ENVIRONMENT FOR SURGICAL TRAINING AND TELE-MENTORING. International Journal of Robotics and Automation 22, 1 (2007). https://doi.org/10.2316/Journal.206.2007.1.206-1007Google ScholarCross Ref
- Xiaodong (Phoenix) Chen, Reed G. Williams, Hilary A. Sanfey, and Douglas S. Smink. 2015. A taxonomy of surgeons’ guiding behaviors in the operating room. The American Journal of Surgery 209, 1 (Jan. 2015), 15–20. https://doi.org/10.1016/j.amjsurg.2014.07.018Google ScholarCross Ref
- Andrew S. A. Chetwood, Ka-Wai Kwok, Loi-Wah Sun, George P. Mylonas, James Clark, Ara Darzi, and Guang-Zhong Yang. 2012. Collaborative eye tracking: a potential training tool in laparoscopic surgery. Surgical Endoscopy 26, 7 (July 2012), 2003–2009. https://doi.org/10.1007/s00464-011-2143-xGoogle ScholarCross Ref
- Herbert H. Clark and Deanna Wilkes-Gibbs. 1986. Referring as a collaborative process. Cognition 22, 1 (Feb. 1986), 1–39. https://doi.org/10.1016/0010-0277(86)90010-7Google ScholarCross Ref
- Alexandra C. Cope, Stella Mavroveli, Jeff Bezemer, George B. Hanna, and Roger Kneebone. 2015. Making Meaning From Sensory Cues: A Qualitative Investigation of Postgraduate Learning in the Operating Room. Academic Medicine 90, 8 (Aug. 2015), 1125–1131. https://doi.org/10.1097/ACM.0000000000000740Google ScholarCross Ref
- Arthur Derathé, Fabian Reche, Pierre Jannin, Alexandre Moreau-Gaudry, Bernard Gibaud, and Sandrine Voros. 2021. Explaining a model predicting quality of surgical practice: a first presentation to and review by clinical experts. International Journal of Computer Assisted Radiology and Surgery 16, 11 (Nov. 2021), 2009–2019. https://doi.org/10.1007/s11548-021-02422-0Google ScholarCross Ref
- A. Durand, D. Bourin, F. Borie, C. Castelli, J.-P. Daures, and J.-M. Kinowski. 2012. Utilisation de l’Ultracision® et du Ligasure® dans un CHU français : bon usage et évaluation économique. Le Pharmacien Hospitalier et Clinicien 47, 2 (2012), 116–122. https://doi.org/10.1016/j.phclin.2011.11.004Google ScholarCross Ref
- Simon Erridge, Derek K. T. Yeung, Hitendra R. H. Patel, and Sanjay Purkayastha. 2019. Telementoring of Surgeons: A Systematic Review. Surgical Innovation 26, 1 (Feb. 2019), 95–111. https://doi.org/10.1177/1553350618813250Google ScholarCross Ref
- Yuanyuan Feng, Hannah McGowan, Azin Semsar, Hamid R. Zahiri, Ivan M. George, Adrian Park, Andrea Kleinsmith, and Helena Mentis. 2019. Virtual pointer for gaze guidance in laparoscopic surgery. Surgical Endoscopy 34 (Oct. 2019), 3533–3539. https://doi.org/10.1007/s00464-019-07141-xGoogle ScholarCross Ref
- Yuanyuan Feng, Hannah McGowan, Azin Semsar, Hamid R. Zahiri, Ivan M. George, Adrian Park, Andrea Kleinsmith, and Helena Mentis. 2020. Virtual pointer for gaze guidance in laparoscopic surgery. Surgical Endoscopy 34, 8 (Aug. 2020), 3533–3539. https://doi.org/10.1007/s00464-019-07141-xGoogle ScholarCross Ref
- Yuanyuan Feng, Hannah McGowan, Azin Semsar, Hamid R. Zahiri, Ivan M. George, Timothy Turner, Adrian Park, Andrea Kleinsmith, and Helena M. Mentis. 2018. A virtual pointer to support the adoption of professional vision in laparoscopic training. International Journal of Computer Assisted Radiology and Surgery 13, 9 (Sept. 2018), 1463–1472. https://doi.org/10.1007/s11548-018-1792-9Google ScholarCross Ref
- Yuanyuan Feng and Helena M Mentis. 2018. Improving Common Ground Development in Surgical Training through Talk and Action. (2018), 10.Google Scholar
- Yuanyuan Feng, Christopher Wong, Adrian Park, and Helena Mentis. 2016. Taxonomy of instructions given to residents in laparoscopic cholecystectomy. Surgical Endoscopy 30, 3 (March 2016), 1073–1077. https://doi.org/10.1007/s00464-015-4300-0Google ScholarCross Ref
- Tomohiro Fukuda, Yoshihiro Tanaka, Astrid M.L. Kappers, Michitaka Fujiwara, and Akihito Sano. 2018. Visual and tactile feedback for a direct-manipulating tactile sensor in laparoscopic palpation. The International Journal of Medical Robotics and Computer Assisted Surgery 14, 2 (April 2018), e1879. https://doi.org/10.1002/rcs.1879Google ScholarCross Ref
- Susan R. Fussell, Robert E. Kraut, and Jane Siegel. 2000. Coordination of communication: effects of shared visual context on collaborative work. In Proceedings of the 2000 ACM conference on Computer supported cooperative work(CSCW ’00). Association for Computing Machinery, New York, NY, USA, 21–30. https://doi.org/10.1145/358916.358947Google ScholarDigital Library
- Susan R. Fussell, Leslie D. Setlock, and Robert E. Kraut. 2003. Effects of Head-mounted and Scene-oriented Video Systems on Remote Collaboration on Physical Tasks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’03). ACM, New York, NY, USA, 513–520. https://doi.org/10.1145/642611.642701Google ScholarDigital Library
- Susan R. Fussell, Leslie D. Setlock, Jie Yang, Jiazhi Ou, Elizabeth Mauer, and Adam D. I. Kramer. 2004. Gestures over Video Streams to Support Remote Collaboration on Physical Tasks. Hum.-Comput. Interact. 19, 3 (Sept. 2004), 273–309. https://doi.org/10.1207/s15327051hci1903_3Google ScholarDigital Library
- Danilo Gasques, Janet G. Johnson, Tommy Sharkey, Yuanyuan Feng, Ru Wang, Zhuoqun Robin Xu, Enrique Zavala, Yifei Zhang, Wanze Xie, Xinming Zhang, Konrad Davis, Michael Yip, and Nadir Weibel. 2021. ARTEMIS: A Collaborative Mixed-Reality System for Immersive Surgical Telementoring. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1–14. https://doi.org/10.1145/3411764.3445576Google ScholarDigital Library
- Darren Gergle, Robert E. Kraut, and Susan R. Fussell. 2013. Using Visual Information for Grounding and Awareness in Collaborative Tasks. Human–Computer Interaction 28, 1 (Jan. 2013), 1–39. https://doi.org/10.1080/07370024.2012.678246 Publisher: Taylor & Francis _eprint: https://www.tandfonline.com/doi/pdf/10.1080/07370024.2012.678246.Google ScholarCross Ref
- Keita Higuch, Ryo Yonetani, and Yoichi Sato. 2016. Can Eye Help You? Effects of Visualizing Eye Fixations on Remote Collaboration Scenarios for Physical Tasks. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems(CHI ’16). Association for Computing Machinery, New York, NY, USA, 5180–5190. https://doi.org/10.1145/2858036.2858438Google ScholarDigital Library
- Ken Hinckley, Randy Pausch, Dennis Proffitt, and Neal F. Kassell. 1998. Two-handed virtual manipulation. ACM Transactions on Computer-Human Interaction 5, 3 (Sept. 1998), 260–302. https://doi.org/10.1145/292834.292849Google ScholarDigital Library
- Stefan Hirschauer. 1991. The Manufacture of Bodies in Surgery. Social Studies of Science 21, 2 (May 1991), 279–319. https://doi.org/10.1177/030631291021002005 Publisher: SAGE Publications Ltd.Google ScholarCross Ref
- Anthony M. Jarc, Swar H. Shah, Troy Adebar, Eric Hwang, Monish Aron, Inderbir S. Gill, and Andrew J. Hung. 2016. Beyond 2D telestration: an evaluation of novel proctoring tools for robot-assisted minimally invasive surgery. Journal of Robotic Surgery 10, 2 (June 2016), 103–109. https://doi.org/10.1007/s11701-016-0564-1Google ScholarCross Ref
- Anthony M. Jarc, Andrew A. Stanley, Thomas Clifford, Inderbir S. Gill, and Andrew J. Hung. 2017. Proctors exploit three-dimensional ghost tools during clinical-like training scenarios: a preliminary study. World Journal of Urology 35, 6 (June 2017), 957–965. https://doi.org/10.1007/s00345-016-1944-xGoogle ScholarCross Ref
- Hyung-il Kim, Boram Yoon, Seo Young Oh, and Woontack Woo. 2023. Visualizing Hand Force with Wearable Muscle Sensing for Enhanced Mixed Reality Remote Collaboration. IEEE Transactions on Visualization and Computer Graphics 29, 11 (2023), 4611–4621. https://doi.org/10.1109/TVCG.2023.3320210Google ScholarDigital Library
- Seungwon Kim, Gun Lee, Weidong Huang, Hayun Kim, Woontack Woo, and Mark Billinghurst. 2019. Evaluating the Combination of Visual Communication Cues for HMD-based Mixed Reality Remote Collaboration. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, Glasgow Scotland Uk, 1–13. https://doi.org/10.1145/3290605.3300403Google ScholarDigital Library
- David Kirk, Tom Rodden, and Danaë Stanton Fraser. 2007. Turn it this way: grounding collaborative action with remote gestures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’07). Association for Computing Machinery, New York, NY, USA, 1039–1048. https://doi.org/10.1145/1240624.1240782Google ScholarDigital Library
- Robert E. Kraut, Susan R. Fussell, and Jane Siegel. 2003. Visual Information As a Conversational Resource in Collaborative Physical Tasks. Hum.-Comput. Interact. 18, 1 (June 2003), 13–49. https://doi.org/10.1207/S15327051HCI1812_2Google ScholarDigital Library
- Benjamin R. Lee, Jay T. Bishoff, Gunter Janetschek, Pichai Bunyaratevej, Wichean Kamolpronwijit, Jeffrey A. Cadeddu, Supoj Ratchanon, Shannon O’Kelley, and Louis R. Kavoussi. 1998. A novel method of surgical instruction: international telementoring. World Journal of Urology 16, 6 (Dec. 1998), 367–370. https://doi.org/10.1007/s003450050082Google ScholarCross Ref
- Gyusung I. Lee and Mija R. Lee. 2018. Can a virtual reality surgical simulation training provide a self-driven and mentor-free skills learning? Investigation of the practical influence of the performance metrics from the virtual reality robotic surgery simulator on the skill learning and associated cognitive workloads. Surgical Endoscopy 32, 1 (Jan. 2018), 62–72. https://doi.org/10.1007/s00464-017-5634-6Google ScholarCross Ref
- Akira Matsuda, Toru Okuzono, Hiromi Nakamura, Hideaki Kuzuoka, and Jun Rekimoto. 2021. A Surgical Scene Replay System for Learning Gastroenterological Endoscopic Surgery Skill by Multiple Synchronized-Video and Gaze Representation. Proceedings of the ACM on Human-Computer Interaction 5, EICS (May 2021), 1–22. https://doi.org/10.1145/3461726Google ScholarDigital Library
- Helena M. Mentis, Amine Chellali, and Steven Schwaitzberg. 2014. Learning to See the Body: Supporting Instructional Practices in Laparoscopic Surgical Procedures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’14). ACM, New York, NY, USA, 2113–2122. https://doi.org/10.1145/2556288.2557387Google ScholarDigital Library
- Helena M. Mentis, Yuanyuan Feng, Azin Semsar, and Todd A. Ponsky. 2020. Remotely Shaping the View in Surgical Telementoring. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems(CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376622Google ScholarDigital Library
- Lorenza Mondada. 2014. Instructions in the operating room: How the surgeon directs their assistant’s hands. Discourse Studies 16, 2 (April 2014), 131–161. https://doi.org/10.1177/1461445613515325Google ScholarCross Ref
- Alexander Mrokon, Peter P. Pott, and Volker Steger. 2019. Endoscopic Pan/Tilt Camera for Thorax Interventions – Design and first results. Current Directions in Biomedical Engineering 5, 1 (Sept. 2019), 517–519. https://doi.org/10.1515/cdbme-2019-0130Google ScholarCross Ref
- Ignacio Oropesa, Patricia Sánchez-González, Pablo Lamata, Magdalena K. Chmarra, José B. Pagador, Juan A. Sánchez-Margallo, Francisco M. Sánchez-Margallo, and Enrique J. Gómez. 2011. Methods and Tools for Objective Assessment of Psychomotor Skills in Laparoscopic Surgery. Journal of Surgical Research 171, 1 (Nov. 2011), e81–e95. https://doi.org/10.1016/j.jss.2011.06.034Google ScholarCross Ref
- Francesco Porpiglia, Enrico Checcucci, Daniele Amparore, Dario Peretti, Federico Piramide, Sabrina De Cillis, Alberto Piana, Gabriel Niculescu, Paolo Verri, Matteo Manfredi, Massimiliano Poggio, Ilaria Stura, Giuseppe Migliaretti, Marco Cossu, and Cristian Fiori. 2022. Percutaneous Kidney Puncture with Three-dimensional Mixed-reality Hologram Guidance: From Preoperative Planning to Intraoperative Navigation. European Urology 81, 6 (June 2022), 588–597. https://doi.org/10.1016/j.eururo.2021.10.023Google ScholarCross Ref
- Jason C. Pradarelli, Megan Delisle, Alexandra Briggs, Douglas S. Smink, and Steven Yule. 2021. Identifying Naturalistic Coaching Behavior Among Practicing Surgeons in the Operating Room. Annals of Surgery 273, 1 (Jan. 2021), 181–186. https://doi.org/10.1097/SLA.0000000000003368Google ScholarCross Ref
- Yu Saito, Maki Sugimoto, Satoru Imura, Yuji Morine, Tetsuya Ikemoto, Shuichi Iwahashi, Shinichiro Yamada, and Mitsuo Shimada. 2020. Intraoperative 3D Hologram Support With Mixed Reality Techniques in Liver Surgery. Annals of Surgery 271, 1 (Jan. 2020), e4–e7. https://doi.org/10.1097/SLA.0000000000003552Google ScholarCross Ref
- Giuseppe Scavo, Fridolin Wild, and Peter Scott. 2015. The GhostHands UX: telementoring with hands-on augmented reality instruction. (2015).Google Scholar
- Azin Semsar, Hannah McGowan, Yuanyuan Feng, H. Reza Zahiri, Adrian Park, Andrea Kleinsmith, and Helena Mentis. 2019. How Trainees Use the Information from Telepointers in Remote Instruction. Proc. ACM Hum.-Comput. Interact. 3, CSCW (Nov. 2019), 93:1–93:20. https://doi.org/10.1145/3359195Google ScholarDigital Library
- Azin Semsar, Hannah McGowan, Yuanyuan Feng, H. Reza Zahiri, Adrian Park, Andrea Kleinsmith, and Helena M. Mentis. 2020. Quality of and Attention to Instructions in Telementoring. Proceedings of the ACM on Human-Computer Interaction 4, CSCW2 (Oct. 2020), 165:1–165:21. https://doi.org/10.1145/3415236Google ScholarDigital Library
- Dehlela Shabir, Nihal Abdurahiman, Jhasketan Padhan, May Trinh, Shidin Balakrishnan, Mohamed Kurer, Omar Ali, Abdulla Al‐Ansari, Elias Yaacoub, Zhigang Deng, Aiman Erbad, Amr Mohammed, and Nikhil V. Navkar. 2021. Towards development of a tele‐mentoring framework for minimally invasive surgeries. The International Journal of Medical Robotics and Computer Assisted Surgery 17, 5 (Oct. 2021). https://doi.org/10.1002/rcs.2305Google ScholarCross Ref
- Dehlela Shabir, Shidin Balakrishnan, Jhasketan Padhan, Julien Abinahed, Elias Yaacoub, Amr Mohammed, Zhigang Deng, Abdulla Al-Ansari, Panagiotis Tsiamyrtzis, and Nikhil V. Navkar. 2023. Tele-Mentoring Using Augmented Reality: A Feasibility Study to Assess Teaching of Laparoscopic Suturing Skills. In 2023 IEEE 36th International Symposium on Computer-Based Medical Systems (CBMS). IEEE, L’Aquila, Italy, 61–66. https://doi.org/10.1109/CBMS58004.2023.00193Google ScholarCross Ref
- Mahesh B. Shenai, R. Shane Tubbs, Barton L. Guthrie, and Aaron A. Cohen-Gadol. 2014. Virtual interactive presence for real-time, long-distance surgical collaboration during complex microsurgical procedures: Technical note. Journal of Neurosurgery 121, 2 (Aug. 2014), 277–284. https://doi.org/10.3171/2014.4.JNS131805Google ScholarCross Ref
- Gary Sutkin, Eliza B. Littleton, and Steven L. Kanter. 2015. How Surgical Mentors Teach: A Classification of In Vivo Teaching Behaviors Part 2: Physical Teaching Guidance. Journal of Surgical Education 72, 2 (March 2015), 251–257. https://doi.org/10.1016/j.jsurg.2014.10.004Google ScholarCross Ref
- Bertrand Trilling, Adrian Mancini, Gaëlle Fiard, Pierre Alain Barraud, Marion Decrouez, Sinara Vijayan, Mathias Tummers, Jean Luc Faucheron, Sophie Silvent, Christel Schwartz, and Sandrine Voros. 2021. Improving vision for surgeons during laparoscopy: the Enhanced Laparoscopic Vision System (ELViS). Surgical Endoscopy 35, 5 (May 2021), 2403–2415. https://doi.org/10.1007/s00464-021-08369-2Google ScholarCross Ref
- Lucile Vadcard. 2022. Gestes Techniques et Formation – Un point de vue didactique. PUG et UGA Editions, Grenoble, France. https://www.decitre.fr/livres/gestes-techniques-et-formation-9782706151835.htmlGoogle Scholar
- Melina C. Vassiliou, Liane S. Feldman, Christopher G. Andrew, Simon Bergman, Karen Leffondré, Donna Stanbridge, and Gerald M. Fried. 2005. A global assessment tool for evaluation of intraoperative laparoscopic skills. The American Journal of Surgery 190, 1 (July 2005), 107–113. https://doi.org/10.1016/j.amjsurg.2005.04.004Google ScholarCross Ref
- Rémi Wolf, Josselin Duchateau, Philippe Cinquin, and Sandrine Voros. 2011. 3D Tracking of Laparoscopic Instruments Using Statistical and Geometric Modeling. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2011, Gabor Fichtinger, Anne Martel, and Terry Peters (Eds.). Vol. 6891. Springer Berlin Heidelberg, Berlin, Heidelberg, 203–210. https://doi.org/10.1007/978-3-642-23623-5_26 Series Title: Lecture Notes in Computer Science.Google ScholarCross Ref
- George J. Xeroulis, Jason Park, Carol-Anne Moulton, Richard K. Reznick, Vicki LeBlanc, and Adam Dubrowski. 2007. Teaching suturing and knot-tying skills to medical students: A randomized controlled study comparing computer-based video instruction and (concurrent and summary) expert feedback. Surgery 141, 4 (April 2007), 442–449. https://doi.org/10.1016/j.surg.2006.09.012Google ScholarCross Ref
- Yue Yu, George Bourantas, Benjamin Zwick, Grand Joldes, Tina Kapur, Sarah Frisken, Ron Kikinis, Arya Nabavi, Alexandra Golby, Adam Wittek, and Karol Miller. 2022. Computer simulation of tumour resection-induced brain deformation by a meshless approach. International Journal for Numerical Methods in Biomedical Engineering 38, 1 (2022), e3539. https://doi.org/10.1002/cnm.3539 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/cnm.3539.Google ScholarCross Ref
Index Terms
- Understanding Takeovers and Telestration in Laparoscopic Surgery to Inform Telementoring System Design
Recommendations
Understanding The Needs of Mentoring in Surgery to Guide the Design of Surgical Telementoring Systems
CSCWAll along their careers, surgeons learn through mentoring, operating on patients side-by-side with expert surgeons. However, access to experts is more and more challenging as they are often located far from learners. Telementoring can increase the reach ...
A Compact Modular Teleoperated Robotic System for Laparoscopic Surgery
Compared with traditional open surgery, minimally invasive surgical procedures reduce patient trauma and recovery time, but the dexterity of the surgeon in laparoscopic surgery is reduced owing to the small incisions, long instruments and limited ...
Cooperative robotic assistant for laparoscopic surgery: CoBRASurge
IROS'09: Proceedings of the 2009 IEEE/RSJ international conference on Intelligent robots and systemsSurgical robotic systems have a great impact on the application of laparoscopic surgeries in the operating room. The advent of externally operated surgical robots has prompted the first significant shift in general laparoscopic surgeries. Miniature in ...
Comments