Danilo Gasques
Janet G. Johnson
Nadir Weibel
ABSTRACT
Traumatic injuries require timely intervention, but medical expertise is not always available at the patient’s location. Despite recent advances in telecommunications, surgeons still have limited tools to remotely help inexperienced surgeons. Mixed Reality hints at a future where remote collaborators work side-by-side as if co-located; however, we still do not know how current technology can improve remote surgical collaboration. Through role-playing and iterative-prototyping, we identify collaboration practices used by expert surgeons to aid novice surgeons as well as technical requirements to facilitate these practices. We then introduce ARTEMIS, an AR-VR collaboration system that supports these key practices. Through an observational study with two expert surgeons and five novice surgeons operating on cadavers, we find that ARTEMIS supports remote surgical mentoring of novices through synchronous point, draw, and look affordances and asynchronous video clips. Most participants found that ARTEMIS facilitates collaboration despite existing technology limitations explored in this paper.
- Deepak Akkil, Jobin Mathew James, Poika Isokoski, and Jari Kangas. 2016. GazeTorch: Enabling gaze awareness in collaborative physical tasks. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. 1151–1158.Google Scholar
Digital Library
- Judith Amores, Xavier Benavides, and Pattie Maes. 2015. Showme: A remote collaboration system that supports immersive gestural communication. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. 1343–1348.Google ScholarDigital Library
- Daniel Andersen, Voicu Popescu, Maria Eugenia Cabrera, Aditya Shanghavi, Gerardo Gomez, Sherri Marley, Brian Mullis, and Juan Wachs. 2016. Virtual annotations of the surgical field through an augmented reality transparent display. The Visual Computer 32, 11 (2016), 1481–1498.Google ScholarDigital Library
- Daniel Andersen, Voicu Popescu, Maria Eugenia Cabrera, Aditya Shanghavi, Gerardo Gomez, Sherri Marley, Brian Mullis, and Juan P Wachs. 2016. Medical telementoring using an augmented reality transparent display. Surgery 159, 6 (2016), 1646–1653.Google ScholarCross Ref
- Rahul Arora, Rubaiat Habib Kazi, Fraser Anderson, Tovi Grossman, Karan Singh, and George W Fitzmaurice. 2017. Experimental Evaluation of Sketching on Surfaces in VR.. In CHI, Vol. 17. 5643–5654.Google Scholar
- Mark Billinghurst and Hirokazu Kato. 2002. Collaborative augmented reality. Commun. ACM 45, 7 (2002), 64–70.Google ScholarDigital Library
- Etai M Bogen, Knut M Augestad, Hiten RH Patel, and Rolv-Ole Lindsetmo. 2014. Telementoring in education of laparoscopic surgeons: An emerging technology. World journal of gastrointestinal endoscopy 6, 5 (2014), 148.Google Scholar
- Andrius Budrionis, Knut Magne Augestad, Hiten RH Patel, and Johan Gustav Bellika. 2013. An evaluation framework for defining the contributions of telestration in surgical telementoring. Interactive journal of medical research 2, 2 (2013), e14.Google Scholar
- Frank K Butler. 2010. Tactical combat casualty care: update 2009. Journal of Trauma and Acute Care Surgery 69, 1 (2010), S10–S13.Google ScholarCross Ref
- Frank K Butler Jr, John Hagmann, and E George Butler. 1996. Tactical combat casualty care in special operations. Military medicine 161, suppl_1 (1996), 3–16.Google Scholar
- Henry Chen, Austin S Lee, Mark Swift, and John C Tang. 2015. 3D collaboration method over HoloLens™ and Skype™ end points. In Proceedings of the 3rd International Workshop on Immersive Media Experiences. 27–30.Google ScholarDigital Library
- Long Chen, Thomas W Day, Wen Tang, and Nigel W John. 2017. Recent Developments and Future Challenges in Medical Mixed Reality. In 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 123–135.Google Scholar
- Lung-Pan Cheng, Thijs Roumen, Hannes Rantzsch, Sven Köhler, Patrick Schmidt, Robert Kovacs, Johannes Jasper, Jonas Kemper, and Patrick Baudisch. 2015. TurkDeck: Physical Virtual Reality Based on People. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. 417–426.Google ScholarDigital Library
- Sebastiano Chiodini, Marco Pertile, Riccardo Giubilato, Federico Salvioli, Marco Barrera, Paola Franceschetti, and Stefano Debei. 2018. Camera rig extrinsic calibration using a motion capture system. In 2018 5th IEEE International Workshop on Metrology for AeroSpace (MetroAeroSpace). IEEE, 590–595.Google ScholarCross Ref
- Charles R Doarn and Rifat Latifi. 2016. Telementoring and teleproctoring in trauma and emergency care. Current Trauma Reports 2, 3 (2016), 138–143.Google ScholarCross Ref
- Barrett Ens, Joel Lanir, Anthony Tang, Scott Bateman, Gun Lee, Thammathip Piumsomboon, and Mark Billinghurst. 2019. Revisiting collaboration through mixed reality: The evolution of groupware. International Journal of Human-Computer Studies 131 (2019), 81–98.Google ScholarDigital Library
- Yuanyuan Feng, Christopher Wong, Adrian Park, and Helena Mentis. 2016. Taxonomy of instructions given to residents in laparoscopic cholecystectomy. Surgical endoscopy 30, 3 (2016), 1073–1077.Google Scholar
- Susan R Fussell, Leslie D Setlock, Jie Yang, Jiazhi Ou, Elizabeth Mauer, and Adam DI Kramer. 2004. Gestures over video streams to support remote collaboration on physical tasks. Human-Computer Interaction 19, 3 (2004), 273–309.Google ScholarDigital Library
- Danilo Gasques, Janet Johnson, Thomas Sharkey, and Nadir Weibel. 2019. What You Sketch Is What You Get: Quick and Easy Augmented Reality Prototyping with PintAR. In Proc. CHI 2019(CHI EA ’19).Google ScholarDigital Library
- Steffen Gauglitz, Benjamin Nuernberger, Matthew Turk, and Tobias Höllerer. 2014. World-stabilized annotations and virtual scene navigation for remote collaboration. In Proceedings of the 27th annual ACM symposium on User interface software and technology. 449–459.Google ScholarDigital Library
- Darren Gergle, Robert E Kraut, and Susan R Fussell. 2004. Action as language in a shared visual space. In Proceedings of the 2004 ACM conference on Computer supported cooperative work. 487–496.Google ScholarDigital Library
- HTC. 2020. VIVE Pro Starter Kit | The professional-grade VR headset. https://www.vive.com/us/product/vive-pro-starter-kit/Google Scholar
- Weidong Huang, Seungwon Kim, Mark Billinghurst, and Leila Alem. 2019. Sharing hand gesture and sketch cues in remote collaboration. Journal of Visual Communication and Image Representation 58 (2019), 428–438.Google ScholarCross Ref
- Intel. 2020. Intel RealSense Depth Camera D435. https://www.intelrealsense.com/Google Scholar
- Janet G Johnson, Danilo Gasques Rodrigues, Madhuri Gubbala, and Nadir Weibel. 2018. Holocpr: Designing and evaluating a mixed reality interface for time-critical emergencies. In Proceedings of the 12th EAI International Conference on Pervasive Computing Technologies for Healthcare. 67–76.Google ScholarDigital Library
- Rohana Abdul Karim, Nor Farizan Zakaria, Mohd Asyraf Zulkifley, Mohd Marzuki Mustafa, Ismail Sagap, and Nani Harlina Md Latar. 2013. Telepointer technology in telemedicine: a review. Biomedical engineering online 12, 1 (2013), 21.Google Scholar
- Seungwon Kim, Gun Lee, Mark Billinghurst, and Weidong Huang. 2020. The combination of visual communication cues in mixed reality remote collaboration. Journal on Multimodal User Interfaces 14, 4 (2020), 321–335.Google ScholarCross Ref
- Robert E Kraut, Darren Gergle, and Susan R Fussell. 2002. The use of visual information in shared visual spaces: Informing the development of virtual co-presence. In Proceedings of the 2002 ACM conference on Computer supported cooperative work. 31–40.Google ScholarDigital Library
- Bernard C. Kress and William J. Cummings. 2017. Invited Paper: Towards the Ultimate Mixed Reality Experience: HoloLens Display Architecture Choices. SID Symposium Digest of Technical Papers 48, 1 (2017), 127–131.Google Scholar
- Gun A. Lee, Theophilus Teo, Seungwon Kim, and Mark Billinghurst. 2018. A User Study on MR Remote Collaboration Using Live 360 Video. In 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 153–164.Google ScholarCross Ref
- Chengyuan Lin, Daniel Andersen, Voicu Popescu, Edgar Rojas-Munoz, Maria Eugenia Cabrera, Brian Mullis, Ben Zarzaur, Kathryn Anderson, Sherri Marley, and Juan Wachs. 2018. A first-person mentee second-person mentor AR interface for surgical telementoring. In 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). IEEE, 3–8.Google ScholarCross Ref
- Stephan Lukosch, Mark Billinghurst, Leila Alem, and Kiyoshi Kiyokawa. 2015. Collaboration in augmented reality. Computer Supported Cooperative Work (CSCW) 24, 6 (2015), 515–525.Google ScholarDigital Library
- JA Martin, Glenn Regehr, Richard Reznick, Helen Macrae, John Murnaghan, Carol Hutchison, and M Brown. 1997. Objective structured assessment of technical skill (OSATS) for surgical residents. British journal of surgery 84, 2 (1997), 273–278.Google Scholar
- Microsoft. 2020. Azure Kinect - Fast Point Cloud. https://github.com/microsoft/Azure-Kinect-Sensor-SDK/tree/develop/examples/fastpointcloudGoogle Scholar
- Microsoft. 2020. Azure Kinect DK hardware specifications. https://docs.microsoft.com/en-us/azure/kinect-dk/hardware-specificationGoogle Scholar
- Microsoft. 2020. HoloLens (1st gen) hardware. https://docs.microsoft.com/en-us/hololens/hololens1-hardwareGoogle Scholar
- Microsoft. 2020. MixedReality-WebRTC. https://github.com/microsoft/MixedReality-WebRTCGoogle Scholar
- Microsoft. 2020. Overview of Dynamics 365 Remote Assist on HoloLens and HoloLens 2. https://docs.microsoft.com/en-us/dynamics365/mixed-reality/remote-assist/overview-hololensGoogle Scholar
- Paul Milgram, Haruo Takemura, Akira Utsumi, and Fumio Kishino. 1995. Augmented reality: A class of displays on the reality-virtuality continuum. In Telemanipulator and telepresence technologies, Vol. 2351. International Society for Optics and Photonics, 282–292.Google Scholar
- Bonnie A Nardi, Heinrich Schwarz, Allan Kuchinsky, Robert Leichner, Steve Whittaker, and Robert Sclabassi. 1993. Turning away from talking heads: The use of video-as-data in neurosurgery. In Proceedings of the INTERACT’93 and CHI’93 Conference on Human Factors in Computing Systems. 327–334.Google ScholarDigital Library
- Ohan Oda, Carmine Elvezio, Mengu Sukan, Steven Feiner, and Barbara Tversky. 2015. Virtual replicas for remote assistance in virtual and augmented reality. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. 405–415.Google ScholarDigital Library
- OptiTrack. 2020. Motive - Optical motion capture software. https://optitrack.com/products/motive/Google Scholar
- OptiTrack. 2020. OptiTrack - Active Components. https://optitrack.com/products/active-components/Google Scholar
- OptiTrack. 2020. OptiTrack - Hardware. https://optitrack.com/hardware/Google Scholar
- Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L Davidson, Sameh Khamis, Mingsong Dou, 2016. Holoportation: Virtual 3d teleportation in real-time. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 741–754.Google ScholarDigital Library
- Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L. Davidson, Sameh Khamis, Mingsong Dou, Vladimir Tankovich, Charles Loop, Qin Cai, Philip A. Chou, Sarah Mennicken, Julien Valentin, Vivek Pradeep, Shenlong Wang, Sing Bing Kang, Pushmeet Kohli, Yuliya Lutchyn, Cem Keskin, and Shahram Izadi. 2016. Holoportation: Virtual 3D Teleportation in Real-time. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 741–754.Google ScholarDigital Library
- Antti Oulasvirta, Esko Kurvinen, and Tomi Kankainen. 2003. Understanding contexts by being there: case studies in bodystorming. Personal and ubiquitous computing 7, 2 (2003), 125–134.Google Scholar
- Nitish Padmanaban, Robert Konrad, Tal Stramer, Emily A Cooper, and Gordon Wetzstein. 2017. Optimizing virtual reality for all users through gaze-contingent and adaptive focus displays. Proceedings of the National Academy of Sciences 114, 9 (2017), 2183–2188.Google ScholarCross Ref
- Thammathip Piumsomboon, Arindam Day, Barrett Ens, Youngho Lee, Gun Lee, and Mark Billinghurst. 2017. Exploring enhancements for remote mixed reality collaboration. In SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications. 1–5.Google Scholar
- Thammathip Piumsomboon, Gun A Lee, Jonathon D Hart, Barrett Ens, Robert W Lindeman, Bruce H Thomas, and Mark Billinghurst. 2018. Mini-me: An adaptive avatar for mixed reality remote collaboration. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–13.Google ScholarDigital Library
- Edgar Rojas-Muñoz, Maria E Cabrera, Chengyuan Lin, Daniel Andersen, Voicu Popescu, Kathryn Anderson, Ben L Zarzaur, Brian Mullis, and Juan P Wachs. 2020. The System for Telementoring with Augmented Reality (STAR): A head-mounted display to improve surgical coaching and confidence in remote areas. Surgery (2020).Google Scholar
- Edgar Rojas-Muñoz, Maria Eugenia Cabrera, Chengyuan Lin, Natalia Sánchez-Tamayo, Dan Andersen, Voicu Popescu, Kathryn Anderson, Ben Zarzaur, Brian Mullis, and Juan P Wachs. 2020. Telementoring in Leg Fasciotomies via Mixed-Reality: Clinical Evaluation of the STAR Platform. Military Medicine 185, Supplement_1 (2020), 513–520.Google ScholarCross Ref
- Azin Semsar, Hannah McGowan, Yuanyuan Feng, H Reza Zahiri, Adrian Park, Andrea Kleinsmith, and Helena Mentis. 2019. How Trainees Use the Information from Telepointers in Remote Instruction. Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019), 1–20.Google ScholarDigital Library
- Dave Snowdon, Elizabeth F Churchill, and Alan J Munro. 2001. Collaborative virtual environments: Digital spaces and places for CSCW: An introduction. In Collaborative virtual environments. Springer, 3–17.Google Scholar
- Maurício Sousa, Rafael Kufner dos Anjos, Daniel Mendes, Mark Billinghurst, and Joaquim Jorge. 2019. WARPING DEIXIS: Distorting Gestures to Enhance Collaboration. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.Google ScholarDigital Library
- Dag Svanaes and Gry Seland. 2004. Putting the users center stage: role playing and low-fi prototyping enable end users to design mobile systems. In Proceedings of the SIGCHI conference on Human factors in computing systems. 479–486.Google ScholarDigital Library
- Theophilus Teo, Louise Lawrence, Gun A Lee, Mark Billinghurst, and Matt Adcock. 2019. Mixed Reality Remote Collaboration Combining 360 Video and 3D Reconstruction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–14.Google ScholarDigital Library
- Balasaravanan Thoravi Kumaravel, Fraser Anderson, George Fitzmaurice, Bjoern Hartmann, and Tovi Grossman. 2019. Loki: Facilitating Remote Instruction of Physical Tasks Using Bi-Directional Mixed-Reality Telepresence. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 161–174.Google ScholarDigital Library
- Ultraleap. 2020. Tracking | Ultraleap. https://www.ultraleap.com/tracking/Google Scholar
- Unity. 2020. Unity Real-Time Development Platform | 3D, 2D VR & AR Visualizations. https://unity.com/Google Scholar
- Reid Vassallo, Adam Rankin, Elvis CS Chen, and Terry M Peters. 2017. Hologram stability evaluation for Microsoft HoloLens. In Medical Imaging 2017: Image Perception, Observer Performance, and Technology Assessment, Vol. 10136. International Society for Optics and Photonics, 1013614.Google Scholar
- Senso VR. 2020. Senso VR | Interactive virtual and augmented reality. https://senso.me/Google Scholar
- Vuforia. 2020. Vuforia Developer Portal. https://www.vuforia.comGoogle Scholar
Index Terms
- ARTEMIS: A Collaborative Mixed-Reality System for Immersive Surgical Telementoring
Index terms have been assigned to the content through auto-classification.
Recommendations
ARTEMIS: Mixed-Reality Environment for Immersive Surgical Telementoring
CHI EA '20: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing SystemsThe golden hour following a traumatic injury holds the highest likelihood where surgical treatment may prevent mortality and morbidity. However, the increasing occurrence of large-scale disasters, overwhelm local medical systems and patients find ...
Depth cues in augmented reality for training of robot-assisted minimally invasive surgery
AcademicMindtrek '17: Proceedings of the 21st International Academic Mindtrek ConferenceTraining of robot-assisted minimally invasive surgery often includes supervised practice with a robotic surgical system. In this case, augmented reality can improve the communication between instructor and trainee, for example, by allowing the ...
A mixed reality application for total hip arthroplasty
AbstractTotal hip arthroplasty (or total hip replacement) is the current surgical solution for the treatment of advanced coxarthrosis, with the objective of providing mobility and pain relief to patients. For this purpose, surgery can be planned using ...
Comments