Preliminary testing of an augmented reality headset as a DICOM viewer during autopsy

: Objective: Augmented reality techniques are being used in clinical medicine and have been suggested for use in forensic medicine as well. In this article, we investigate the feasibility of using a head mounted AR display for conducting virtual autopsies and for augmenting autopsy under realistic conditions. Materials and Methods: For this study we used the Microsoft HoloLens (Microsoft Corp., Redmond, USA) as augmented reality device and DICOM-Viewer. We conducted this study in 2 phases with 5 participants. In phase 1, we conducted virtual autopsies on PMCT datasets using the AR device. In phase 2, participants performed 6 AR augmented autopsies. The time needed was recorded and the participants were interviewed about this new concept after each autopsy. Results: In phase 1 (AR Virtopsy), all participants were able to determine the main pathology for all cases, requiring an average of 122 seconds per case. Participants who expressed that they are more comfortable with technology were usually faster. In phase 2 (AR autopsy), most interactions happened in the first half an hour of the internal examination of the autopsy, with the interactions becoming less frequent and shorter over time. Conclusion: While AR headsets might not be useful to read and analyse CT images, AR headsets could help to bring image data such as PMCT into the autopsy hall. In such a scenario, this technology seems to be most useful in the first half an hour of the internal examination during autopsy.


Introduction
Classical autopsy is one of the main tools of a forensic pathologist. A full autopsy including the external and internal examination can take several hours, depending on the complexity of the case.
At the Zurich Institute of Forensic Medicine, a post-mortem computed tomography (PMCT) is conducted for every corpse that is admitted to the Institute. PMCT-Data is analysed and reported by forensic imaging experts, board certified radiologists with a specialization in forensic imaging, at the morning briefing, prior to autopsy. Based on the imaging findings, autopsies can be planned optimally.
During autopsy, image data is available through a dedicated computer system positioned in the autopsy hall. This helps both the localization of a pathology or an intracorporal foreign body during autopsy and further planning of the autopsy procedure. Interacting with a computer system during autopsy has the disadvantage of interrupting the autopsy in order to operate the computer.
Several solutions for this issue have been presented in the past. Ebert et al. showed that gesture control systems allow for a contact-less interaction with a computer [1]. While this removes the issue of contamination of input devices, the computer screen is not typically positioned in close vicinity to the pathologist for the same reason. The second solution is the application of augmented reality (AR) techniques. Kilgus et al. presented a system, where an Ipad is combined with a depth camera, and a rendering of the CT scan is overlaid to the camera images of the tablet in real-time [2]. This brings the screen to the pathologist, but has contamination issues. Affolter et al. presented a system, that combines the advantages of gesture control and augmented reality by utilizing a head mounted augmented reality display [3] .
Augmented reality has also been suggested to be used in clinical medicine in order to solve contamination problems similar to the ones in forensic medicine. Grasso et al. found that, in addition, the time required for an intervention using AR can be reduced [4]. Agten et al. used AR techniques for lumbar facet joint injections and came to a similar conclusion [5]. In pathology this concept has also been studied to redefine education. Hanna et al. described the use of AR technology during autopsy providing real-time annotations and voice instruction for supervision and teaching purpose. [6].
In this article, we investigate the feasibility of using a head mounted AR display for conducting virtual autopsies and for augmenting autopsy under realistic conditions.

Material and methods
The setup consisted of 2 hardware components and 2 software components. The hardware components were the computer and the HoloLens augmented reality headset (Microsoft Corp., Redmond, USA). The standard laptop computer was used to upload the image data to the headset. The software components were 2 programs which we used to upload to and display the data on the headset. Affolter et al. provides a more detailed description of the system [3].

Hardware
In this study we used 2 Microsoft HoloLenses as an Augmented Reality Headset, which were equipped with a protective foil. Additionally, we used the Lenovo X1 Yoga ThinkPad (Lenovo, Quarry Bay, Hong Kong) to upload the Digital Information and Communication in Medicine (DICOM)-Data on the HoloLens. The laptop we used runs under Windows 10 x64 (Microsoft Corp. Redmond, USA) with an Intel Core i7-7600 2.80 GHz and 8GB of RAM. PMCT data was generated using a Siemens Somatom Definition Flash (Siemens Healthineers, Erlangen, Germany) using the standard protocol described by Flach et al. [7]. Autopsies were filmed using a Nikon D800 camera (Nikon, Chiyoda, Tokio, Japan) on a tripod.

Software
In order to visualize DICOM-data on the HoloLens, a custom-made two-part software was used. The first part (ViHo connector), the virtopsy connector, converts the DICOM-data into jpg files after applying different window settings. Furthermore, the software establishes a connection with the HoloLens and loads the converted DICOM-Data onto the headset. The second part (ViHo) is an app that is running on the AR headset. The ViHo app allows us to visualize the uploaded data using hand and figure gestures. The system provides a variety of gestures for interaction, such as browsing the dataset or changing window levels.

Study design
The study consisted of two phasesusing augmented reality for reading radiological images (augmented virtual autopsy) in phase one and a second phase, where the augmented reality headset was utilized to provide radiological images during autopsy (augmented autopsy). For augmented virtual autopsy, our study population consisted of 3 residents without board certification in forensic medicine, a forensic technician and a forensic nurse, all of them with no experience in either forensic -or clinical radiology. For augmented autopsy, 1 resident and the forensic nurse dropped out of the study.
Prior to the study, all participants were given a standardized introduction into using the AR headset and the required gestures.

Phase 1: Augmented virtual autopsy
Augmented Virtual Autopsy was carried out over a period of five months, with 5 sessions. For the augmented virtual autopsies, we chose 10 cases (Thorax/Abdomen) from the archives, which had a defined lesion/diagnosis (Table 1), confirmed by autopsy. For each session, 2 cases were shown to each participant. Because 2 HoloLens headsets were available, 2 participants could use the system simultaneously. Cases were randomly selected from our case database.
For testing the AR headset for virtual autopsy, we used the Thorax/ Abdomen scans, showing a variety of pathologies like gunshot wounds, internal bleeding or mechanical ileus. All the participants were presented the exact same chronological order of cases. After uploading the data to the AR headset, the headset was handed to the participant in default mode, without any running apps. We asked the participants to identify the main pathology using the AR headset and we measured the time required in order to quantify training effects, starting the start of the ViHo app. It is important to note, that it was not the aim to conduct a full reading of the images but rather the quick identification of the main finding. Participants were allowed to ask questions during this phase. Finally, feedback in form of a verbal interview was gathered with the participants after each session.

Phase 2: VR augmented autopsy
Over the course of 4 months after phase 1, cases would be selected during the morning briefing. For all of these cases, PMCT images were available. Exclusion criteria were dismembered (e.g. train accident) or skeletonized bodies. Suitable cases, when one of our participants was conducting the autopsy, were selected in agreement with the head of the department. The Thorax/Abdomen scan was extracted from the Picture Archiving and Communications System (PACS), anonymised and uploaded to the AR headset. Loading the image data to the AR headset took another 2-3 minutes. Then a protection foil was attached to the HoloLens to protect its lenses from contamination.
After preparing the AR headset, a video camera was installed in the autopsy hall, recording the pathologist conducting the autopsy and his interaction with the AR device. The AR headset was handed over to the pathologist. We recorded the autopsy from the time the headset was used for the first time until it was removed. This means, that usually not the entire autopsy was recorded. Especially the external examination of the body which takes place at the beginning of the autopsy was not part of the study. Figure 1 demonstrates what the participant sees through the AR device in our study and shows the setup we used in the autopsy hall.

Evaluation
In order to evaluate the training, we measured the time required to determine the main pathology during training for all 10 cases. In addition, we asked for general feedback regarding the use of the AR headset.
In order to analyse the videos, we added a timecode using DaVinci Resolve 16 (Blackmagic Design, Port Melbourne, Australia). We used VideoLAN (École Centrale Paris, Paris, France) to further analyze the videos. We recorded every interaction with the AR device. An interaction is either an obvious gaze at the virtual image or an interaction using gestures. We also recorded issues that were visible, such as undetected gestures that had to be repeated or a readjustment of the headsets during autopsy. For statistical analysis the average and standard deviation were calculated for Phase 1 and 2.
After the autopsy, the participants were questioned about the comfort, the usability and handiness.
To collect and illustrate the results of our study we used Microsoft Excel (Microsoft Corp., Redmond, USA). Statistical analysis was carried out using SPSS (IBM, Armonk, USA).

Augmented virtual autopsy
Solving the Virtual autopsy cases using AR took from 28 seconds up to 214 seconds with an average of 128 seconds and 13.8 seconds of standard deviation. The fastest case was solved in an average of 122 seconds (+-39 seconds), while the longest case took 144 seconds (+-52 seconds). The fastest participant (28 seconds) had also the lowest mean time (114 seconds) with a standard deviation of 52.9 seconds. The slowest one (214 seconds) had as expected also the highest mean time (144 seconds) with a standard deviation of 52.9 seconds. There was a tendency, that participants who expressed themselves as more technology affine performed faster.

Augmented autopsy
3 participants who concluded the training also conducted autopsies using the AR headset: 2 forensic pathologist and 1 forensic technician. To be specific the forensic technician assisted the forensic pathologists while using the AR headset. We recorded a total of 6 autopsies, amounting to 10 hours and 58 minutes of recording. The first forensic pathologist performed 3 augmented reality enhanced autopsies, the second one 2 and the forensic technician 1.
The AR headset was mounted between 47 minutes to 148 minutes and was actively used on average 118 seconds ranging from 34 seconds to 165 seconds per autopsy. While an autopsy lasted on average 1 hour and 50 minutes the AR device was used on purpose in 0.012% of the time.
We recorded a total of 39 interactions in total, ranging from 1 interaction to 18 interactions per autopsy. A total of 12 interaction took around 1 second, representing a brief look at the currently displayed image. Figure 2 displays the time point and the duration of the interactions. There is a tendency, that at the beginning of the internal examination of an autopsy, interactions are longer and occur more often. After about 30 minutes into the internal examination, interactions become rare.
While operating the AR headset, a couple of smaller issues arose, that were solved quickly. In one case, a gesture had to be repeated several times and in another cases, the protective plastic sheet got loose and had to be replaced during autopsy.
During training and autopsy, participants raised a couple of issues. According to one participant, the field of view was too small. Another participant commented on the weight of the device which was considered to be uncomfortable and caused a headache in one case, so the device hat to be removed prematurely. One participant commented on the scrolling speed of the software being too fast. In general, the technology was perceived to be useful, considering the current technological drawbacks.

Discussion
In this study, we investigated the feasibility of using an AR headset to display PMCT data during virtual autopsy and autopsy under realistic conditions.
The system showed no issues in phase 1, the solving of virtual autopsy cases in VR. In general, we could see, that participants who are affine to technology and have more experience with PMCT image data performed better during this phase, which is expected. In phase 2, during augmented autopsy, we could see, that at the beginning of the internal examination during autopsy, interactions with the AR headset are more frequent and have a longer duration. This can be explained by the way, an autopsy is conducted. At the beginning, image data is used for orientation and autopsy planning, hence the interactions are longer at the beginning. Once a relevant slice is selected, only short gazes or a change in window settings are required. During organ dissection, frequency of usage goes down or even stops. Most of the interactions we observed happened in the first half an hour of the internal examination during autopsy. In general, the device works as well during augmented autopsy conditions as it did during augmented virtual autopsy. Wet rubber gloves as well as the mounted plastic cover did not affect the finger tracking capabilities of the device. While the software is limited to axial slices with predefined windows and does not provide any 3D volume rendering capabilities, the provided functionality seemed to be sufficient.
There are however a couple of limitations when using the AR headset in an autopsy scenario, most of them due to technical limitations of the headset. The device is not instantly available, uploading the image data can take approximately 15 minutes. The device only offers a limited field of view, requiring the user to place the image at a sufficient distance in order to look at an entire CT slice. The weight of the device caused discomfort for one user after some time. Furthermore, we could not record the participants field of view during autopsy, which would have made a more accurate evaluation possible, because this would have severely slowed down the framerate. These technical limitations could be removed with the release of the next generation of AR headsets. As a replacement for a computer screen for virtual autopsy, an AR headset is probably not suited due to muscle fatigue when used over a longer period of time and the lower display quality. Using a much lighter headset like the Google Glass [Googel LLC, Mountain View, California, USA] could resolve the muscle fatigue issue and the discomfort but would not be of any help regarding the low display quality. During autopsy however, additional benefits, such as the ability to place the working window anywhere in the field of view or the ability to operate the device with contaminated gloves, could make AT suitable for the autopsy room, provided some of the technical limitations are eliminated.
There are also some limitations to the study design. For the augmented autopsy tests of phase 2, the number of cases is too low to make any proper statistical analysis. Because of this, we cannot make a recommendation, what types of cases would benefit most of using the AR headset. The study group is heterogeneous and changed from phase 1 to phase 2. Because none of the participants had prior experience of using AR during actual autopsy, this study might not fully represent an actual use.
Future studies should include more cases to make a proper statistical analysis. It should also include more cases with a defined lesion for the autopsy. We think it could be interesting to include cases where a foreign material must be found. Future work should also investigate the possibility of a new teaching approach. Using the recording function and the handless property of the HoloLens can allow streaming the video feed into a lecture hall whilst demonstrating any structure.

Conclusion
While AR headsets might not be useful to read CT images in the course of a real autopsy, AR headsets could help to bring image data such as PMCT into the autopsy hall. In such a scenario, this technology seems to be most useful in the first half an hour of the internal examination during autopsy.

Contributions of the authors
The software used in the study was already programmed by Raffael Affolter. With this software I have developed a standardized training. Dominic Gascho selected and prepared the imaging data. With Till Siebert and Lars Ebert I developed the study protocol. I planned the organizational part of both phases with Sebastian Eggert. I conducted the study independently and evaluated the data. Garyfalia Ampanozi was involved in the statistics. Michael Thali helped me with the implementation of the study. I wrote the paper with individual contributions and input from the co-authors. I submitted the paper to the journal.