Ultrasound in augmented reality: a mixed-methods evaluation of head-mounted displays in image-guided interventions

Purpose Augmented reality (AR) and head-mounted displays (HMD) in medical practice are current research topics. A commonly proposed use case of AR-HMDs is to display data in image-guided interventions. Although technical feasibility has been thoroughly shown, effects of AR-HMDs on interventions are not yet well researched, hampering clinical applicability. Therefore, the goal of this study is to better understand the benefits and limitations of this technology in ultrasound-guided interventions. Methods We used an AR-HMD system (based on the first-generation Microsoft Hololens) which overlays live ultrasound images spatially correctly at the location of the ultrasound transducer. We chose ultrasound-guided needle placements as a representative task for image-guided interventions. To examine the effects of the AR-HMD, we used mixed methods and conducted two studies in a lab setting: (1) In a randomized crossover study, we asked participants to place needles into a training model and evaluated task duration and accuracy with the AR-HMD as compared to the standard procedure without visual overlay and (2) in a qualitative study, we analyzed the user experience with AR-HMD using think-aloud protocols during ultrasound examinations and semi-structured interviews after the task. Results Participants (n = 20) placed needles more accurately (mean error of 7.4 mm vs. 4.9 mm, p = 0.022) but not significantly faster (mean task duration of 74.4 s vs. 66.4 s, p = 0.211) with the AR-HMD. All participants in the qualitative study (n = 6) reported limitations of and unfamiliarity with the AR-HMD, yet all but one also clearly noted benefits and/or that they would like to test the technology in practice. Conclusion We present additional, though still preliminary, evidence that AR-HMDs provide benefits in image-guided procedures. Our data also contribute insights into potential causes underlying the benefits, such as improved spatial perception. Still, more comprehensive studies are needed to ascertain benefits for clinical applications and to clarify mechanisms underlying these benefits. Electronic supplementary material The online version of this article (10.1007/s11548-020-02236-6) contains supplementary material, which is available to authorized users.


Original version in German -English translation below
Categories are denoted as letters (N, L, …), subcategories as digits following the category letter. The first subcategory of each category defines the general category and is used when a phrase belongs to the category, but not to any of the defined subcategories.
Participants are referred to as P and a number, for example P2 or P4.

English Translation
Categories are denoted as letters (N, L, …), subcategories as digits following the category letter. The first subcategory of each category defines the general category and is used when a phrase belongs to the category, but not to any of the defined subcategories.
Participants are referred to as P and a number, for example P2 or P4.

Description Examples Differentiations to similar categories N1: Benefits
Coded when a concrete or directly experienced/perceived benefit is described Speculative or predicted benefits (e.g. for other applications and for other users) are coded as assessments, as are unspecific positive statements N2: Navigation facilitated HMD facilitates the localization of the ultrasound image in the anatomy and/or the transducer on the patient "[…] makes it much easier to locate the ultrasound image in the body." (P4) "so it was much easier with the glasses to visually confirm [...] where the transducer is located on the patient" (P3) For this category, only the effect on individual ultrasound images is relevant; if several images are combined or spatial correlation is required, N3 is to be coded N3: Improved 3D understanding HMD facilitates 3D understanding of anatomical structures and the correlation of several images "this is actually kind of a 3Dfeeling, which otherwise is only virtually reconstructed with a monitor, basically" (P3)

N4: Hedonistic aspects
Phrases expressing enjoyment, amazement or novelty value of technology "So it's definitely fun" (P1) "That really is cool" (P1) Using haptic input elements of the ultrasound system is harder when using the HMD "but it was more difficult then to operate the system on the side […]" (P3)

L5: Movement required
Visualization through the HMD requires users to move to see some views "that's a bit exhausting right now, that you have to rotate your head like this so that you're always really aligned to the plane of the ultrasoun image" (P5) Not coded if movement was not executed ("I would need to move")this rather belongs to cartegory L2

L6: Finding views with HMD is difficult
Certain ultrasound views/planes cannot be appropriately viewed with the HMD "I really tried but I can't get this with the HMD." (P2)

L7: Optical Tracking
The required line of sight between the trackers and the 3D camera limits the user in performing the task "but if I try to point the spheres to the 3D camera like this, I can't see anything on the patient anymore " (P1) If limitations occur due to both tracking and angle of view, code L7

L8: Image quality
Resolution, contrast or latency of the image are insufficient "things feel a little delayed right now." (P5) "I have to say that the images were a lot clearer there. [on conventional ultrasound monitor]" (P6)

G1: Unfamiliarity and adaption
Coded if statements describe an unfamiliar or confusing situation, but also if such a situation is overcome

G2: Unfamiliar
Statements directly mentioning unfamiliarity "it's a little unfamiliar to see it like this" (P3)

G3: Adaption
Coded if participants verbalize overcoming a confusing or unfamiliar situation "Hm. Right. Liver. Okay. Getting better now." (P1, S. 2) "Starting to get used to it by now" (P5)

G4: Confusion
Statements which directly mention confusiong and/or implicity express lack of understanding of the HMD application "Okay, uh. I need to keep looking at this for a bit so I know what I'm seeing" (P6) "Hm. Does that work? Maybe I need to do something differently with the..." (P1)

G5: Improvements through practice
Prediction that confusion or limitations caused by the HMD can be reduced or eliminated through practice "Well I think it will get better once I get used to it […]" (P5) "I think if I was familiar with it, I could utilize it a lot better." (P6)