survey

Evaluation of XR Applications: A Tertiary Review

Authors:
Artur Becker

Institute of Informatics, Federal University of Rio Grande do Sul, Brazil

Institute of Informatics, Federal University of Rio Grande do Sul, Brazil

0000-0002-7258-0468
View Profile

,
Carla M. Dal Sasso Freitas

Institute of Informatics, Federal University of Rio Grande do Sul, Brazil

Institute of Informatics, Federal University of Rio Grande do Sul, Brazil

0000-0003-1986-8435
View Profile

Authors Info & Claims

ACM Computing Surveys Volume 56 Issue 5Article No.: 110pp 1–35https://doi.org/10.1145/3626517

Published:25 November 2023Publication History

ACM Computing Surveys

Abstract

Extended reality (XR) applications—encompassing virtual reality, augmented reality, and mixed reality—are finding their way into multiple domains. Each area has different motivations for employing and different criteria for evaluating XR. Multiple surveys describe XR and its evaluation in particular fields. However, these surveys do not always agree on the definition of XR. This lack of consensus makes it hard to compare and use learnings from XR research across areas. Through a tertiary systematic literature review, we analyzed 81 surveys from several fields to provide a comprehensive summary of the state of XR research regarding the evaluation of XR applications. We seek to understand (i) how is XR defined? (ii) why is XR employed? (iii) how is XR evaluated? (iv) what are the main criticisms and future research paths outlined by the surveys? and (v) how good are the surveys? We present our findings describing XR research in 10 categories. Given our findings, we propose that future research should build upon a solid XR taxonomy and depart from effectiveness into efficiency research—to understand not only if but also how XR achieves the desired outcomes.

Supplemental Material

Available for Download

pdf

3626517.supp.pdf (2.2 MB)

Supplementary material

REFERENCES

[1] Ahmed Kamran, Jawad Muhammed, Abboudi May, Gavazzi Andrea, Darzi Ara, Athanasiou Thanos, Vale Justin, Khan Mohammad Shamim, and Dasgupta Prokar. 2011. Effectiveness of procedural simulation in urology: A systematic review. Journal of Urology 186, 1 (July2011), 26–34. DOI:Google ScholarCross Ref
[2] Akçayır Murat and Akçayır Gökçe. 2017. Advantages and challenges associated with augmented reality for education: A systematic review of the literature. Educational Research Review 20 (Feb.2017), 1–11. DOI:Google ScholarCross Ref
[3] Alaker Medhat, Wynn Greg R., and Arulampalam Tan. 2016. Virtual reality training in laparoscopic surgery: A systematic review & meta-analysis. International Journal of Surgery 29 (May2016), 85–94. DOI:Google ScholarCross Ref
[4] Alashram Anas R., Annino Giuseppe, Padua Elvira, Romagnoli Cristian, and Mercuri Nicola Biagio. 2019. Cognitive rehabilitation post traumatic brain injury: A systematic review for emerging use of virtual reality technology. Journal of Clinical Neuroscience 66 (Aug.2019), 209–219. DOI:Google ScholarCross Ref
[5] Arora Asit, Lau Loretta Y. M., Awad Zaid, Darzi Ara, Singh Arvind, and Tolley Neil. 2014. Virtual reality simulation training in otolaryngology. International Journal of Surgery 12, 2 (Feb.2014), 87–94. DOI:Google ScholarCross Ref
[6] Azuma R., Baillot Y., Behringer R., Feiner S., Julier S., and MacIntyre B.. 2001. Recent advances in augmented reality. IEEE Computer Graphics and Applications 21, 6 (Dec.2001), 34–47. DOI:Google ScholarDigital Library
[7] Azuma Ronald T.. 1997. A survey of augmented reality. Presence: Teleoperators and Virtual Environments 6, 4 (Aug.1997), 355–385. DOI:Google ScholarDigital Library
[8] Aïm Florence, Lonjon Guillaume, Hannouche Didier, and Nizard Rémy. 2016. Effectiveness of virtual reality training in orthopaedic surgery. Arthroscopy: The Journal of Arthroscopic & Related Surgery 32, 1 (Jan.2016), 224–232. DOI:Google ScholarCross Ref
[9] Benbow Amanda A. and Anderson Page L.. 2019. A meta-analytic examination of attrition in virtual reality exposure therapy for anxiety disorders. Journal of Anxiety Disorders 61 (Jan.2019), 18–26. DOI:Google ScholarCross Ref
[10] Bernhardt Sylvain, Nicolau Stéphane A., Soler Luc, and Doignon Christophe. 2017. The status of augmented reality in laparoscopic surgery as of 2016. Medical Image Analysis 37 (April2017), 66–90. DOI:Google ScholarCross Ref
[11] Blattgerste Jonas, Renner Patrick, and Pfeiffer Thies. 2019. Augmented reality action assistance and learning for cognitively impaired people: A systematic literature review. In Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments (PETRA ’19). ACM, New York, NY, 270–279. DOI:Google ScholarDigital Library
[12] Bluett Brent, Bayram Ece, and Litvan Irene. 2019. The virtual reality of Parkinson’s disease freezing of gait: A systematic review. Parkinsonism & Related Disorders 61 (April2019), 26–33. DOI:Google ScholarCross Ref
[13] Borsci Simone, Lawson Glyn, and Broome Simon. 2015. Empirical evidence, evaluation criteria and challenges for the effectiveness of virtual and mixed reality tools for training operators of car service maintenance. Computers in Industry 67 (Feb.2015), 17–26. DOI:Google ScholarDigital Library
[14] Bosc R., Fitoussi A., Hersant B., Dao T. H., and Meningaud J. P.. 2019. Intraoperative augmented reality with heads-up displays in maxillofacial surgery: A systematic review of the literature and a classification of relevant technologies. International Journal of Oral and Maxillofacial Surgery 48, 1 (Jan.2019), 132–139. DOI:Google ScholarCross Ref
[15] Bozgeyikli Lal, Raij Andrew, Katkoori Srinivas, and Alqasemi Redwan. 2018. A survey on virtual reality for individuals with autism spectrum disorder: Design considerations. IEEE Transactions on Learning Technologies 11, 2 (April2018), 133–151. DOI:Google ScholarCross Ref
[16] Brunckhorst Oliver, Aydin Abdullatif, Abboudi Hamid, Sahai Arun, Khan Muhammad Shamim, Dasgupta Prokar, and Ahmed Kamran. 2015. Simulation-based ureteroscopy training: A systematic review. Journal of Surgical Education 72, 1 (Jan.2015), 135–143. DOI:Google ScholarCross Ref
[17] Budgen David, Brereton Pearl, Drummond Sarah, and Williams Nikki. 2018. Reporting systematic reviews: Some lessons from a tertiary study. Information and Software Technology 95 (March2018), 62–74. DOI:Google ScholarCross Ref
[18] Cacciata Marysol, Stromberg Anna, Lee Jung-Ah, Sorkin Dara, Lombardo Dawn, Clancy Steve, Nyamathi Adeline, and Evangelista Lorraine S.. 2019. Effect of exergaming on health-related quality of life in older adults: A systematic review. International Journal of Nursing Studies 93 (May2019), 30–40. DOI:Google ScholarCross Ref
[19] Cardoş Roxana A. I., David Oana A., and David Daniel O.. 2017. Virtual reality exposure therapy in flight anxiety: A quantitative meta-analysis. Computers in Human Behavior 72 (July2017), 371–380. DOI:Google ScholarDigital Library
[20] Carl Emily, Stein Aliza T., Levihn-Coon Andrew, Pogue Jamie R., Rothbaum Barbara, Emmelkamp Paul, Asmundson Gordon J. G., Carlbring Per, and Powers Mark B.. 2019. Virtual reality exposure therapy for anxiety and related disorders: A meta-analysis of randomized controlled trials. Journal of Anxiety Disorders 61 (Jan.2019), 27–36. DOI:Google ScholarCross Ref
[21] Caudell T. P. and Mizell D. W.. 1992. Augmented reality: An application of heads-up display technology to manual manufacturing processes. In Proceedings of the 25th Hawaii International Conference on System Sciences, Vol. 2. IEEE, Los Alamitos, CA, 659–669. DOI:Google ScholarCross Ref
[22] Chen Yu, Abel Kingsley Travis, Janecek John T., Chen Yunan, Zheng Kai, and Cramer Steven C.. 2019. Home-based technologies for stroke rehabilitation: A systematic review. International Journal of Medical Informatics 123 (March2019), 11–22. DOI:Google ScholarCross Ref
[23] Chi B., Chau B., Yeo E., and Ta P.. 2019. Virtual reality for spinal cord injury-associated neuropathic pain: Systematic review. Annals of Physical and Rehabilitation Medicine 62, 1 (Jan.2019), 49–57. DOI:Google ScholarCross Ref
[24] Clark Anna D., Barone Damiano G., Candy Nicholas, Guilfoyle Mathew, Budohoski Karol, Hofmann Riikka, Santarius Thomas, Kirollos Ramez, and Trivedi Rikin A.. 2017. The effect of 3-dimensional simulation on neurosurgical skill acquisition and surgical performance: A review of the literature. Journal of Surgical Education 74, 5 (Sept.2017), 828–836. DOI:Google ScholarCross Ref
[25] Cogné M., Taillade M., N’Kaoua B., Tarruella A., Klinger E., Larrue F., Sauzéon H., Joseph P. A., and Sorita E.. 2017. The contribution of virtual reality to the diagnosis of spatial navigation disorders and to the study of the role of navigational aids: A systematic literature review. Annals of Physical and Rehabilitation Medicine 60, 3 (June2017), 164–176. DOI:Google ScholarCross Ref
[26] Corbetta Davide, Imeri Federico, and Gatti Roberto. 2015. Rehabilitation that incorporates virtual reality is more effective than standard rehabilitation for improving walking speed, balance and mobility after stroke: A systematic review. Journal of Physiotherapy 61, 3 (July2015), 117–124. DOI:Google ScholarCross Ref
[27] Corrêa Cléber G., Nunes Fátima L. S., Ranzini Edith, Nakamura Ricardo, and Tori Romero. 2019. Haptic interaction for needle insertion training in medical applications: The state-of-the-art. Medical Engineering & Physics 63 (Jan.2019), 6–25. DOI:Google ScholarCross Ref
[28] Coyle Hannah, Traynor Victoria, and Solowij Nadia. 2015. Computerized and virtual reality cognitive training for individuals at high risk of cognitive decline: Systematic review of the literature. American Journal of Geriatric Psychiatry 23, 4 (April2015), 335–359. DOI:Google ScholarCross Ref
[29] Cruz-Neira Carolina, Sandin Daniel J., and DeFanti Thomas A.. 1993. Surround-screen projection-based virtual reality: the design and implementation of the CAVE. In Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’93). ACM, New York, NY, 135–142. DOI:Google ScholarDigital Library
[30] Cárdenas Katherine and Aranda Mariana. 2017. Psychotherapies for the treatment of phantom limb pain. Revista Colombiana de Psiquiatría (English ed.) 46, 3 (July2017), 178–186. DOI:Google ScholarCross Ref
[31] Davis Simon, Nesbitt Keith, and Nalivaiko Eugene. 2014. A systematic review of cybersickness. In Proceedings of the 2014 Conference on Interactive Entertainment (IE ’14). ACM, New York, NY, 1–9. DOI:Google ScholarDigital Library
[32] Deng Wenrui, Hu Die, Xu Sheng, Liu Xiaoyu, Zhao Jingwen, Chen Qian, Liu Jiayuan, Zhang Zheng, Jiang Wenxiu, Ma Lijun, Hong Xinyi, Cheng Shengrong, Liu Boya, and Li Xiaoming. 2019. The efficacy of virtual reality exposure therapy for PTSD symptoms: A systematic review and meta-analysis. Journal of Affective Disorders 257 (Oct.2019), 698–709. DOI:Google ScholarCross Ref
[33] Detmer Felicitas J., Hettig Julian, Schindele Daniel, Schostak Martin, and Hansen Christian. 2017. Virtual and augmented reality systems for renal interventions: A systematic review. IEEE Reviews in Biomedical Engineering 10 (2017), 78–94. DOI:Google ScholarCross Ref
[34] Dias Lucas Pfeiffer Salomão, Barbosa Jorge Luis Victória, and Vianna Henrique Damasceno. 2018. Gamification and serious games in depression care: A systematic mapping study. Telematics and Informatics 35, 1 (April2018), 213–224. DOI:Google ScholarCross Ref
[35] Experience World Leaders in Research-Based User. 2012. Usability 101: Introduction to Usability. Retrieved October 17, 2023 from https://www.nngroup.com/articles/usability-101-introduction-to-usability/Google Scholar
[36] Feng Zhenan, González Vicente A., Amor Robert, Lovreglio Ruggiero, and Cabrera-Guerrero Guillermo. 2018. Immersive virtual reality serious games for evacuation training and research: A systematic literature review. Computers & Education 127 (Dec.2018), 252–266. DOI:Google ScholarCross Ref
[37] Fernández-Álvarez Javier, Rozental Alexander, Carlbring Per, Colombo Desirée, Riva Giuseppe, Anderson Page L., Baños Rosa María, Benbow Amanda A., Bouchard Stéphane, Bretón-López Juana María, Cárdenas Georgina, Difede JoAnn, Emmelkamp Paul, García-Palacios Azucena, Guillén Verónica, Hoffman Hunter, Kampann Isabel, Moldovan Ramona, Mühlberger Andreas, North Max, Pauli Paul, Castro Wenceslao Peñate, Quero Soledad, Tortella-Feliu Miquel, Wyka Kataryzna, and Botella Cristina. 2019. Deterioration rates in virtual reality therapy: An individual patient data level meta-analysis. Journal of Anxiety Disorders 61 (Jan.2019), 3–17. DOI:Google ScholarCross Ref
[38] Fite-Georgel Pierre. 2011. Is there a reality in industrial augmented reality? In Proceedings of the 2011 10th IEEE International Symposium on Mixed and Augmented Reality. IEEE, Los Alamitos, CA, 201–210. DOI:Google ScholarDigital Library
[39] Calderón María Antonia Fuentes, Miralles Ainhoa Navarro, Pimienta Mauricio Jaramillo, Estella Jesús María Gonçalves, and Ledesma María José Sánchez. 2019. Analysis of the factors related to the effectiveness of transcranial current stimulation in upper limb motor function recovery after stroke: A systematic review. Journal of Medical Systems 43, 3 (Feb.2019), 69. DOI:Google ScholarDigital Library
[40] Gao Yifan, Gonzalez Vicente A., and Yiu Tak Wing. 2019. The effectiveness of traditional tools and computer-aided technologies for health and safety training in the construction sector: A systematic review. Computers & Education 138 (Sept.2019), 101–115. DOI:Google ScholarDigital Library
[41] Garzón Juan and Acevedo Juan. 2019. Meta-analysis of the impact of augmented reality on students’ learning gains. Educational Research Review 27 (June2019), 244–260. DOI:Google ScholarCross Ref
[42] Ghiţă Alexandra and Gutiérrez-Maldonado José. 2018. Applications of virtual reality in individuals with alcohol misuse: A systematic review. Addictive Behaviors 81 (June2018), 1–11. DOI:Google ScholarCross Ref
[43] Greco Francesco, Cadeddu Jeffrey A., Gill Inderbir S., Kaouk Jihad H., Remzi Mesut, Thompson R. Houston, Leeuwen Fijs W. B. van, Poel Henk G. van der, Fornara Paolo, and Rassweiler Jens. 2014. Current perspectives in the use of molecular imaging to target surgical treatments for genitourinary cancers. European Urology 65, 5 (May2014), 947–964. DOI:Google ScholarCross Ref
[44] Guedes Hugo Gonçalo, Ferreira Zêmia Maria Câmara Costa, Leão Layra Ribeiro de Sousa, Montero Edna Frasson Souza, Otoch José Pinhata, and Artifon Everson Luiz de Almeida. 2019. Virtual reality simulator versus box-trainer to teach minimally invasive procedures: A meta-analysis. International Journal of Surgery 61 (Jan.2019), 60–68. DOI:Google ScholarCross Ref
[45] Gujjar Kumar Raghav, Wijk Arjen van, Kumar Ratika, and Jongh Ad de. 2019. Are technology-based interventions effective in reducing dental anxiety in children and adults? A systematic review. Journal of Evidence Based Dental Practice 19, 2 (June2019), 140–155. DOI:Google ScholarCross Ref
[46] Han Jieun, Kang Hyo-Jin, Kim Minjung, and Kwon Gyu Hyun. 2020. Mapping the intellectual structure of research on surgery with mixed reality: Bibliometric network analysis (2000–2019). Journal of Biomedical Informatics 109 (Sept.2020), 103516. DOI:Google ScholarDigital Library
[47] Howard Matt C.. 2017. A meta-analysis and systematic literature review of virtual reality rehabilitation programs. Computers in Human Behavior 70 (May2017), 317–327. DOI:Google ScholarDigital Library
[48] Huang Youliang, Huang Qian, Ali Sajid, Zhai Xing, Bi Xiaoming, and Liu Renquan. 2016. Rehabilitation using virtual reality technology: A bibliometric analysis, 1996–2015. Scientometrics 109, 3 (Dec.2016), 1547–1559. DOI:Google ScholarDigital Library
[49] Ibáñez María-Blanca and Delgado-Kloos Carlos. 2018. Augmented reality for STEM learning: A systematic review. Computers & Education 123 (Aug.2018), 109–123. DOI:Google ScholarDigital Library
[50] ISO. 2018. ISO 9241-11:2018. Retrieved October 17, 2023 from https://www.iso.org/cms/render/live/en/sites/isoorg/contents/data/standard/06/35/63500.htmlGoogle Scholar
[51] ISO/IEC. 2016. ISO/IEC 25066:2016. Retrieved October 17, 2023 from https://www.iso.org/cms/render/live/en/sites/isoorg/contents/data/standard/06/38/63831.htmlGoogle Scholar
[52] Joda T., Gallucci G. O., Wismeijer D., and Zitzmann N. U.. 2019. Augmented and virtual reality in dental medicine: A systematic review. Computers in Biology and Medicine 108 (May2019), 93–100. DOI:Google ScholarDigital Library
[53] Kampmann Isabel L., Emmelkamp Paul M. G., and Morina Nexhmedin. 2016. Meta-analysis of technology-assisted interventions for social anxiety disorder. Journal of Anxiety Disorders 42 (Aug.2016), 71–84. DOI:Google ScholarCross Ref
[54] Karakus Mehmet, Ersozlu Alpay, and Clark Aaron C.. 2019. Augmented reality research in education: A bibliometric study. Eurasia Journal of Mathematics, Science and Technology Education 15, 10 (May2019), em1755. DOI:Google ScholarCross Ref
[55] Kennedy Cassie C., Maldonado Fabien, and Cook David A.. 2013. Simulation-based bronchoscopy training: Systematic review and meta-analysis. Chest 144, 1 (July2013), 183–192. DOI:Google ScholarCross Ref
[56] Kersten-Oertel Marta, Jannin Pierre, and Collins D. Louis. 2013. The state of the art of visualization in mixed reality image guided surgery. Computerized Medical Imaging and Graphics 37, 2 (March2013), 98–112. DOI:Google ScholarCross Ref
[57] Kim Kangsoo, Billinghurst Mark, Bruder Gerd, Duh Henry Been-Lirn, and Welch Gregory F.. 2018. Revisiting trends in augmented reality research: A review of the 2nd decade of ISMAR (2008–2017). IEEE Transactions on Visualization and Computer Graphics 24, 11 (Nov.2018), 2947–2962. DOI:Google ScholarCross Ref
[58] Kitchenham Barbara and Charters Stuart. 2007. Guidelines for Performing Systematic Literature Reviews in Software Engineering. Technical Report 2007-01. EBSE.Google Scholar
[59] Klinke Marianne E., Hafsteinsdóttir Thóra B., Hjaltason Haukur, and Jónsdóttir Helga. 2015. Ward-based interventions for patients with hemispatial neglect in stroke rehabilitation: A systematic literature review. International Journal of Nursing Studies 52, 8 (Aug.2015), 1375–1403. DOI:Google ScholarCross Ref
[60] Koskina Antonia, Campbell Iain C., and Schmidt Ulrike. 2013. Exposure therapy in eating disorders revisited. Neuroscience & Biobehavioral Reviews 37, 2 (Feb.2013), 193–208. DOI:Google ScholarCross Ref
[61] Kurilovas Eugenijus. 2016. Evaluation of quality and personalisation of VR/AR/MR learning systems. Behaviour & Information Technology 35, 11 (Nov.2016), 998–1007. DOI:Google ScholarDigital Library
[62] LaValle Steven. 2015. Virtual Reality. Retrieved October 17, 2023 from http://lavalle.pl/vr/Google Scholar
[63] LaViola Joseph J., Kruijff Ernst, McMahan Ryan P., Bowman Doug A., and Poupyrev Ivan. 2017. 3D User Interfaces: Theory and Practice (2nd ed.). Addison-Wesley, Boston, MA.Google Scholar
[64] Lin I.-Hsien, Tsai Han-Ting, Wang Chien-Yung, Hsu Chih-Yang, Liou Tsan-Hon, and Lin Yen-Nung. 2019. Effectiveness and superiority of rehabilitative treatments in enhancing motor recovery within 6 months poststroke: A systemic review. Archives of Physical Medicine and Rehabilitation 100, 2 (Feb.2019), 366–378. DOI:Google ScholarCross Ref
[65] Malloy Kevin M. and Milling Leonard S.. 2010. The effectiveness of virtual reality distraction for pain reduction: A systematic review. Clinical Psychology Review 30, 8 (Dec.2010), 1011–1018. DOI:Google ScholarCross Ref
[66] Martin Steven, Sutcliffe Paul, Griffiths Frances, Sturt Jackie, Powell John, Adams Ann, and Dale Jeremy. 2011. Effectiveness and impact of networked communication interventions in young people with mental health conditions: A systematic review. Patient Education and Counseling 85, 2 (Nov.2011), e108–e119. DOI:Google ScholarCross Ref
[67] Martins Anabela Correia, Santos Cláudia, Silva Catarina, Baltazar Daniela, Moreira Juliana, and Tavares Nuno. 2018. Does modified Otago Exercise Program improves balance in older people? A systematic review. Preventive Medicine Reports 11 (Sept.2018), 231–239. DOI:Google ScholarCross Ref
[68] Mazur Travis, Mansour Tarek R., Mugge Luke, and Medhkour Azedine. 2018. Virtual reality–based simulators for cranial tumor surgery: A systematic review. World Neurosurgery 110 (Feb.2018), 414–422. DOI:Google ScholarCross Ref
[69] Merchant Zahira, Goetz Ernest T., Cifuentes Lauren, Keeney-Kennicutt Wendy, and Davis Trina J.. 2014. Effectiveness of virtual reality-based instruction on students’ learning outcomes in K-12 and higher education: A meta-analysis. Computers & Education 70 (Jan.2014), 29–40. DOI:Google ScholarCross Ref
[70] Milgram P. and Kishino F.. 1994. A taxonomy of mixed reality visual displays. IEICE Transactions on Information Systems E77-D, 12 (1994), 1–15.Google Scholar
[71] Moglia Andrea, Ferrari Vincenzo, Morelli Luca, Ferrari Mauro, Mosca Franco, and Cuschieri Alfred. 2016. A systematic review of virtual reality simulators for robot-assisted surgery. European Urology 69, 6 (June2016), 1065–1080. DOI:Google ScholarCross Ref
[72] Mohammadi Roghayeh, Semnani Alireza Vaezpour, Mirmohammadkhani Majid, and Grampurohit Namrata. 2019. Effects of virtual reality compared to conventional therapy on balance poststroke: A systematic review and meta-analysis. Journal of Stroke and Cerebrovascular Diseases 28, 7 (July2019), 1787–1798. DOI:Google ScholarCross Ref
[73] Moreno Alexander, Wall Kylie Janine, Thangavelu Karthick, Craven Lucas, Ward Emma, and Dissanayaka Nadeeka N.. 2019. A systematic review of the use of virtual reality and its effects on cognition in individuals with neurocognitive disorders. Alzheimer’s & Dementia: Translational Research & Clinical Interventions 5, 1 (Jan.2019), 834–850. DOI:Google ScholarCross Ref
[74] Munzner Tamara. 2015. Visualization Analysis and Design. Boca Raton, FL.Google Scholar
[75] Neguţ Alexandra, Matu Silviu-Andrei, Sava Florin Alin, and David Daniel. 2016. Task difficulty of virtual reality-based assessment tools compared to classical paper-and-pencil or computerized measures: A meta-analytic approach. Computers in Human Behavior 54 (Jan.2016), 414–424. DOI:Google ScholarDigital Library
[76] Neumann David L., Moffitt Robyn L., Thomas Patrick R., Loveday Kylie, Watling David P., Lombard Chantal L., Antonova Simona, and Tremeer Michael A.. 2018. A systematic review of the application of interactive virtual reality to sport. Virtual Reality 22, 3 (Sept.2018), 183–198. DOI:Google ScholarDigital Library
[77] Ng Yu-Leung, Ma Flora, Ho Frederick K., Ip Patrick, and Fu King-Wa. 2019. Effectiveness of virtual and augmented reality-enhanced exercise on physical activity, psychological outcomes, and physical performance: A systematic review and meta-analysis of randomized controlled trials. Computers in Human Behavior 99 (Oct.2019), 278–291. DOI:Google ScholarDigital Library
[78] Nocentini Annalaura, Zambuto Valentina, and Menesini Ersilia. 2015. Anti-bullying programs and information and communication technologies (ICTs): A systematic review. Aggression and Violent Behavior 23 (July2015), 52–60. DOI:Google ScholarCross Ref
[79] Nour Monica, Yeung Sin Hang, Partridge Stephanie, and Allman-Farinelli Margaret. 2017. A narrative review of social media and game-based nutrition interventions targeted at young adults. Journal of the Academy of Nutrition and Dietetics 117, 5 (May2017), 735–752.e10. DOI:Google ScholarCross Ref
[80] Pereira Clayton R., Pereira Danilo R., Weber Silke A. T., Hook Christian, Albuquerque Victor Hugo C. de, and Papa João P.. 2019. A survey on computer-assisted Parkinson’s disease diagnosis. Artificial Intelligence in Medicine 95 (April2019), 48–63. DOI:Google ScholarDigital Library
[81] Perrochon Anaick, Borel Benoit, Istrate Dan, Compagnat Maxence, and Daviet Jean-Christophe. 2019. Exercise-based games interventions at home in individuals with a neurological disease: A systematic review and meta-analysis. Annals of Physical and Rehabilitation Medicine 62, 5 (Sept.2019), 366–378. DOI:Google ScholarCross Ref
[82] Pfandler Michael, Lazarovici Marc, Stefan Philipp, Wucherer Patrick, and Weigl Matthias. 2017. Virtual reality-based simulators for spine surgery: A systematic review. Spine Journal 17, 9 (Sept.2017), 1352–1363. DOI:Google ScholarCross Ref
[83] Ravi D. K., Kumar N., and Singhi P.. 2017. Effectiveness of virtual reality rehabilitation for children and adolescents with cerebral palsy: An updated evidence-based systematic review. Physiotherapy 103, 3 (Sept.2017), 245–258. DOI:Google ScholarCross Ref
[84] Michael K. Rooney, Fan Zhu, Erin F. Gillespie, Jillian R. Gunther, Ryan P. McKillip, Matthew Lineberry, Ara Tekian, and Daniel W. Golden. 2018. Simulation as more than a treatment-planning tool: A systematic review of the literature on radiation oncology simulation-based medical education. International Journal of Radiation Oncology *Biology* Physics 102, 2 (October 2018), 257–283. DOI:Google ScholarCross Ref
[85] Santos Marc Ericson C., Chen Angie, Taketomi Takafumi, Yamamoto Goshiro, Miyazaki Jun, and Kato Hirokazu. 2014. Augmented reality learning experiences: Survey of prototype design and evaluation. IEEE Transactions on Learning Technologies 7, 1 (Jan.2014), 38–56. DOI:Google ScholarCross Ref
[86] Savran Mona Meral, Sørensen Stine Maya Dreier, Konge Lars, Tolsgaard Martin G., and Bjerrum Flemming. 2016. Training and assessment of hysteroscopic skills: A systematic review. Journal of Surgical Education 73, 5 (Sept.2016), 906–918. DOI:Google ScholarCross Ref
[87] Scapin Soliane, Echevarría-Guanilo Maria Elena, Junior Paulo Roberto Boeira Fuculo, Gonçalves Natália, Rocha Patrícia Kuerten, and Coimbra Rebeca. 2018. Virtual reality in the treatment of burn patients: A systematic review. Burns 44, 6 (Sept.2018), 1403–1416. DOI:Google ScholarCross Ref
[88] Scheffler Michael, Koranyi Susan, Meissner Winfried, Strauß Bernhard, and Rosendahl Jenny. 2018. Efficacy of non-pharmacological interventions for procedural pain relief in adults undergoing burn wound care: A systematic review and meta-analysis of randomized controlled trials. Burns 44, 7 (Nov.2018), 1709–1720. DOI:Google ScholarCross Ref
[89] See K. W. M., Chui K. H., Chan W. H., Wong K. C., and Chan Y. C.. 2016. Evidence for endovascular simulation training: A systematic review. European Journal of Vascular and Endovascular Surgery 51, 3 (March2016), 441–451. DOI:Google ScholarCross Ref
[90] Sharp Helen. 2019. Interaction Design 5e. John Wiley & Sons, Indianapolis, IN.Google Scholar
[91] Slater Mel and Wilbur Sylvia. 1997. A framework for immersive virtual environments (FIVE): Speculations on the role of presence in virtual environments. Presence: Teleoperators and Virtual Environments 6, 6 (Dec.1997), 603–616. DOI:Google ScholarDigital Library
[92] Subramanian Sandeep K. and Prasanna Shreya S.. 2018. Virtual reality and noninvasive brain stimulation in stroke: How effective is their combination for upper limb motor improvement?—A meta-analysis. PM&R 10, 11 (2018), 1261–1270. DOI:Google ScholarCross Ref
[93] Suh Ayoung and Prophet Jane. 2018. The state of immersive technology research: A literature analysis. Computers in Human Behavior 86 (Sept.2018), 77–90. DOI:Google ScholarCross Ref
[94] Tan Bhing-Leet, Lee Sara-Ann, and Lee Jimmy. 2018. Social cognitive interventions for people with schizophrenia: A systematic review. Asian Journal of Psychiatry 35 (June2018), 115–131. DOI:Google ScholarCross Ref
[95] Tay Charison, Khajuria Ankur, and Gupte Chinmay. 2014. Simulation training: A systematic review of simulation in arthroscopy and proposal of a new competency-based training framework. International Journal of Surgery 12, 6 (June2014), 626–633. DOI:Google ScholarCross Ref
[96] Thomsen Ann Sofia S., Subhi Yousif, Kiilgaard Jens Folke, Cour Morten la, and Konge Lars. 2015. Update on simulation-based surgical training and assessment in ophthalmology: A systematic review. Ophthalmology 122, 6 (June2015), 1111–1130.e1. DOI:Google ScholarCross Ref
[97] Turner Wesley A. and Casey Leanne M.. 2014. Outcomes associated with virtual reality in psychological interventions: Where are we now? Clinical Psychology Review 34, 8 (Dec.2014), 634–644. DOI:Google ScholarCross Ref
[98] Viñas-Diz S. and Sobrido-Prieto M.. 2016. Virtual reality for therapeutic purposes in stroke: A systematic review. Neurología (English Ed.) 31, 4 (May2016), 255–277. DOI:Google ScholarCross Ref
[99] Vogt Sarah, Skjæret-Maroni Nina, Neuhaus Dorothee, and Baumeister Jochen. 2019. Virtual reality interventions for balance prevention and rehabilitation after musculoskeletal lower limb impairments in young up to middle-aged adults: A comprehensive review on used technology, balance outcome measures and observed effects. International Journal of Medical Informatics 126 (June2019), 46–58. DOI:Google ScholarCross Ref
[100] Weiss Patrice L., Kizony Rachel, Feintuch Uri, and Katz Noomi. 2006. Virtual reality in neurorehabilitation. In Textbook of Neural Repair and Rehabilitation, Selzer Michael, Clarke Stephanie, Cohen Leonardo, Duncan Pamela, and Gage Fred (Eds.). Cambridge University Press, Cambridge, UK, 182–197. DOI:Google ScholarCross Ref

Index Terms

Evaluation of XR Applications: A Tertiary Review
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
    2. Interaction paradigms
      1. Mixed / augmented reality

Recommendations

What is XR? Towards a Framework for Augmented and Virtual Reality
Abstract
Augmented Reality (AR), Virtual Reality (VR), Mixed Reality, and Extended Reality (often – misleadingly – abbreviated as XR) are commonly used terms to describe how technologies generate or modify reality. However, academics and ...
Highlights
- XR should not be used as an abbreviation for “Extended Reality”; X should be the placeholder for “all” new reality formats.
Read More
Virtual, mixed, and augmented reality: a systematic review for immersive systems research
Abstract
Immersive systems can be used to capture new data, create new experiences, and provide new insights by generating virtual elements of physical and imagined worlds. Immersive systems are seeing increased application across a broad array of fields. ...
Read More
Unified Representation for XR Content and its Rendering Method
Web3D '20: Proceedings of the 25th International Conference on 3D Web Technology

Virtual Reality (VR) and Augmented Reality (AR) have become familiar technologies with related markets growing rapidly every year. Moreover, the idea of considering VR and AR as one eXtended reality (XR) has broken the border between virtual space and ...
Read More

Comments

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Published in
ACM Computing Surveys Volume 56, Issue 5
May 2024
1019 pages
ISSN:0360-0300
EISSN:1557-7341
DOI:10.1145/3613598
Editor:
Albert Zomaya
University of Sydney, Australia
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
- Published: 25 November 2023
- Online AM: 9 October 2023
- Accepted: 18 September 2023
- Revised: 14 May 2023
- Received: 15 July 2022
Published in csur Volume 56, Issue 5

Permissions
Request permissions about this article.
Request Permissions

Check for updates
Author Tags
Evaluation
virtual reality
augmented reality
systematic review
tertiary review
Qualifiers
- survey
Conference
Funding Sources
Other Metrics
View Article Metrics

Article Metrics
- 0
  Total Citations
  View Citations
- 743
  Total Downloads
- Downloads (Last 12 months)743
- Downloads (Last 6 weeks)89
Other Metrics
View Author Metrics
Cited By
This publication has not been cited yet

PDF Format

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

View Full Text

Evaluation of XR Applications: A Tertiary Review

ACM Computing Surveys

Abstract

Supplemental Material

Available for Download

REFERENCES

Cited By

Index Terms

Recommendations

What is XR? Towards a Framework for Augmented and Virtual Reality

Virtual, mixed, and augmented reality: a systematic review for immersive systems research

Unified Representation for XR Content and its Rendering Method