A research agenda for augmented and virtual reality in architecture, engineering and construction

This paper presents a study on the usage landscape of augmented reality (AR) and virtual reality (VR) in the architecture, engineering and construction sectors, and proposes a research agenda to address the existing gaps in required capabilities. A series of exploratory workshops and questionnaires were conducted with the participation of 54 experts from 36 organisations from industry and academia. Based on the data collected from the workshops, six AR and VR use-cases were defined: stakeholder engagement, design support, design review, construction support, operations and management support, and training. Three main research categories for a future research agenda have been proposed, i.e.: (i) engineering-grade devices, which encompasses research that enables robust devices that can be used in practice, e.g. the rough and complex conditions of construction sites; (ii) workflow and data management; to effectively manage data and processes required by AR and VR technologies; and (iii) new capabilities; which includes new research required that will add new features that are necessary for the specific construction industry demands. This study provides essential information for practitioners to inform adoption decisions. To researchers, it provides a research road map to inform their future research efforts. This is a foundational study that formalises and categorises the existing usage of AR and VR in the construction industry and provides a roadmap to guide future research efforts.


Introduction
Augmented Reality (AR) and Virtual Reality (VR) are technologies of utmost importance for the Architecture, Engineering and Construction (AEC) sectors as the built environment is intrinsically associated to three-dimensional (3D) space and AEC professionals rely heavily on imagery for communication.VR and AR -to a lesser degree-have been used by built environment professionals to support the visualisation of design, construction and city operations since around 1990s [184].AR is a technology that overlays information and computer-generated imagery on the real environment to enhance or augment the contextual perception of the user's surroundings [108,109].Augmentations are visualised using a mobile device, tablet or a head-mounted display (HMD).On the other hand, VR is a technology that creates virtual environments entirely generated by a computer, replacing the user's perception of the surrounding environment with a virtual environment using HMDs, glasses and multi-display setups.
AR and VR are considered in the top 10 Gartner strategic technology trends for 2019 [131].The main AR and VR applications are in the gaming and entertainment sectors, but marketing, tourism, sports, and education have experimented with AR and VR as well [142].Companies from a wide range of sectors are using AR and VR for training and productivity improvements.For example, UPS [174] a parcel delivery company, is using VR for driver safety training; while Boeing [19], an aerospace company, reported up to 40% productivity improvements in electrical wiring installation tasks when using AR HMDs.However, AR and VR technologies are not yet robust and reliable enough to comply with real-life industrial requirements [130].Technical limitations are one of the main factors that limit adoption in the dynamic and rough environments typical in the construction sector.Other factors are also at play.Yung and Khoo-Lattimore [193] note that lack of awareness of the technology, poor usability, the large time commitment for implementation, and the unwillingness to accept a virtual substitute are issues that need to be addressed.Glegg and Levac [67] note that regardless of the AR and VR level of technical maturity, there is the need https://doi.org/10.1016/j.aei.2020.101122Received 8 December 2019; Received in revised form 5 April 2020; Accepted 14 May 2020 to study the actual effectiveness of the AR and VR for particular construction tasks, and for the targeted development of implementation research.
Previous studies have reviewed and classified research output related to AR [181] and VR [171] in the built environment.However, and up-to-date study and discussion are necessary.There is no granular study that analyses how and for what purposes AEC companies are using AR and VR technologies.More specifically, no study exists that systematically maps (i) how AR and VR technologies can be used, (ii) the potential benefits, (iii) prevalent issues, and (iv) a future research agenda.The study presented in this paper seeks to fill this gap.The overall aim of this study is to formulate a roadmap and research agenda to realise the potential of AR and VR for the AEC sectors.The specific objectives are: 1 To define specific AR and VR use-cases for the AEC sectors. 2 To provide a detailed quantitative and qualitative analysis of the AR and VR usage in the AEC sectors.3 To provide an AR and VR future research agenda for the specific AEC needs.
The next section presents the methodology used in this study.Section 3 presents the defined use-cases of AR and VR in the AEC sectors; while Section 4 presents the usage analysis.Section 5 presents the proposed future research agenda.The discussion, implications for practice, limitations and further work are presented in Section 6. Lastly, conclusions are given in Section 7.

Research method
The findings reported in this paper are part of a study on the status of AR and VR usage in the AEC sector conducted from July to December 2018.A research network so-called Vision Network was formed consisting of representatives from five companies, i.e.Arcadis, A&H Group, Bentley Systems, Mott MacDonald, and Willmott Dixon; and five universities, i.e.Cardiff University, Coventry University, Cranfield University, Loughborough University, and the University of the West of England to drive the research project.The Vision Network organised a series of exploratory workshops and quantitative instruments to collect data regarding the actual use of AR and VR in practice.A literature review was carried out to identify the latest research efforts.Data from the sources above were used to develop and characterise the use-cases and to define a research agenda specific for the AEC sectors.More detailed information about the research work carried out by the Vision Network including the exploratory workshops, literature review, and quantitative instruments is presented in the Vision Network report [40].
In order to obtain a broad understanding of the AR and VR landscape in the AEC sectors, a multidisciplinary group of experts was invited to participate in the exploratory workshops.Fifty-four experts from 36 organisations from industry (i.e.construction companies, design offices, engineering consultancies and technology developers) and academia participated in a series of exploratory workshops and questionnaires.All the invited experts were working on AR and VR implementation or development, had more than three years of experience, and were affiliated to an organisation from the categories mentioned above.In order to maximise the generalisation of the results and to ensure that the findings are representative of the very heterogeneous AEC sector, a very targeted sampling method was used to ensure that (i) participants came from various parts across the AEC sectors, and thus are representative of the entire spectrum, i.e. large and small construction, engineering, and architecture firms, (ii) technology development companies and academics were invited to obtain a different perspective on the subject, and (iii) participants were experts in the discussed topics.The participating construction, engineering and architecture firms operate in North America, Europe and Australia.The technology development companies that participated are actively developing hardware and software solutions for AR and VR in the AEC sectors.
A combination of qualitative and quantitative data collection and analysis methods were used, as those were found to be a more effective way to analyse complex systems [36].Fig. 1 details the main parts of the research method, i.e. qualitative and quantitative analyses; and the three research outcomes, i.e. use-cases, levels of adoption, and the research agenda presented in sections 3, 4, and 5, respectively.

Qualitative analysis
During the exploratory workshops, four focus group discussions (FGDs) with experts from academia construction companies, engineering consultancies, design firms and technology development companies were conducted in the qualitative part of the study.The four FGDs lasted for 45 min with 14, 13, 16, and 11 experts participating in each of the FGDs respectively.The FGDs were used to identify and characterise (i) the main use-cases of AR and VR currently in use in the AEC sector, (ii) the main benefits and challenges of using AR and VR, and (iii) gaps in current capabilities and research.FGDs were used because they are effective tools for qualitative and exploratory analysis as they allow the participants to build on arguments from the other participants [90]; which is not the case with other methods such as individual interviews, in which relevant information could be missed.During each FGD two activities were carried out, a break-out session, in which smaller groups of participants collaborated to identify use-cases, benefits, challenges and capabilities; and a group discussion session, in which all the findings were discussed among all the participants.All the recorded data during the FGDs was compiled, segmented and grouped in related themes, as shown in Fig. 1.The collected data was used to develop a quantitative data collection instrument explained in the next sub-section.

Quantitative analysis
A questionnaire was developed to quantify the general levels of adoption of the AEC sector and of the specific use-cases identified in the qualitative part of the study.A 5-point Likert scale was used in the questionnaire to codify the responses, which, despite its limitations, is a very effective way for response codification [186].The questionnaire was pilot-tested by six experts (4 from industry and two from academia) to ensure the clarity of the questions and the structure and logic of the questionnaire.Convenience and stratified sampling methods were used to select participants from academia, construction companies, engineering consultancies, design firms and technology development companies, which have implemented AR and VR.Stratification was carried out by defining categories based on the type and size of the participants' organisations.Three to five experts were approached from each the following categories: (i) top construction companies by revenue, (ii) small and medium construction companies, (iii) top engineering consultancy companies by the number of employees, (iv) small and medium engineering consultancy companies, (v) top design firms by the number of employees, (vi) small and medium design firms, (vii) technology development companies, and (viii) academia.Instead of selecting experts randomly from the defined categories, the experts that were readily available to participate were selected (convenience sampling).
Thirty-four completed questionnaires were received out of 45 approached participants, representing a 75.5% response rate.The distribution of the respondents is 11.8% from academia, 20.6% from construction companies, 32.4% from engineering consultancies, and 17.6% from both architecture offices and technology development companies.An expertise level factor was developed to obtain an indication of the relevant experience of the respondents.This factor is the average of the following attributes captured in the 5-point Likert scale in the questionnaire: (i) years of professional experience, (ii) years of experience using AR and VR, (iii) level of implementation complexity in VR, and (iv) level of implementation complexity in AR.The expertise level factor of the participants is 2.9% beginner (1-1.9 in Likert scale), 20.6% intermediate (2-2.9),50% advanced (3-3.9), and 26.5% experts (4)(5).More than 75% of the respondents identify themselves as advanced or experts in the field.

Characterising use-cases and defining a research agenda
Using the data collected through the FGDs and the quantitative instrument regarding the AR and VR usage, six usage categories were defined following the built asset's life cycle phases, i.e.(a) AR and VR for stakeholder engagement, (b) AR and VR for design, (c) AR and VR for design review, (d) AR and VR for construction, (e) AR and VR for operations, and (f) AR and VR for training.Then, a literature survey was carried out.Literature searches in SCOPUS and Google Scholar were carried out using the terms augmented reality and virtual reality in combination with the following terms based in the categories above, i.e.: (a) stakeholder engagement, client engagement, virtual tour, walkthrough; (b) design, design support; (c) design review, design sign off; (d) construction, construction support, progress monitoring, assembly, safety; (e) operations, maintenance, facility management, inspections; and (f) training and education.After analysing the resulting papers from the literature survey, six use-cases were defined (see Section3) by correlating the information that participants provided on how AR and VR are used in practice [40] and research collected from literature.Note that category (d) AR and VR for construction was broken down into four subcategories to help exemplify the variety of existing work in this category.
The use-cases were characterised by explaining the main benefits of each technology for the use-case, presenting a few examples from literature, and discussing the existing challenges.The prior studies collected from literature were assigned into these six categories according to the initial literature searches.The examples from literature were selected by giving preference to recent publications, published in peerreviewed journals, and that the application was closely related to the AEC sectors.A longitudinal approach in the literature survey was taken to provide a wide overview of the use-case rather than an exhaustive survey.Lastly, the challenges were compiled by combining issues raised by the participants and issues reported in literature.
A research agenda was defined as well.Gaps in capabilities and research were identified and classified in three themes (i) engineering issues, (ii) process management issues, and (iii) technology issues.These gaps were expressed into research topics, and enabling technologies were identified.The research topics were mapped in a table, and dependencies among them were identified.Timeframes and technology readiness levels were estimated, taking into consideration the expectations from the AEC and technology development practitioners.The final research agenda was formulated, consisting of three main categories, as presented in Section 5. Note that the use-cases presented here intend to structure the existing research to improve understanding of the research field, thus facilitating to define a research agenda.However, in many cases, the prior studies presented here can be categorised in more than one usecase, as the technologies presented are relevant to various use-cases.Note that the categorisation rationale follows the initial literature searches explained in Section 2.3, but some adjustments have been made to facilitate the understanding of the six use-cases.These usecases follow the categorisation closely initially defined in the FGDs; thus, it does not have a significant impact on the adoption level study presented in Section 4. Note as well that the differences between AR and VR are decreasing as both technologies advance.The differences in features, implementation requirements, benefits, and challenges are not very significant.Nevertheless, for each use-case examples for AR and VR have been addressed separately to provide a clearer understanding of the research landscape.

Stakeholder engagement
AR and VR can be used to engage with potential clients or with the public to provide a more realistic representation of a built asset and to generate more relevant or informed feedback.AR and VR representations can give consumers an opportunity to better examine built-assets at real-scale, in an immersive environment, and provide a better understanding than pictures or videos [69].For example, having a better understanding of the end-product will contribute to aligning the stakeholders' expectations with the actual design.Both AR and VR approaches for stakeholder engagement are the most used and easier to implement use-case (see Section 4.2).Nowadays, it is relatively easy to develop an AR mobile app that visualises a virtual model of a built asset on top of a table or a VR walkthrough of a built asset using only a mobile phone and a mobile VR headset.For example, Pejic et al. [135] present a system that employs both AR and VR for stakeholder engagement.
AR examples.In the case of AR, the technology is better suited for visualising renovations and retrofit works, as it combines the real environment with virtual objects.Mutis and Ambekar [118] investigated the existing AR challenges to implement project walkthroughs and presented a system that can visualise future interventions in construction sites.AR has been used as well to support virtual tours and in-situ walkthroughs.For example, Kim et al. [83] presented an AR system to enrich cultural heritage tours by overlaying virtual objects on cultural sites.Tan and Lim [167] presented an AR virtual tour system to improve the experience of exploring historical sites.Andri et al. [6] presented a system to support campus tours using AR.
VR examples.VR has been tested extensively for stakeholder engagement in real estate, e.g.[79,129], as it is the ideal medium to immerse stakeholders into a virtual environment helping them to understand how the end-product will look like and how it will feel.Kini and Sunil [87] presented a VR system to visualise different types of sustainable construction techniques in rural communities, and Xia [187] presented the development of a VR system for campus virtual walkthroughs.Most of the recent research has been focused on providing multi-user capabilities.For example, Du et al. [55] presented a multiuser VR platform that enabled several parties (e.g.clients, architects, engineers, and general contractors) to interact in a unified VR environment improving the stakeholders' engagement and communication; and Lin et al. [97] presented a VR approach to improving the communication between design teams and healthcare stakeholders.
Challenges.The main issues of using VR for stakeholder engagement are that (i) for most of the users the interface is not sufficiently user-friendly (e.g. a chaperone is always required), (ii) there is a sense of isolation, and (iii) VR experiences are not easily shared.The main AR challenge for stakeholder engagement is that the augmentations do not seem sufficiently real, and the immersive experience is poor.This is caused by four main issues (i) registration inaccuracies, (ii) inconsistent and unrealistic luminance of virtual objects, (iii) drifting errors, and unreliable location and motion tracking, and (iv) overlaying virtual objects at large distances from the user is problematic.

Design support
AR and VR can support designers to identify the consequences of their design decisions and to have a better understanding of the final results.AR examples.For example, Sandor and Klinker [152] presented an AR system called ARCHIE that supports architects to work collaboratively on a virtual model placed on a table and present the design intent to stakeholders.Nee et al. [122] described how AR could be used to support collaborative design, in which various users interact with a single virtual model instead of using physical mock-ups.More recently, Lin et al. [95] presented an AR system that displays the results of computer fluid dynamics simulations of indoor thermal environments on mobile devices.Fukuda et al. [66] presented an AR system that enables to visualise results of thermal simulations to support retrofitting.
VR examples.Roach and Demirkiran [144] presented a VR-enabled 3D modelling software that facilitates the creation of 3D models.The authors compared their VR-enabled software with other VR-enabled 3D modelling software (i.e.Google Blocks and Make VR Pro) and with traditional 3D modelling software (i.e.Blender, FreeCAD, and Solid-Works).The authors reported that VR-enabled modelling software is easier to learn, more intuitive to use and allows faster modelling than tradition 3D modelling software, albeit for very simple geometries.Nguyen et al. [125] presented a VR system for city planning and object modelling that included special communication protocols and gesture recognition techniques.Motamedi et al. [116] showed an approach to testing the effectiveness of signages of Japanese subway stations using VR environments.Lastly, Natephra et al. [121] presented a VR system that enables designers to experience various different designs in realistic scenarios, thus enabling them to identify qualitative characteristics besides the quantitative analysis using traditional lighting simulation tools.
Challenges.The main challenges of using AR and VR for design support are (1) that it is very difficult to translate the changes made using the AR and VR systems into Building Information Modelling (BIM) models, and (2) that it is not possible to archive AR and VR outputs for later review or to record the experiences that the user had in AR and VR environments.

Design review
AR and VR facilitate the communication of design intent, enables designs to be reviewed in a more efficient way; issues can be identified more easily, and sign off can be done more efficiently.
AR examples.Dong et al. [51] presented an AR system for collaborative design review and planning of operations in civil engineering applications.Schubert et al. [156] presented an AR-based decisionsupport system for design review of urban design problems.The presented system combines physical models and hand-drawn sketches with virtual 3D models in a multi-touch table.The AR augmentations are enabled through tablets and mobile phones.
VR examples.Berg and Vance [16] evaluated the effects of using VR for design reviews by conducting a study with manufacturing engineers to carry out design reviews in a projection-based VR environment.The system enabled the participants to view and interact with the geometry at real scale.The authors reported that the participants gained a better understanding of the spatial relationships between product components as well as the interactions required to assemble the product.Dunston et al. [56] presented a VR system for design review of hospital patient rooms.The authors indicate that VR-enabled design reviews improve interaction and have a greater influence on design decision-making.Boton [20] presented a method to support constructability analysis meetings using VR environments.The method enables BIM-based construction simulations to be exported into a VR application for immersive visualisations.
VR has been employed as well to asses occupation comfort of a variety of spaces from underground shopping streets [165] to aeroplanes cabins [37].Liu and Kang [99] used VR to explore the correlations between urban environments and visual and audio comfort in streets by varying the street width to building height ratio, while Echevarria Sanchez et al. [58], developed a VR system that enables to assess how visual aspect can reduce noise discomfort.
Challenges.Similar to design support, the main challenges of using AR and VR for design review include the difficulty to translate design changes to BIM models and to record the experiences and discussions that the users had within the AR and VR environments.A two-way seamless and automatic communication between BIM models used for construction and AR and VR models is required.These challenges have been partially addressed by the work of Dris et al. [53], which proposed an ontology that enables a bi-directional link between a BIM model and a VR application.

Construction support
The use of AR and VR for construction support can be grouped into four areas: (i) Construction planning.The primary objective of AR and VR in this area is to anticipate potential problems and improve delivery.VR focuses on creating immersive construction simulations; while AR focuses on visualising virtual object to be constructed directly on sites.
AR examples.Mutis and Ambekar [118] investigated the existing AR challenges to visualise virtual objects on construction sites, and Chu et al. [35] investigated how AR can be used to improve information retrieval from BIM models and thus reducing time in construction planning tasks.
VR examples.Turner et al. [173] analysed the combination of VR and discrete event simulations for supporting complex manufacturing and assembly tasks.Being able to simulate complex construction operations in advance will reduce the risks and potential delays.Other research has focused on reducing time delays caused by design changes.For instance, Davidson et al. [39] present a VR environment in which the bill of quantities is updated in real-time according to changes made by the user, expediting and facilitating design changes during construction.Mastli and Zhang [107] presented a VR system to help plan crane routes during the construction of highways to minimise the length and time of the crane trips, thus reducing interruptions to traffic.
Challenges.The main challenges limiting AR and VR for construction planning are (i) the lack of interoperability between BIM systems and AR and VR models, and (ii) the difficulty to automatically update BIM models and construction schedules from the AR and VR systems.
(ii) Progress monitoring.AR has the potential to improve significantly the ability to identify what has been built and what is missing in a quick and understandable manner.This is very important as early detection of schedule delays is critical to ensure timely delivery.
AR examples.Golparvar-Fard et al. [68] presented an AR system that automates progress monitoring and provides a colour-coded overlay to easily identify sections of the construction site that are ahead, on, or behind schedule.More recently, Zhou et al. [195] presented an AR approach to support the inspection of segment displacement during tunnelling construction.The approach overlays a quality control baseline model onto the real segment and measures the differences.
VR examples.VR enables to carry out progress monitoring remotely, especially for dangerous sites.For example, Robbins et al. [145] presented a VR system that helped tracked maintenance operations of fusion reactors.More recently, Pour Rahimian et al. [138] presented a system that enables progress monitoring of buildings in a virtual environment using BIM data and real images from construction sites.
Challenges.There are still many technical challenges to be resolved for AR-enabled progress monitoring including (i) automatic 3D reconstruction of building components from point clouds, and not only the creation of polygon meshes; (ii) reliable marker-less object recognition; and (iii) comparison and automatic update of as-built and asplanned models.Challenges for VR include (i) the difficulty of managing various data sources and technology pipelines used in the construction site and for VR visualisations; (ii) the lack of integration between data standards that make difficult the integration of BIM data, photogrammetry and VR platforms; and (iii) the lack of ways to validate the progress monitoring data (i.e.does the VR environment truly reflects reality), and lack of an indication of accuracy.
(iii) Construction Safety.Construction safety is one of the clearest use-cases for AR and VR.AR can be used to provide safer working environments by contributing to hazard identification, safety inspection; and VR can support safety education.Li et al. [93] presented a review of the AR and VR applications in construction safety, in which defined three main application domains, i.e. hazard identification, safety training and education, and safety inspection and instruction.Moore and Gheisari [112] also carried out a review of the use of AR and VR for construction safety noting that AR and VR help workers reduce risks and carry out tasks in a safer manner AR examples.AR is the ideal technology to display hazard information to construction workers, e.g.[110].Park and Kim [132] presented an AR framework that enables workers to improve identification of field safety risks, improve risk recognition, and enhance realtime communication between the construction manager and workers.Albert et al. [5] presented an AR system to facilitate hazard identification in construction sites.Pham et al. [136] developed an AR system that supports hazard understanding and recognition in construction sites.
VR examples.Sacks et al. [150] presented a VR system for construction safety training.The authors note that VR training was more effective in terms of maintaining trainees' attention and concentration.Hasanzadeh et al. [73] investigated the risky behaviours of workers while performing a simulated roofing task in a VR environment.The authors report that users of the system perceived the task and the risk of falling very similar to the real-world.Shi et al. [159] used a VR system to assess distinct types of reinforced learning methods on the behaviour of construction workers associated with fall risks.
Challenges.Lack of customised hardware, lack of interoperability among systems, uncomfortable devices, and lack of standardised evaluation limit the effectiveness of both AR and VR for construction safety [112].
(iv) Operative support.AR is the ideal technology for this area, as it focuses on supporting workers to carry out a task in the most efficient manner.AR focuses on finding ways to provide more contextual information to site workers; while VR focuses on teleoperation of construction equipment.
AR examples.Nee et al. [122] present examples of how AR can be used to support assembly tasks.The most used approach is to overlay information about the task to be carried out in the field of view of the worker.For instance, the AR system will highlight what tools and what components are necessary to complete the task at hand.This is an area undergoing intense study, as it represents a huge potential to tackle the low productivity common in the construction industry.Fazel and Izadi [62] presented an AR system that supports construction workers to construct complex double-curved brick walls.Visual guides are displayed on an HMD that indicate the correct position and orientation of the bricks.Chalhoub and Ayer [24] presented an AR system that supports workers to install electrical installations at the correct positions.A 3D model of electrical conduits is overlaid at the correct position in the room obviating the need for 2D drawings.Ahn et al. [4] presented an AR approach for visualising information to improve construction panel manufacturing.The approach helps to improve the quality of the final manufactured product by reducing the offset distances and ensuring that they are within the tolerance levels.Wang et al. [183] presented a systematic review of the use of AR for assembly support.The authors note that the major limitations are tracking and registration issues, lack of collaborative AR interfaces, lack of automated 3D workspace scene capture, insufficient context-awareness, and sub-optimal information representation methods.
AR can also be used to support machinery operators.For example, Chen et al. [29] presented an AR system to assist crane operators for increased safety by recommending movement paths and providing warnings to avoid clashes.Marker-based systems for teleoperation of cranes have been investigated as well, e.g.[32].Chen et al. [28] used marker-less AR for crane teleoperation and investigating the operators' attention using three views: focused views, ambient views, and alert views.Authors report that the AR interface reduced the potential collisions and increased operator satisfaction.
VR examples.Teleoperation of robots using VR has been investigated widely; however, for construction applications, few research efforts have been reported in literature, e.g.[75,169].The main focus has been on developing construction robots working at disaster sites and extreme environments [81], and for demolition tasks [80].
Challenges.The main issues of using AR for operative support are (i) the lack of safety-approved hardware, which bars large-scale deployment; (ii) the low accuracy and unreliable tracking and mapping that are not at a level that will ensure correct assembly and construction; (iii) short battery life of devices, which significantly diminishes usability (HMDs batteries usually last only 30 min); and (iv) the potentially limited internet access on construction sites, which would impede connections with enterprise systems and federated BIM models to get information, update and record changes.The main challenge for VR teleoperation systems is that while they increase safety, they also highly reduce efficiency, as the operators take significantly more time to carry out the same task [80].This is caused by the latency of communication between the construction equipment and the remote operator, unsatisfactory visual information, and lack of tactile and body sensory feedback.

Operations and management
AR has great potential to support building operations and management because it can provide useful information to site workers that operate and maintain the facilities.VR can provide a way to operate the facility remotely in an immersive environment.A combination of both technologies can support field and remote office workers at the same time and improve collaboration, as detailed by El Ammari and Hammad [60].
AR Examples.AR systems can support building maintenance, repair, and inspection tasks by directing technicians to the specific equipment, showing the tasks to be completed and providing technical information in context.For instance, Irizarry et al. [76] presented an AR system that provides information stored in BIM models to users for supporting facility management tasks.Palmarini et al. [130] presented a thorough literature review of AR for maintenance in the manufacturing sector.The authors noted that AR had been used to support varied tasks, including inspection and diagnosis, repair, and assembly/ disassembly tasks.Neges and Koch [123] presented an AR system that supports onsite building maintenance by creating AR-based maintenance instructions, tracking of the maintenance object and visualisation of affected parts.Neges et al. [124] presented an AR framework that supports facility maintenance operators when navigating indoors.
The framework combines a step counter device and visual live video feed to provide accurate indoor navigation support.Baek et al. [11] presented an AR approach for facility management that presents location-specific data using image-based indoor localisation.The approach estimates the user's indoor position and orientation by comparing the user's perspective with a predefined BIM model.AR has been used as well to visualise thermal conditions of spaces and improve thermal management [188].VR Examples.Carreira et al. [23] presented a VR system that enables to carry out facility management tasks and control a building management system remotely.Shi et al. [160] explored how multi-user VR can improve facility management by enabling various experts in remote locations to examine the same facility in a virtual environment.Jiang et al. [77] noted that using VR to support operation and maintenance of facilities enables better visualizations of maintenance scheduling decisions along with improved maintenance resources distribution.
VR can also be used to provide more precise disaster scenarios and post-action evaluations, for example, to simulate and evaluate indoor evacuation under fire emergency conditions [22], and to assess the level of preparedness for building evacuations during earthquakes [101].Yang and Li [189] used VR to test different strategies to evacuate high-rise buildings during fires.Lin et al. [96] used VR to explore the influence of virtual crowds on human evacuation behaviour during building fire emergencies while Montecchiari et al. [111] used VR to analyse human behaviour during the evacuation of ships.
Challenges.Similar to construction support, AR systems for operations and management face challenges related to tracking and registration, collaborative interfaces, context-awareness, and information visualisation methods.The main challenges facing the use of both AR and VR for facility management are the lack of integration with other facility management systems, the low accuracy and speed for updating information across several systems, and the difficulty to archive and revisit AR and VR experiences.

Training
VR can provide realistic scenarios in which the users acquire knowledge and skills from performing simulations of real activities rather than acquiring general information from a decontextualized body of knowledge [89].Both AR and VR technologies can reduce the cost of training, by simulating the use of expensive equipment, simulate dangerous environments, reducing travel costs, as well as improving health and safety.
AR Examples.AR can be used as well to train construction machinery operators.AR can display instructions to the operators [179] and simulate realistic scenarios [157].Sekizuka et al. [158] presented an AR system to train and evaluate hydraulic excavator's operators.The system uses an omnidirectional camera, and an HMD enabling a similar operating experience as that of a real excavator.Eiris Pereira et al. [59] presented an AR system to improve fall hazards training.AR can be used to improve education, as well.Turkan et al. [172] presented an AR system to teach structural analysis and to help students understanding the behaviour of structural elements in a three-dimensional context; while, Luo and Mojica Cabico [103] presented an AR system to teach students about complex bridge structures.
VR Examples.Wang et al. [178] presented a review of the use of VR for construction training.On example for VR training in construction is VR systems to train equipment operators.For instance, Dong et al. [50] presented a VR system to train crane operators.Barkokebas et al. [13] presented a VR system to train workers on the maintenance and repair of construction machinery.Zhao and Lucas [194] presented a VR-based training that provides a virtual working environment where workers can rehearse tasks involving electrical hazards in a safe manner.Sampaio and Martins [151] developed a VR system to help understanding complex bridge construction methods.VR can also improve spatial understanding.For example, Fogarty et al. [63] investigated how VR can help students to understand complex spatial arrangements in structural engineering.The authors report that students could identify and visualise buckling modes more accurately.
Challenges.The main challenges of using AR and VR for training are (i) that there is a shortage of experts to produce AR and VR content, (ii) a lack of systematised evaluation processes, and (iii) lack of integration with existing qualification standards.
Table 1 summarises the use-cases presented here.It presents the benefits and challenges that AR and VR implementation entail, examples in literature, and the adoption levels (see Section 4.2).Note that the benefits and challenges were compiled from the data collected in the FGDs, the adoption levels from the questionnaire, and the examples from the literature review.

Usage analysis of AR and VR in the AEC sectors
The AR and VR levels of adoption in the AEC sectors remains low [40].A 2017 study among large construction companies and infrastructure providers into the maturity and applicability of AR and VR found that only 37% of construction companies have some experience with AR and VR [117].This section presents a more granular and detailed analysis of the AR and VR usage in the AEC sectors.

Overall adoption, expertise levels, and future investment
Fig. 3 presents the results of the quantitative analysis carried out in this study that provide an overall indication of the AR and VR levels of adoption, expertise levels, and future investments.Approximately 60% of the companies have less than three years that they have tested AR or VR., and ~62% of the companies have used AR and VR in less than 5% of their projects.Around 80% of the participants consider that they have an intermediate level of expertise in VR.However, 8.8% report that their companies have not used VR; 14.7% that are in early testing without client involvement; 47.1% basic implementation, that is used in pilot projects with clients or internally; 26.5% partially used, that is used in up to 25% of projects with clients, and 2.9% fully implemented, that is used with more than 25% of projects with clients.Regarding AR, around 80% of the participants consider that they have an intermediate level of expertise as well.But in this case, the level of adoption is lower.Twenty-three-point-five per cent have not used AR, 23.5% have carried out early testing, 29.4% basic implementation, 23.5% partially used, and none of the companies has fully implemented AR.Regarding expected future investments, 73.5% report that it is likely or very likely that their companies will invest in VR in the next three years; while for AR, the figure is 61.8%.(continued on next page)

AR and VR adoption levels per use-case
that training has a mean value slightly higher than construction support and operations and management.Looking at Fig. 4 is clear that VR has been used mostly during the design part of the project life-cycle.Pratama and Dossick [139] conducted a study with AEC companies and identified that the majority of companies use VR mostly for immersive building walkthroughs.Fig. 5 presents the AR levels of adoption per use-case.In this case, user engagement has the highest median and mean values, 3 and 2.47, respectively.Design support and design review have a median value of 2; while construction support, operations and management, and training have a median value of 1. Overall the VR adoption levels are higher than AR, and the use-case with the highest adoption level is user engagement.The difference in levels of adoption between VR and AR can be explained in part because in general people are more aware of the technology.A 2018 online study found that 90% of consumers in the U.S. and the U.K. are aware of VR, while only 65% are aware of AR [21].Also, the commercial applications of VR are more closely related to AEC practice than those of AR.

State-of-the-practice in AR and VR adoption in the AEC sectors
The state-of-practice in AR and VR adoption in the AEC sectors is not homogeneous.Medium and small companies lag significantly behind large companies that have the resources to acquire equipment and have in-house content and technology development teams.Large companies have the capacity to test AR and VR for all the use-cases; while small companies can only focus on the most basic use-cases, i.e. stakeholder engagement and design review.Small companies have to rely on the limited off-the-shelf solutions in the market to implement AR and VR, thus reducing their capacity to use AR and VR.For instance, automatic BIM to VR solutions is being used to quickly deploy VR models for client engagement and design review (see 5.3.2).Besides those two use-cases, large companies have also been focusing on using VR for training and AR for construction support.Note that, as shown in the results presented in Figs. 4 and 5, in very few instances, the companies have rolled out the technology completely or used it in projects with clients.
Large AEC companies are in a similar position than the more advanced research groups in academia.This is because the use of AR and VR in the AEC sectors is being driven primarily by collaborations with academia.A large proportion of the research referred here is a collaboration between academia and industry, thus are a good example of what is happening in industry.This is because AEC sectors have a poor research track record [18], with low investments and a weak innovation culture [64,105].So large AEC companies leaned on the academia expertise to start research in the area.However, more recently, business to business collaboration between technology development companies and large AEC firms are taking place, which will drive further research efforts in industry.This is exemplified by the Microsoft HoloLens partnership program that includes AEC companies that have contributed to direct the development of AR technologies for the AEC sector.

Research agenda
AR and VR systems are still immature technologies, and various issues still need to be resolved including form factor, see-through quality, the field of view, image quality, handling occlusion, vision correction capabilities, etc.However, this section focuses only on presenting a research agenda specific to the AEC requirements.Note that this research agenda is the result of following the process presented in Section 2.3, and it is based on the findings acquired from the qualitative analysis, quantitative analysis, and a literature review.In other words, the proposed research agenda is the result of a combination of (i) concerns and issues raised from industry practitioners, (ii) views and expectations from technology development companies, and (iii) - [22] Note that the information presented here was compiled from the data collected in the FGDs, the questionnaire, and literature review.* Adoption levels discussed in 4.2, presenting [median] and [mean] values (1 = not used, 2 = early testing, 3 = basic implementation, 4 = partially used, 5 = fully implemented).identified gaps in literature review.The intention is that the research agenda reflects the actual needs of practitioners and guides the academics' future research to improve adoption.
The proposed agenda consists of three categories: (1) Engineeringgrade devices, (2) Workflow and data management, and (3) New capabilities.Fig. 6 presents a mapping of the proposed research agenda.The research topics of the research agenda are plotted according to the estimated time required to address them (horizontal axis) and the technology readiness level (vertical axis); which range from a low level that requires more research, i.e. proof of concept, to a high level close to  commercial deployment that requires less research and more development, i.e. system development.The three categories are identified with different colours, and the enabling technologies are listed on the right side of Fig. 6.The arrows represent dependencies among research topics.

Engineering-grade AR and VR devices
This category compiles the technical requirements that AR and VR devices need to be used for the six identified use-cases.

Comfort and safety approved
Current AR and VR HMDs are not sufficiently comfortable to be used for long periods of time.Many of the identified use-cases require long usage of HMDs, e.g.design review; which makes them unfit for purpose.User studies should be carried out to identify the sources of discomfort and the compiled data should be used to define the characteristics of more comfortable devices.AR devices have the potential to interfere with health and safety (H&S) regulations in construction sites.Most of the current AR devices most probably could not meet the minimum requirements to be approved for use on-site.AR devices must be robust so that they can withstand rough industrial environments, and they must not cause a threat to the user or others around them.AR devices must not prevent other safety equipment or limit the user senses from operating normally.Studies to test the use of AR devices, worn in combination with other safety equipment, in a variety of environments and situations should be carried out.Then, evaluations of risks, mitigation methods, comparisons of benefits against risks should be carried out.Other primary concern is that poorly designed AR interfaces may distract workers and increase cognitive loads unnecessarily.Various types of AR interfaces should be evaluated to assess their potential effects on H&S regulations.In addition, AR could be used to provide H&S warnings to workers.Some research exists in this aspect, for example, Kim et al. [86] note that AR user interfaces should minimise cognitive and physical demands, optimise the presentation of data and provide smart health and safety warnings.Eye-tracking data can be used for this purpose as it provides information on how visual attention is distributed and the levels of cognitive loads can be determined [17].

High accuracy tracking
Tracking objects in real-time provides an invaluable tool for AR because once an object has been tracked, its position can be used to overlay virtual objects and annotations.There are various ways to track objects in the real environment, most of them are a combination of marker-based approaches, in which an image is used to align an existing 3D model, or geometry and position data, of the real object to be tracked to align the augmentations with the real object, e.g.[177]; and marker-less-based approaches; in which markers are not required and only the features of the object's geometry and texture are used [113,134].In general, marker-based tracking is more robust and consistent but requires the placement of trackers onto objects and scenes before augmentations can be displayed.the other hand, marker-less tracking can be used without initial preparations, but the accuracy is usually lower and is highly dependent on the characteristics of the object to be tracked.For example, objects with very few features such as large flat surfaces, similarities among objects, or large objects and scenes are particularly problematic.Specifically, for the AEC sector, 3D models tend to be very detailed, and they will require high accuracy tracking.Current object tracking approaches are not robust enough, and the large similarities among objects in buildings and industrial facility are still a challenge that needs to be addressed.Most probably, a combination of marker and marker-less approaches, including sensor data as well will be required to enable such building-wide accurate tracking, e.g.[192].

Improved indoor localisation systems
Indoor localisation, or positioning, systems locate objects or people inside buildings and facilities using a myriad of sensory information such as radio waves, magnetic fields, acoustic signals, inertial data, images, etc. Accurate indoor localisation is essential for AR and VR applications as the applications require accurate locations of people and objects to generate correct augmentations.For example, indoor localisation is required to guide workers within a facility, identify the shortest escape routes, and identify assets and equipment.However, there are not sufficiently robust indoor localisation systems that work in complex layouts and that distinguish different but similar rooms.Inertial Measurement Units (IMUs) are the most widely used approaches for indoor localisation.IMUs use accelerometers, gyroscopes, and magnetometers (usually found in mobile devices) to calculate the current position using a previously determined position.This approach uses data from mobile devices to estimate the location of the user within a building, e.g.[190].However, this approach is limited by noisy sensor data, unconstrained mobile device placement and the complexity of human activities.Further research should focus on using a combination of methods, for example by leveraging WiFi signals [30,180], Bluetooth beacons [196], and magnetic fields [161].

Dynamic 3D mapping of changing environments
The capacity to map continually changing environments such as construction sites is essential for the adoption of AR and VR.The main challenges for 3D mapping construction sites are twofold (1) new components are included constantly, e.g. when new walls are built, or new windows are installed, and (2) the environment is highly dynamic, workers and machinery are continually moving, materials are transported to different locations, etc.; all of which affects makes more difficult the mapping processes.Accurate dynamic mapping of the construction site is necessary, for instance, for indicating the location of components to be installed, proposing alternatives in real-time when clashes can potentially occur, confirming that components were installed at the right location, and showing what component needs to be installed next, among others.Current mapping technology, such as simultaneous localisation and mapping (SLAM) is not sufficiently robust and accurate and have not been tested successfully in such adverse conditions [85].A mapping system is required that constantly captures and updates the physical world, to enable augmentation and virtualisation of dynamic environments.For example, some preliminary research has been carried out in this area in which a drone and a VR system are combined to scan a 100 m 2 environment [115] and dynamic mapping of furniture and people using depth cameras [127].

Explicit indication of accuracy
AR and VR will not take off in the AEC industry if the technology cannot provide some value of certainty.Inaccurate AR and VR tools that lead to errors and mistakes will not be adopted.For example, if the accuracy is unknown for an AR pipe mapping system that indicates the available areas for excavation, users will not use it risking breaking a pipe.However, if the AR system provides an indication of its accuracy, then it is easier for users to trust the system, and they can take the necessary actions to manage that uncertainty.Thus, it is essential to have a clear indication of accuracy and a clear graphical representation of those metrics.Studies that formalise and standardise the measurement of inaccuracies and identify the sources errors are imperative.A unified graphical representation for uncertainty and confidence levels are also required.Research studies should demonstrate that errors and inaccuracies are clear to understand and statistically proven to be correct with an appropriate confidence level.Research on what are the best ways to visualise uncertainty in AR and VR devices and to calculate uncertainty in real-time are also required.

Larger model capacity
A limiting factor of current AR devices and some VR mobile devices is the limited capacity to load large and complex models.3D and BIM models commonly used in the AEC sectors are very complex, contain very many detailed elements, requiring a large memory and processing capacity to handle appropriately.Besides current advances on mobile memory and processors, these limitations can be addressed using a variety of approaches including (i) loading only a small portion of the model that is within the field of view in a dynamic manner [102]; leveraging levels of detail (LOD) to display detailed models of components only when they are close to the user's view and a low detailed model when they are far [78]; and (iii) exploring encoding, packaging and streaming approaches to reduce the amount data to be processed [74,164].

Longer battery life
One of the main limitations of using AR and VR on-site is the insufficient battery capacity of the devices.For example, anecdotical evidence suggests that AR HMDs only last around 30 min of heavy use, which is insufficient for most of the work carried out on-site.The main problem is that access to electrical power at on-site locations is not readily available and other more critical equipments take precedent.Regardless of research on new battery technologies, there are various ways to address this limitation to improve the usability of AR and VR devices on-site including: (i) easily replaceable batteries like other construction equipment such as cordless drills; (ii) incorporating larger batteries outside the AR and VR HMDs and tablets that are connected to the device and carried in the worker's tool belt, and (iii) carrying out the processing not onboard of the AR and VR devices but on an external computing device and stream only the visual information to the AR and VR devices; which will decrease the power demand of the AR VR devices.

Workflow and data management
This category groups the research required to manage data and processes in AR and VR workflows effectively.

Archiving AR and VR content and experiences
Large amounts of AR and VR content is being created constantly.However, there are no existing approaches to easily capture this content for preservation, instruction, and outreach purposes.AR and VR content usually can be only accessed through compiled applications or through specialised authoring software.More importantly, there is no way to easily record and archive the experiences that users experiment when interacting with AR and VR applications.For example, when a trainee goes through a VR training session, the used 3D models and the VR application could be stored, but there is no robust way to record his or her experience.Other users cannot experience the same situation.Having access to other users' experiences has the potential of providing highly valuable insight.For example, this issue has been highlighted in the context of supporting VR experiences for teaching and learning in library settings [71].It is imperative to develop methods for recording AR and VR content and experiences so that they can be shared with other users.Current technologies are only able to provide a first-person video recording of the AR or VR experience which does not convey the richness of immersive experiences.

Visualising data in a 3D spatial and temporal context
The visualisation and analysis of data are usually carried out on 2D computer displays without any spatial context or reference to the physical world from which was originated.However, many data analysis tasks can be significantly enriched by visualising data relevant to the task in its original context.AR and VR environments provide the basis to visualise data in a way that improves its value and facilitates its use and interpretation.For example, Rowen et al. [149] noted that visualising data in context using AR displays greatly improves the performance of maritime ship operators.Unfortunately, current AR and VR data visualisation approaches are just an extension of traditional desktop visualisation techniques that use the "window paradigm", in which the data is visualised and organised on defined rectangular areas.New innovative visualisation approaches are required that enable to visualise data, both spatially and temporally, that transcend the window paradigm commonly used in paper and tablets.In addition, the implications that new AR displays represent for data visualisation in spatial context must be investigated.Willett et al. [185] presented a conceptual framework that formalises and categorises different types of visualisations based on the relationships between real-life objects to which data corresponds and the visual and physical representations of their data.The categorisation includes (a) non-situated visualisation, (b) situated visualisation, (c) embedded visualisation, and (d) embedded physicalisation.This is a good first step into formalising research in this area; however, many research gaps remain such as ways to determine (i) when richer insights are gained by visualising data in context and when it leads to unnecessary complexity, (ii) how to measure the levels of perceptual distortion generated by visualising data in context, and (iii) how different are these distortions to the ones experimented using traditional 2D data visualisation techniques.Lastly, research efforts should be placed on investigating the best approaches to visualise underlying and meta-data related to real-life objects, virtual augmentations and data as well.There is a lack of integration among the visualisation of 3D models, sensor data, time-series data, spatial relationships, object hierarchies, etc. [41].

Developing data exchange standards
Current AR and VR technologies are not compatible with AEC standard data formats such as Industry Foundation Classes (IFC) [94]; which makes the integration of standard AEC software packages and AR and VR software tools difficult.A few software tools exist that convert between IFC and AR and VR compatible formats.However, there is no standardised approach, and issues remain relating to object information, materials and textures, Levels of Detail (LOD), meta-data, among others [43].There are some recent research efforts in this area.Dris et al. [53] proposed an ontology that improves the use of Building Information Modelling (BIM) models for virtual reality applications.The ontology enables a bi-directional link between a BIM model and a VR application.Lin et al. [97] developed a VR approach compatible with BIM models to improve communication between the design teams and healthcare stakeholders.Regarding AR and VR open-source standards, the Khronos Group launched the OpenXR 1.0 in 2019 [82].OpenXR is an open-source standard to address the fragmentation of AR and VR development platforms and hardware devices.OpenXR will eventually consist of two components an Application Programming Interface (API) and a device plugin interface that will enable AR and VR applications to run in any system that uses the OpenXR standard.Research in this area, besides the extension of existing standards, should also focus on enabling robust data exchange among different standards.For example, instead of extending BIM standards with VR capabilities, investigating how VR and BIM standards can exchange data seamlessly.

System integration with other built environment systems
AR and VR systems need to be seamlessly integrated with other software systems used in the AEC industry such as facility management (FM), building management systems (BMS), enterprise databases, BIM cloud solutions, SCADA systems, etc.A seamless integration, without the need for programming intervention, will open new use-cases between the involved parties; which will create a robust technology ecosystem [104].Data exchange standards, as discussed in the section above, are not enough to a achieve a robust technology ecosystem; which requires real-time integration of AR and VR systems with application programming interfaces (APIs) of other built environment systems.The development of the ecosystem will require the collaboration of many stakeholders with different interests and responsibilities of different development toolchains, but that still influence the overall results.These stakeholders are part of the requirements value chain.Understanding the structure and functioning of requirements value chains is essential for developing a robust ecosystem [65].Research on software ecosystems and on the best approaches to bring together different AR, VR, and built environment systems to develop a robust ecosystem is necessary [175].Few research efforts have been made in this area.For example, Wang et al. [182] presented a conceptual framework to extend BIM functionalities to construction sites by leveraging AR.Future research efforts should draw inspiration from existing knowledge in other fields such as enterprise systems integration [119]and data virtualization [47,176].

Multi-user and multi-device capabilities
Multi-user and multi-device capabilities are essential for AR and VR systems to be effectively employed in the identified use-cases.For example, multi-user immersive experiences interacting on the same BIM model with multiple viewports can facilitate engagement with clients, designers, and stakeholders.Two-way, multi-user communication between office workers and site workers will reduce the time to identify anomalies and take effective actions.AR and VR systems should enable many-to-many communication capabilities including (i) various devices of the same type and different brands (e.g.many VR devices), (ii) various devices of different types and different brands (e.g.many AR devices, VR devices, and tablets), (iii) various devices of different types, different brands and in different locations (e.g.many AR devices at the construction site, VR devices at the head office, and tablets at another stakeholder office.).Some research has been reported in this area.For example, Abramovici et al. [2] presented a framework to create AR collaboration assistant systems that enable team-wide communication and the display of warnings and other coordination-related indications.Du Shi et al. [54] introduced a cloud-based multiuser VR system that facilitates interpersonal project communication in an interactive VR environment.

Addressing security, privacy and data ownership issues
AR and VR toolchains and software applications do not generally handle information security and privacy issues out of the box.As with any other emerging technology, there are new security and privacy issues inherent to the new technologies.Significant efforts have been made to anticipate and address security and privacy issues [147].Concerns include physiological attacks, deceptive holograms, virtual clutter, inappropriate content, bystander privacy, and invasive applications, among others [91].Data ownership needs to be addressed, as well.For example, the ownership of virtual objects and virtual spaces need to be established as well as controlling access to personal virtual objects, preventing unwanted content from other users or parties, negotiating access to other user's content, and navigating partially shared AR and VR environments, among others.An overarching framework and guidelines for AR and VR developers is required to take into consideration the already identified security and privacy concerns.But, more importantly, an investigation needs to be carried out into new potential security concerns inherent or specific to the construction industry (e.g. in construction sites, prefab manufacturing facilities, and facility management).

Develop an upskilling roadmap
A qualified AEC workforce is a crucial factor for the successful adoption of AR and VR.The AEC workforce will need to be trained on the use of new AR and VR systems.It is, therefore, essential to (i) identify the required skills for each of the use-cases, (ii) to gather an indication of the qualifications of the existing workforce, (iii) identify gaps, and (iv) propose a roadmap to upskill the workforce.Similar studies have been carried out on the delivery of high energy performance buildings; as it was found that a significant factor hindering successful delivery was the lack of the qualifications of the construction workforce [52].Simpson et al. [163] carried out preliminary research to set the basis for developing an upskilling framework for the AEC workforce.The authors note that initiatives focused on training alone will not be enough; but, skills supply, skills demand, and skill management should be considered.

New capabilities
This section compiles the research areas required to de develop new technical capabilities that current AR and VR systems lack.

Object and gesture recognition
Object recognition is a key technology behind many state-of-the-art technologies.For example, it enables driverless cars to recognize a stop sign or to distinguish a pedestrian from a lamppost.Similarly, automatic identification of objects and gestures is indispensable for both AR and VR applications.Ideally, AR and VR systems need to recognise every object in the users' surroundings without the need for pre-existing 3D models or markers.If AR and VR systems can recognise hand gestures in a robust manner more intuitive user interfaces can be developed, and other use-cases can be unlocked, e.g.teleoperation of machinery [26].There is a large amount of research being carried out in this area, but there is a lack of integration with AR and VR systems.
Research is required in developing optimal ways to integrate object recognition algorithms into AR and VR devices.More importantly, the most used machine learning algorithms for object recognition need very large labelled datasets for training.These datasets are not widely available for the tasks required in the AEC industry.To facilitate the use of state-of-the-art object recognition algorithms, efforts should be placed on (i) creating the necessary real-life datasets, (ii) creating synthetic datasets in case real-life datasets is prohibitively expensive [72], (iii) leveraging transfer learning techniques by using datasets developed for other tasks [197], and (iv) investigating the applicability of unsupervised machine learning algorithms that do not require large labelled datasets for training [49].

Real-time model modification
AR and VR systems must be able to modify virtual models in realtime and during run-time.Usually, BIM and 3D models used for AR and VR visualisations are not the same and are not linked; which greatly hinders usability.For example, if during a VR-enabled design review meeting changes are made to the model, the changes have been only made in the VR model and not in the BIM model used to generate fabrication drawings.This lack of integration reduces the usability of the VR system and introduces additional work and sources of errors.In the case of AR, the virtual objects used for augmentations may not represent the physical object accurately due to errors during mapping and scanning.AR systems should be capable of proposing modifications to the existing virtual objects based on later observations.Two-way modifications between AR and VR models and BIM models should be possible in real-time.Research in this aspect has been reported in literature.The approach of Du et al. [55] enables real-time synchronization between BIM data and VR applications.The approach enables to update a BIM model based on the changes made in VR application automatically and in real-time.Note that there are commercial solutions as well that provide varying degrees of automated BIM to VR synchronisation, such as IrisVR, InsiteVR, The Wild, Enscape, among others.These solutions enable to convert a BIM model into a model that can be visualised in a VR environment, and in some cases, reflect changes made in the BIM model into the VR model.However, VR to BIM and bi-directional synchronisation have not been addressed satisfactorily yet.One of the main factors that make difficult bi-directional synchronisation are the different data schemas used for BIM models and VR models and the lack of compatible databases systems for federated models; thus, this should be one of the priorities for future research.

Diminished reality and real-time occlusion
Contrary to AR, which adds computer-generated objects to the real environment, diminished reality uses various technologies to hide reallife objects from the users' view.Real-time occlusion enables realistic augmentations by not rendering parts of virtual objects that are behind physical objects in the perspective of the user's point of view.Both diminished reality and real-time occlusion are necessary capabilities for future AR systems as physical objects are often in the way of augmentations.For instance, when displaying a building model in the location where it is going to be built, trees and land topography may be in the way.Diminished reality and real-time occlusion could be used to make those objects disappear, making the augmentation more perceptually clear.Diminished reality uses four main techniques [114]: (1) diminish, which degrades or distorts visual fields; (2) see-through, which covers real objects with imagery of their occluded background; (3) replace, which overlaps a virtual object on top of a real object.;and (4) in-paint, which generate plausible background images based on the surrounding environment.These capabilities are required for many use-cases such as construction planning, clash detection, visualisation of hidden infrastructure, etc.Note that some limited occlusion capabilities are enabled by existing AR frameworks (e.g.ARKit, ARCore, ARToolkit), but they do not provide the entire capabilities as described above.

Automatic environment capture
An important new capability for AR and VR systems is a robust system that can capture the surroundings of the user in real-time.For example, VR systems need to be aware of the users' surroundings to alert the users of nearby objects to avoid collisions; AR systems require an accurate mapping of the surroundings to overlay the virtual objects properly.Automatic environment capture is essential for situations in which the environment is continuously changing, such as construction sites that require to be constantly scanned.For instance, accurate environment capture can be used to display pipe models on the ground and cables hidden behind walls.A variety of technologies will be necessary to develop robust automatic environment capture systems including a variety of sensors on board of AR and VR devices, external sensors and cameras, and various algorithms including object recognition, automatic meshing, and automatic 3D modelling.Note that this capability can be delivered only by combining results from other themes of the research agenda such as high accuracy tracking, indoor localisation systems, dynamic 3D mapping, real-time model modification, and diminished reality and real-time occlusion.

Real-time integration with internet of things (IoT) devices
The integration of built environment hardware and software systems, including AR and VR, with real-time data from IoT devices, presents a step-change opportunity to improve operational efficiencies and support decision-making.IoT solutions have various applications in the built environment ranging from smart buildings [155], structural monitoring [42], and preventive maintenance [27].For AR and VR applications is also essential to enrich the virtual environments and objects with real-time data.For example, during structural inspection tasks data from IoT sensors displayed in context using AR will enrich the visual inspection enabling a more complete and accurate assessment of the structural condition.A few examples of AR and VR integration with IoT have been reported in literature including a framework for a seamless IoT and VR integration [191], and a system that visualises the source of wireless signals [133].For the AEC sectors, the major challenge to integrate AR and VR systems with IoT data is its integration with BIM systems using open standards [38].Tang et al. [168] note that to facilitate the integration between IoT devices and built environment systems the following is required: integrating built environment systems with relational databases, a new data schema, a new query language, leverage semantic web technologies, and develop and hybrid approach for data integration.Note that Chi et al. [33] identified that cloud computing and mobile devices were some of the main technologies that will drive the development and implementation of augmented reality in the AEC sectors.

Real-time physics simulations, predictive and prescriptive analytics
A step-change in the value that AR and VR can bring to the AEC sectors will be achieved when AR and VR can visualise simulations of future situations, visualise predictions of the outcomes of actions, visualise what-if scenarios and optimal action plans.Predictive analytics have been employed for a wide range of applications in the AEC sectors from predicting cooling loads [61], occupational hazards [154] to construction costs [46].Simulations and optimisation have been widely employed as well, including optimal structural configurations [44], mechanised construction [126], and waste management [48], among many others.The visualisation in AR and VR devices of all these types of simulations and predictive and prescriptive analysis will enable better and more efficient decision-making.Few studies have been carried out on the visualisation of simulations using immersive technologies [92], including the visualisation of an aerodynamic simulation of indoor air [84] and of X-ray radiation in medical scanners [146].In this regard, future research could investigate the adoption of high-performance computing paradigms [106], massive parallelisation [88], inmemory processing [137], among others.

Multimodal human-computer interaction (HCI)
A modality of interaction refers to a channel of sensory input or output between a computer and a human.For example, a computer interacts with a human using the vision modality by displaying information on a display.Multi-modal HCI refers to systems that enable many modalities to be used as inputs or outputs, including vision, audition, haptics, gustation, olfaction, thermoception, nociception, and equilibrioception.However, the variety of modalities used is still limited; as the interaction focuses primarily on vision, audition, and increasingly on haptics.The rest of the modalities are under-exploited in HCI research [31].Multimodal HCI presents a huge potential for more intuitive AR and VR systems, and it is a new area of intense research [25].
The potential of haptics and multisensory feedback to improve AR and VR systems has been investigated for many engineering applications such as machinery teleoperation [14,170].However, for architecture and construction, few research efforts have been reported in literature.For instance, Hasanzadeh et al. [73] investigated how passive haptics could be used in a virtual reality CAVE system to simulate virtual fall hazards.The system monitored users risk-taking behaviours when performing a roof tiling virtual task.
Multi-sensory user interfaces for AR and VR systems have been investigated from some time now, e.g.[10,34].For example, Ranasinghe et al. [141] presented a VR system that simulates different weather conditions by adding thermal and wind stimuli using fans, air-pumps and heat-pumps attached to the HMD.Narumi et al. [120] presented an AR system that can change the perceived taste of food by changing its appearance and scent.An AR HMD display with two cameras overlays images of different types of cookies on a wafer.An air-pump, mounted on the HMD, sprays a scent in front of the user's nose to simulate different flavours.The authors report that around 80% of people that tested the system noticed changes in flavours, but the intensity was modest.Sardo et al. [153] presented an AR tablet system that provides touch, taste, and smell stimulus to the user to enrich the experience of visiting a museum exhibition.Fans, vibration motors, vaporisers, and heath-pumps are used to deliver a wide variety of stimuli.The authors note that there are still many improvements to be made to the interfaces and that size and weight restrictions remain a big challenge to include multisensory capabilities to AR systems.
Additionally, brain-computer interfaces (BCI) enable to interact with computers based on brain activity using non-invasive electroencephalogram (EEG) systems and intracortical systems such as electrocorticography (ECoG) [70].BCI has been gaining interest among the AR [162] and VR [1] community since BCI has the potential of providing more intuitive and implicit interactions [100].For example, the integration of AR, VR and BCI has been investigated to improve 3D object control [100], enhance human-robot interactions [9], smart home control [140], and supporting inspection of industrial facilities [8].
Regarding AEC applications, research on multimodal AR and VR systems should focus on three aspects (i) integrating existing developments on vision, audition, haptics and BCI to provide more intuitive interactions, for example by making the interaction with virtual 3D objects and environments more natural.(ii) Explore what advantages can be gained by including other modalities such as gustation and olfaction, and (iii) investigate the effects of multimodal HCI on users to avoid cognitive overload and ensure safety.Note that, one of the biggest obstacles for multimodal HCI is the lack of understanding of personal multisensory experiences [128].Also, it is very important to determine the contribution of the different senses, along with their interactions in order to design more effective and engaging digital multisensory experiences [31].

AR and VR teleoperation and plant control
Teleoperation of equipment offers clear safety advantages compared to physical operation in-situ in hazardous and harmful environments, for example, operating machinery in mining operations [15].VR interfaces for controlling equipment and robotic arms in manufacturing facilities have been investigated as well [98].The future of the construction industry will require a combination of people and automated robotic systems working together and interacting in both the physical and virtual work enhancing each other's capabilities [45].In addition, operating construction machinery is a complex task that requires skilled and experienced workers.Controlling equipment can be compromised by lack of training, unexpected situations, sudden changes in weather conditions, among others.AR can be useful to support workers on the operation of plant equipment by providing operation instructions, adjustment procedures and emergency actions.These capabilities could be used advantageously by facilitating the operation of complex plants, with less risk, by operators who are not necessarily familiar with the specifics of a given plant.For example, it has been investigated how AR can facilitate the use of sophisticated medical equipment [143], for controlling nuclear fusion equipment [57], and IoT devices [166].Further research in this area should focus on improving user interfaces for teleoperation making use of multimodal HCI and BCI.

Discussion
In this section, the implications for practice and for theory that this study represents are addressed, as well as the limitations of this study and further work.

Implications for practice
The major implication for practice of this study is that it provides valuable information to practitioners to inform on AR and VR adoption decisions.This information can be used to devise strategic adoption plans and to gain a first-mover competitive advantage.More specifically, this is study provides practitioners with clear AR and VR usecases in the construction industry.It provides examples and explains the benefits and remaining issues.This information is essential for practitioners to inform what use-case is best suited given their own circumstances and provides realistic expectations.This study also provides a description of the adoption landscape in industry, for AR/VR overall and per use-case.Practitioners can use this information to compare their own adoption level with the overall adoption levels presented here.This information gives an indication of which direction the industry is moving.Practitioners can then decide whether to follow the trend or focus on other use-cases.For technology developing companies, the presented research agenda provides an overview of the gaps in capabilities which can inform their future development efforts.For construction companies, it presents an overview of the existing technology capabilities, which can inform their AR and VR adoption plans.In this sense, it is important for practitioners to examine innovation diffusion factors that may influence adoption such as the relative advantage of adoption (advantages vs disadvantages of the technology), compatibility with the currently used workflows, the complexity of the use-case, how easy is to test the technology, and to which extent results can be measured [148].

Implications for theory
This is a foundational study that formalises and categorises the use of AR and VR in the construction industry.This formalisation will help to sort and map future research efforts.This study categorises and maps the research efforts required for the robust adoption of AR and VR in construction.This study can be used as a roadmap for other studies that seek to advance AR and VR in construction.For example, the presented research agenda will help other researchers in the field to map their own research efforts and explore the connections with other research topics.For researchers that want to venture into these topics, it provides an overview of the research gaps that can inform their future research efforts.

Limitations
The main limitation of this study is the limited sample of professionals and academics consulted for this study.This limitation was mitigated by using a very targeted sampling method to ensure that (i) participants came from various parts across the very heterogeneous construction sector, and (ii) participants were experts in the discussed topics.Regarding adoption levels, the principal limitation is that only academics and professionals based in the UK participated in this study.However, most of the companies that participated in this study are transnational and have operations all over the world.Thus, the findings and insights presented here are potentially relevant for AEC practitioners across the developed world; because, in the current globalised world, construction industries in the developed world have many similarities.For example, construction labour productivity and labour productivity growth are very similar in most European countries, the US, Australia and Israel [12].

Further work
Overarching and thorough literature reviews of research and implementations of AR and VR in construction for all the use-cases presented here are required.Existing research efforts in this area only cover limited scopes, e.g.construction safety [93,112], training [178], and construction support [3].Regarding levels of adoption, investigating the state of adoption in other regions of the world for comparison purposes is necessary.These additional studies will help to find potential discrepancies and the contributing factors responsible for these discrepancies.Regarding the proposed research agenda, a crossdisciplinary investigation of research carried out in other fields that could be relevant to the agenda presented here is necessary.AR and VR are poised to revolutionise many industries.Valuable research efforts are being made in other fields ranging a wide spectrum from aerospace and advanced manufacturing to medical, psychology, and pedagogy.These efforts should be mapped and taken into consideration to enrich future AR and VR research efforts.Also, research is required that identifies the requirements and work needed to make AR and VR more accessible to the AEC sectors.A way forward will be to use the three main categories of the research agenda presented here as a basis to define requirements and develop a roadmap for adoption.Lastly, the successful adoption of AR and VR technologies in the construction industry will require a major upskilling of construction workers.Research that informs and guides this effort is imperative.Necessary outputs include a detailed upskilling road map, a list of skills required for workers, a mapping between skills and construction tasks, etc.

Conclusions
This paper presented a study on the current usage landscape of AR and VR in the construction industry.The research leveraged different methods to collect data, including qualitative and quantitative methods, and a literature review to come up with findings that are relevant for both academy and industry.The primary outcomes, as set by the study's objectives, are: (i) a definition of the use-cases in which AR and VR can be used, (ii) a general indication of adoption levels in the industry and per use-case, and (iii) a set of research topics necessary to meet the requirements for a successful AR and VR implementation.
This study shows that AR and VR can be used in various ways throughout the entire life cycle of a built asset; which have been categorised in six use-cases: stakeholder engagement, design support, design review, construction support, operations and maintenance support, and training.The overall adoption in industry remains low.VR has been adopted to a larger extent than AR; while stakeholder engagement is the most adopted use-case.Despite the low levels of adoption, there is an indication that construction companies have a high interest in investing in AR and VR technologies.
This study also demonstrates that AR and VR are not ready to be fully adopted in the construction industry and that research and development gaps remain.This study presents an overarching research agenda that can inform practitioners on the best way to prepare for adoption or extend and maximise their implementation efforts.Also, it presents a robust foundational study that sets the direction for the research necessary to enable the successful adoption of AR and VR in the research industry.

Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Fig. 1 .
Fig. 1.Diagram of the research method used in this study and the three research outputs.
general use-cases have been characterised.A brief description of the use-cases enriched with examples from literature are presented below, i.e.: (1) Stakeholder engagement, (2) Design support, (3) Design review, (4) Construction support, which has four sub-categories construction planning, progress monitoring, construction safety, and operative support; (5) Operations and management, and (6) Training.Fig. 2 presents visual representations of the six use-cases, including the level of adoption of AR and VR per use-case (see Section 4.2 for more information on adoption levels).The intention of the image is to provide visual support to the reader to facilitate the understanding of each use-case.

Fig. 2 .
Fig. 2. AR and VR cases in the AEC sectors and their estimated levels of adoption (see Section 4.2).The plots indicate the level of adoption in projects for the given use-case (1 = not used, 2 = early testing, 3 = basic implementation, 4 = partially used, 5 = fully implemented).

Fig. 4
Fig.4presents the VR levels of adoption per use-case.User engagement, design support and design review have the highest level of adoption with a median value of 3; while construction support, operations and management, and training have a median value of 2. Note

Fig. 3 .
Fig. 3. AR and VR general levels of adoption, expertise, implementation levels and future investment.

Fig. 4 .
Fig. 4. Levels of adoption of Virtual Reality for the six use-cases defined in the AEC sector.Note that "fully implemented" only was recorded for design support and training.

Fig. 5 .
Fig. 5. Levels of adoption of Augmented Reality for the six use-cases defined in the AEC sector.Note that "fully implemented" only was recorded for stakeholder engagement and training.

Fig. 6 .
Fig. 6.AR and VR future research agenda mapped along with technology readiness and time axes.

Table 1
AR and VR benefits and challenges per use-case, including examples and levels of adoption.