Usability assessment of compaction operator support systems using virtual prototyping

The successful adoption of Hot Mix Asphalt (HMA) compaction Operator Support Systems (OSSs), which use sensory information to help operators improve the safety and productivity of their operations


Introduction
Pieces of construction equipment are an indispensable part of modern construction projects and have a significant impact on the safety, cost, and duration of projects [1]. This is mainly because with the growing complexity of the construction project, the operators of construction equipment need to process an overwhelming amount of information in a short span of time to keep projects on track and safe. That is why in recent years manufactures and researchers have developed several real-time Operator Support Systems (OSS) with the aim to provide operators with safety and productivity-related guidance. These systems collect and process a large volume of data (e.g., 3D model of the site, project schedule, locations of other equipment, conditions of the project, etc.) in real-time to identify an efficient, i.e., safe and productive, way in which operators can execute their tasks [2][3][4][5]. While the development and utilization of OSSs have been long studied and investigated, little attention is paid to the usability aspect of these systems. OSSs need to convey a large volume of information to operators who are supposed to process this information in a fragment of a second to devise an operational strategy. Without a proper design of the way in which this information is presented to the operators, there is a major risk of providing operators with information that can be incomprehensible, distracting, misleading, or overwhelming [6,7]. Information overload or infobesity in OSSs can significantly overshadow the effectiveness and applicability of these systems [8].
The above-mentioned problem is more visible in the domain of paving operations where the limited time window for the proper compaction of the hot asphalt, highly collaborative and equipmentdriven work environment, and complex construction procedure make the usability aspect of the OSS highly critical for the successful adoption of the technology. Among different pieces of equipment involved in paving operations (i.e., roller, paver, and trucks), the operation of the roller is less structured compared to trucks and pavers. In other words, the operators of rollers have much larger maneuvering room and strategy alternatives for performing their tasks. This maneuvering space is a major contributor to the paving process variability. Therefore, an efficient compaction OSS is expected to have a major impact on improving the efficiency of the overall paving operations. That is why this research focuses on the compaction OSS.
In the domain of compaction OSS, as shown in the previous work of the authors [9,11], the de facto standard for OSSs is to provide highly descriptive guidance to the operators. In other words, the existing OSSs merely collect real-time temperature and compaction count and present the results to the operators, as shown in Fig. 1 (temperature and compaction contour plots) [11]. This mode of data representation is likely to subject operators to cognitive overload. This is because, on top of paying regular attention to equipment conditions, other pieces of equipment, and safety, operators are expected to analyze temperature and compaction contour plots on the fly to develop operational strategies. Based on the history of OSS development for other pieces of equipment, e.g., excavators, it can be envisioned that compaction OSSs need to transition to a more prescriptive mode of guidance where the operators are either provided with more processed information in form of a compaction priority index, which combines the element of temperature and compaction, or a prescribed compaction trajectory [9,11]. However, there is an ongoing debate about whether such a prescriptive approach towards OSSs can be successful. The main argument against this approach is that the use of prescriptive guidance can give operators the impression that they are losing control of the operation and that their trade is becoming de-professionalized. This can be a serious adoption barrier that may hamper the use of prescriptive systems. Therefore, it seems essential that before the development of compaction OSS is pushed any further, a thorough usability analysis of various alternatives is carried out.
Conventionally, the usability testing of OSSs is done through physical prototyping and testing on sites [4,10,11]. However, this approach has a number of inherent risks: (1) If the prototype has minor stability or hardware/software glitches, which happens more often than not in rapid prototyping, it would majorly deflect the attention of the end-users from the interface design and usability. As a result, the system needs to be developed to a high level of maturity and stability before proper usability testing can be carried out. At this level of development, the cost of modifying the system architecture based on the feedback from the endusers is high and it is difficult to modify and adjust the system based on the feedback received from the end-users [12]; (2) The usability testing of OSSs on construction sites poses a liability issue because contractors are seldom willing to compromise the safety and quality of the projects and ask operators to follow the support/guidance provided by systems that are still under development [9,11]; (3) Finally, the conventional method of usability testing of OSSs is hampered by the operators' lack of trust in the system. In many instances, operators are indisposed to follow the support provided by these systems especially if they are against their intuition or experience. This would cast a shadow on the reliability of the usability testing on the actual sites.
To be able to address this issue in the design of compaction OSSs in a more systematic manner, there is a need to engage the end-users during the design and development cycle. Virtual Prototyping (VP) has been long used in the manufacturing industries and product development as a testing platform [13]. In VP, a virtual representation of the final product is built to allow the participatory and user-centered design of complex systems. Through the use of Virtual Reality (VR), VP models allow endusers to experiment with various design alternatives at the early stage of the design and provide feedback to the developers [14]. This process is shown to make the development process shorter and make the product more likely to be accepted by the end-users [12]. In the construction domain, VR has become popular for a wide range of applications such as operator training, design review, and optimization, monitoring of infrastructure assets, etc. [12,13,15,16]. Nevertheless, VR has seldom been used as a means to implement the VP approach in the usability assessment of construction OSSs, especially for compaction equipment.
In this paper, it is hypothesized that VP can be used in the domain of OSS for compaction equipment to enable fast and reliable usability testing of various OSS alternatives. Through the use of a VR simulator, it is expected that more insights can be generated over the desirability of and expectations from the transition from descriptive to prescriptive OSS design. Therefore, the VP-based usability analysis of compaction OSS is expected to contribute to the debate over descriptive versus prescriptive OSSs. Consequently, the aim of this research is to assess the usability of various descriptive and prescriptive compaction OSS alternatives through the use of a VR-based compaction equipment simulator as a VP platform in order to better comprehend the user preferences and adopt strategies for the development of the next generation of compaction OSS.
The rest of the paper is structured as follows. Section 2 provides the Fig. 1. Common representation of temperature and compaction count (paving OSS), (adopted from [10]). literature review, Section 3 describes research methodology, Section 4 introduces the VP platform, Section 5 goes through the data collection, Section 6 presents the results of the data collection, and finally Sections 7 and 8 present the discussion and conclusions.

Compaction operator support systems
Asphalt construction has remained a largely traditional industry. Generally, compaction operations rely heavily on implicit experience and gut feeling of site workers [17]. Nevertheless, previous research indicated that the use of OSSs in the construction industry has gained significant momentum in recent years [11]. There are many OSSs that are designed to support operators of compaction equipment (i.e., rollers and pavers) by providing relevant information and guidance [4,10,11]. However, the majority of these systems are still in the domain of descriptive guidance. With the nascent of autonomous vehicles, this landscape is changing. From the equipment manufacturing side, there is a great interest to transition to higher levels of automation for OSS, where for instance operators are "guided" rather than "informed" or even auto-drive features are included. The previous work of the authors [9,11] provides a discussion of this transition from the technical standpoint and shows that this transition can indeed enhance certain aspects of paving operation, e.g., the effectiveness of the compaction.
Nonetheless, the transition to a higher degree of guidance requires more than technical feasibility [18]. Previous research indicates that a successful implementation of advanced technologies in traditional job sites, such as the construction industry, heavily depends on the acceptance of the technology by the end-users, and therefore the development process requires an active engagement of the end-users [19]. However, the participatory design and development of compaction OSS are very challenging due to safety, liability, and cost issues associated with the experimentation on actual sites [9,11]. On the other hand, the experience of equipment operators remains unknown to the developers of OSSs. In the previous work of authors [20], it was found that VR can help explicate asphalt construction operators' knowledge. It was shown that VR technology can be used to allow planners to experiment with different operational strategies. However, despite the fact that VR technologies are shown to be effective for the assessment of OSSs of other types of equipment, e.g., cranes [21,22], they have never been used for the design and assessment of compaction OSSs. To the best of the authors' knowledge, most previous research in the domain of OSSs and Intelligent Compaction (IC) systems resorted to physical prototyping for the validation and feasibility study [4,10,11]. This has multiple reasons: (1) the development has been always approached from the technical feasibility perspective and not from the usability and user acceptance perspective; (2) different alternatives towards the provision of guidance to operators have never been the subject of a thorough study; and (3) a high-quality VR simulator of compaction operation is not available. This gap calls for more attention to the facilitation of participatory design of compaction OSSs. In order to be able to apply a participatory design approach for OSS development, it is important to investigate the usability criteria that are pertinent to these systems. These criteria can be used to frame the assessment of OSS by the endusers during the design phase.
Another framework used for the testing of systems' usability is the System Usability Scale (SUS) [47]. This framework uses the subjective experience of end-users to assess the usability of industrial systems through a set of standard questions. SUS is widely used for the assessment of a wide range of IT systems and websites in different domains [48][49][50][51][52]. SUS has also been used for the usability assessment of invehicle systems in the automotive industry [53]. Also, Lorenz et al. [54] applied SUS successfully to assess the usability of excavator OSS.
Although a combination of SUS and NASA TLX frameworks is used to assess various aspects of usability of construction-related systems, to the best of the authors' knowledge, it is not used for the assessment of compaction OSSs. As shown by Lorenz et al. [54] and Li et al. [53], this combination is able to provide a good indication of user acceptance of operator/driver support systems. In this research, this combination is considered and slightly adapted, i.e., to match the specific aspects of compaction OSS, to assess the usability of various descriptive and prescriptive alternatives for compaction OSS.

Virtual prototyping (VP)
VP is used extensively in manufacturing to evaluate the design fit, performance, and manufacturability [13]. It is shown that VP is a powerful approach to support the development of complex systems that can be used for various purposes and in different stages of development [55,56]. VP has been used in aerospace [57][58][59], automotive [60][61][62][63][64], medical [65], railroad [66], and maritime [67] industries. Karkee et al. [68] enumerated the following advantages of using VP in the system development lifecycle of complex systems: (1) the use of VP reduces the number of development and test cycles, (2) VP helps in testing procedures for the front-and back-end of the designed solution, (3) VP reduces the risks that might occur during implementation on real sites, (4) VP helps to build proper communication between designers and users.
Li et al. [69] demonstrated that VP can also be used in the construction industry to support the project team during the early stage planning and scheduling. Fang et al. [70] used VP for lift planning and to help crane operators better plan their lifts and quickly identify hazards using a VR simulator. VP is also used in the development process of excavators [71] and also for the assessment of carbon emission on construction sites [72]. Nonetheless, to the best of the authors' knowledge, the VP approach has been barely used for the usability assessment of construction OSS, as part of the design cycle. In a closely related study, Meusel et al. [73] implemented a VP approach for the assessment of different alternatives for the OSS used in combines. While the motivation and approach in this research are similar to that of this paper, the striking contextual differences between agriculture and construction industries necessitates dedicated research on the usability of compaction OSSs.
One of the greatest challenges in the use of VP for the usability assessment is the achievement of the physics and graphics fidelity that is required for the end-user to have a realistic experience [74][75][76]. The absence of this fidelity can significantly detract from the effectiveness of virtual prototyping and can even be counter-productive. This is because immersion in an unrealistic environment distracts the users and diverts their attention from the design-related aspects. Therefore, the success of the VP approach to a great extent depends on the VR simulator used for the usability testing. That is why it is important to review the current state-of-the-art of VR simulators in construction.

Virtual reality in construction
VR is shown to have a significant impact on education in all domains because of its enhanced visualization, interactivity, immersiveness, scalability, customizability, and real-time feedback [77]. VR contribution to active learning is well documented in the literature [78]. Because of these characteristics, VR-based learning has been implemented successfully both in academia and industry [79,80]. The construction industry also uses VR technologies to improve different practices [81]. The VR implementation in the construction industry mainly focuses on (a) acquiring basic skills needed for operating different construction equipment and machinery, e.g. excavator [82], crane [83], wheel loader [84], etc., (b) simulation of various construction sites and operational situations to gain insights into different working strategies [85]; (c) building safety awareness for current and future site workers [86], and equipment operators [87], and (d) rapid improvements, design and development of hi-tech construction solutions (e.g. support systems) [20,68].
Recent research on project-based educational games suggested that the VR environment can be a breeding ground for the development of required competencies in situations where on-site training poses safety risks [88]. Another major benefit of using VR for construction training is that the training can be offered at any time in a controlled and safe environment [89]. The learning process becomes more proactive and multiple decision-making procedures can be trained in dynamically changing conditions of the virtual site [90]. Additionally, VR training allows instructors to monitor the performance and progress of trainees at much higher levels of detail, owing to its ability to gather and store a wide range of information about the trainees' performance [91]. It is shown that construction VR training can contribute to the accelerated learning curve [92].
Nevertheless, while extensive research has been conducted on VR training simulators for excavators [82], loaders [84], and cranes [83,92], to the best of the author's knowledge, such development has seldom been considered for asphalt paving and compaction equipment. This attests to a clear gap with respect to VR application in the construction sector because the paving operation is one of the most recurrent operations in the construction sector and it requires great tactical and strategic skills. Fig. 2 presents the overall methodology used in this research, which consisted of 3 phases namely, (1) Development of VP platform, (2) Data collection, and (3) Data Analysis. In short, in phase one, several different levels of support for compaction OSS were determined based on the review of the literature and available OSSs for other types of construction equipment. Then, a VR simulator that can represent different support levels of compaction OSS was developed. In phase two, a set of workshops with expert operators were organized in which operators were exposed to the developed training simulator, used various support levels of OSS, and then asked to (a) assess different support levels in terms of usability (SUS) and cognitive overload (NASA TLX), and (b) assess the usefulness of the VR simulators as a VP platform for usability testing of OSSs. Finally, collected data were analyzed to systematically compare different support levels of the compaction OSS and the usefulness of the VR simulator for this assessment. Each one of these phases is elaborately discussed in the following subsection.

Development of a virtual prototyping platform
Given that the main objective of the VP in this study is to assess various levels of support that can be provided to the paver and roller operators, the first step is to determine different support levels that are currently more pertinent to operators considering market availability and technological maturity. Therefore, given the limited market availability of (semi-) autonomous compaction equipment at this time, this level of automation is kept out of the scope of this research.

Different levels of support in compaction OSS
As indicated in the previous work of the authors [11], in order to achieve high-quality pavement, the operators need to ensure that compaction effort is (1) applied within the compaction temperature window, and (2) consistent and compliant with the prescribed number of compaction passes determined by the asphalt mixture design. In other words, roller operators strive to achieve a predefined desired number of roller passes within a certain asphalt mixture temperature range [T min ~ T max ]. Considering these two parameters, and in accordance with generic levels of autonomy proposed by Anderson [93], three levels of support can be envisioned for compaction OSSs [94] as shown in Fig. 3. Different levels of support result in different types of information that can be communicated to users through an information display.
The first level of support, which is called the assistance level, presents the current temperatures of the asphalt mixture and the performed compaction over the asphalt layer in two separate but side-by-side plots, as shown in Fig. 3(a). For both plots, a color-coding scheme is used to indicate the temperature and achieved compaction pass of the different parts of the mat with respect to the corresponding predefined upper and lower thresholds. During the task, the VR user is expected to closely observe the display to monitor the cooling process of the asphalt layer and areas of the mat that are compacted properly or require more roller passes. This level of support, and the data presentation method thereof, is in keeping with the majority of available intelligent compaction systems [4,10,11]. Since this level of support only describes the conditions of the asphalt layer, it can be considered fully descriptive.
The second level of assistance is referred to as the semi-guidance level. At this level, the consideration for the compaction temperature window and compaction effort (i.e., pass count) are combined into a time-variant index, i.e., in the range of 0 to 1, that represents the compaction priority of different part of the asphalt mat, as shown in Fig. 3(b). The details of how this index is calculated are provided in the previous work of the authors [9]. In a nutshell, this index considers the cooling rate of the asphalt, the current temperature, the time left before the asphalt is too cold for the compaction, the required number of compaction passes, and the number of the achieved compaction passes to highlight the part of the asphalt mat that requires more attention from the machine operator at any given point in time. This index is then translated into a color-coded scheme and presented to the operator on the display. The previous work of the authors showed that this level of support indeed helps operators improve the quality of their compaction operation in terms of enhanced compaction consistency and efficiency, i. e., compaction within the right temperature window. Because this level of support neither describes the conditions of the asphalt layer nor does it prescribe an action, it can be considered as the middle ground between descriptive and prescriptive.
Finally, the last level of support is dedicated to guiding the operator through the compaction operation by indicating a compaction path. Therefore, this level of support is called guidance, as shown in Fig. 3(c). At this level, the speed of compaction equipment (i.e., roller and paver), the cooling rate of the asphalt, the current temperature, the compaction temperature window, the width of the road, and the achieved and target compaction passes are considered to identify the stretch of road that needs to be compacted at any given time. As shown in Fig. 4, the stretch is determined based on controlling the following conditions: (1) the asphalt in the stretch is within the compaction temperature window and remain so until the compaction of the stretch is complete, (2) once the compaction of the stretch is finished, there is at least a stretch of a similar length ahead of a roller that can be compacted by the roller without disrupting the continuity of the compaction. Once the stretch is determined, a trajectory representing a full compaction pass cycle on this stretch can be developed. The detailed mathematical description of this path planning approach is out of the scope of this research and will be presented in the future work of the authors. As a clear compaction path is offered to the operator, this level of support can be considered prescriptive.
As shown above, within the scope of non-autonomous solutions, the above three support levels cover the ground between descriptive and prescriptive OSS. Therefore, by getting exposed to these three levels in the VP platform, operators will be able to get the feel about a range of alternatives and assess their usability.

Development of VR simulator
In order to be able to conduct the VP-based usability assessment, the VR simulator that captures the levels of support discussed in Section 4.1 had to be developed. Fig. 5 shows the instruments used for the development of the VR simulator. In this simulator, the GameSeat Pro+ [95] was used as the main platform of the simulation. A flat panel LED TV (Samsung LED screen, 43 in.) [96] was mounted on the platform as the screen. A Logitech G29 steering wheel and joystick [97] were used as the control elements for user interaction. Finally, a Lenovo P51 laptop (15.6" FHD, 7th Gen Core i7, 16 Gb DDR4) [98] was used as a processing unit that runs the prototype's software. A cross-platform game engine Unity and C# were used as the main programming platforms for the development. Relevant C# software modules were written for different levels of support based on the methods explained in Section 4.1. To be able to realize the generation of the relevant levels of support, the asphalt layer is developed as a rasterized grid, as shown in Fig. 6. The grid structure resembles the common pattern of collecting temperature and compaction count data by intelligent compaction systems [11]. Therefore, it can very well replicate the feel of working on a roller with an OSS. Also, this alignment between the data collection pattern and the representation in the VR environment, allows the use of temperature data from the actual construction site to provide a more realistic cooling behavior of the asphalt. As shown in Fig. 6, the gird structure allows assigning each cell attributes for the current temperature (T i,j ) and the current number of passes (NP i,j ). Using these attributes and the method explained in the previous work of the authors [11], the priority index of each cell can be determined. It should be highlighted that the guidance level, which includes the generation of a compaction trajectory, uses the grid structure for the determination of the compaction stretch at any given time, however, translates this into a trajectory, which is overlaid on the grid structure as a vector.
As shown in Fig. 7(a), this simulator is designed to immerse roller operators into the VR environment where the user needs to perform a   simple compaction operation. The user is placed inside the rollers' cabin and can decide where to place the information display. Different types of support are presented in this display and the user can easily toggle between different levels of support, which are shown in Fig. 3, by using a button on the steering wheel. The final setup of the developed VP is shown in Fig. 7(b). The environment modeled in the simulator includes the following components, as shown in Fig. 8: (1) a paver that lays down fresh HMA with the predefined speed. Here, the assumption is made that there are not any logistic problems with HMA delivery, thus, trucks are not included in the simulation; (b) a roller that is operated by the simulator; (c) a roller that is operated by the simulator's user; (d) barriers that mark the road section and separated the traffic from the paving fleet; and (e) the surrounding car traffic, to make the simulation more realistic. Fig. 9 presents the flowchart of the simulator. At the beginning of a session, the research team sets the desired number of roller passes and asphalt mixture temperature range (compaction window). When the VR scenario starts, the user chooses the desired level of support and the placement of the information display. Based on the user's actions and the predefined cooling algorithm for the asphalt mixture, the corresponding support information is generated. When the task is finished, either because of the time limit of the scene or by user's interruption, the VR simulator provides users with achieved performance scores in terms of cells that are (1) over-compacted, (2) under-compacted, (3) compacted below the compaction window, (4) compacted above the compaction window, and (5) compacted enough at the sufficient temperature, as shown in Fig. 10.

Data collection
To collect (1) the usability assessment data about different levels of support, and (2) the usefulness of the VP platform for this purpose, five workshop sessions with operators of paving and compaction equipment were held at different locations in the Netherlands. Fig. 11 shows a few examples of these workshop sessions. The participants were classified into three groups of experienced with OSS affinity, experienced without OSS affinity trainees, and beginners. In total, 50 users were exposed to the VR simulator and filled the questionnaires. Of these, 14 participants were categorized as inexperienced, i.e., still in the training and no practical experience. 20 participants were professional operators with practical experience who have never worked with any kind of OSSs before. The remaining 16 participants were professional operators who had prior exposure/experience to at least one kind of compaction OSS. Participants were given the chance to use the simulator and get exposed to all three levels of support. Each participant spent around 5 min on the simulator and during this time could easily toggle between different levels of support as they would see fit, as shown in Fig. 12.
At the beginning of each session, a presentation was given to the  participants to familiarize them with the research and VR simulator. Then, in an open session, all the patricians were given the chance to observe how the simulator works and even try it on their own, in the hope that this can reduce the impact of unfamiliarity with the VR technology on the usability assessment. After that, the participants were asked to perform the compaction of a stretch of newly paved asphalt layer in a tandem with the other roller. The participants were given the instructions to perform two passes on the asphalt layer within the temperature window of [80-120 • C]. At the end of a session, each participant was given two different questionnaires, i.e., one for the usability of different levels of support and one for the usefulness of VP. Table 1 presents the set of usability questions. For each question in this questionnaire, participants needed to give separate scores for each level of support. This would allow having a side-by-side comparison of different levels of support. The sets of standard SUS and NASA TLX questions needed to be adapted because some of the questions would have little relevance in the context of this study, e.g., a question about the physical demand of the system. Other questions were also slightly paraphrased to better reflect the context of the VR simulator. As shown in this table, other than the adopted SUS and NASA TLX questions, a set of customized questions were used to further assess the perceived value of different levels of support. To analyze the results, each question is given a keyword that represents the main category of the question, as shown in Table 1.
Regarding the customized questions, a few clarifications need to be made. Since the VR simulator presents the operation of another roller, which is driven by the VR, and since the provided support (in terms of compaction plot, priority map, and guidance) takes into account the operations of both rollers, the users are expected to assess to what extent the provided support (a) helped them clearly understand what the other roller is doing (i.e., Q13) and (b) adopt a strategy in congruence with the operation of the other roller (i.e., Q14). Also, in the pre-experiment briefing session, it was explained to the users that earlier research suggests that there is a strong correlation between asphalt quality and the consistency and homogeneity of compaction operation [17]. Therefore, the operators were asked to take notice of their performance indicators (as shown in Fig. 10) and associate this with the final quality of the asphalt (i.e., Q15). Table 2 presents the set of questions that were used to assess the usefulness of the VP platform for the analysis of the usability of compaction OSS. This set of questions addresses various aspects of the developed VP such as its realism, contribution to skill acquisition, etc.

Usability assessment
The data collected from the workshops were processed and analyzed. Tables 3 and Fig. 13 present the result of the usability assessment of different levels of support. As shown in Table 3, Semi-guidance level of OSS received the highest overall SUS/NASA TLX score, i.e., 71.97/100. Nevertheless, Assistance level of OSS scored very close to semi-guidance in almost all categories, with an overall score of 67.43/100. In comparison, Guidance level of OSS is the least appreciated system and received the lowest score in almost all categories. When it comes to a customized set of questions, Assistance level scores the highest (66.72/ 100) but only marginally compared to semi-guidance (66.68/100). To examine whether the observed variations in the scores have statistical significance, Kruskal-Wallis test [99] was performed between three levels of support, as shown in Table 3. As shown in this table, all variations have statistical significance. To further shed light on this, a pairwise comparison was also carried out, as shown in Table 4. Here, it is evident that the difference in the scores of customized questions between Assistance and Semi-guidance levels is not statistically significant. This means that, from the statistical standpoint, the mean scores of these two types of support levels for the customized questions can be considered equal at the confidence level of 95%.
As shown in Table 3, the top three most appreciated categories of the present support levels, in order, are comprehensibility, nondistractiveness, and clarity. The semi-guidance level of support scored the highest in all these categories. The further analysis of the magnitude of variations between the average scores of different categories of different support levels reveals that the users see the greatest difference, in order of significance, non-distractiveness, ease of use, and explicativeness (i.e., reveals aspects of the operation otherwise difficult to see). These can be interpreted as categories that make the most difference between various OSSs. When the two rankings are considered concurrently, i.e., to identify the top categories that also account for the greatest difference between levels of support, non-distractiveness, ease of use, and assistiveness stand out, in order of significance.
To further investigate whether the observed variations between the scores of different categories have statistical significance, Kruskal-Wallis test was performed at the category level. As shown in Table 3, this analysis reveals that the top 7 categories based on the variation have statistical significance at P < .05. In general, it can be discerned that the top 3 categories (considering the combined effect of score and variation) have statistical significance at P < .05. This can be interpreted as that the three categories of non-distractiveness, ease of use, and assistiveness of the compaction operator support system are the main governing categories that (1) define the differences between the three levels of support, and (2) are highly appreciated in the semi-guidance compaction OSS.
The collected usability data were further analyzed to compare the scores of different participant groups (i.e., inexperienced, professionals without OSS affinity, and professionals with OSS affinity). The result of this analysis can be found in Table 5 and Fig. 14. The analysis revealed that for almost all categories inexperienced participants have scored the OSSs higher than professionals. This is better reflected in the comparison of the overall scores shown in Fig. 14. Between the two groups of professionals, those with prior experience gave higher scores to all systems (except for the SUS & NASA TLX aspect of the Semi-guidance level). Among different systems, professionals with prior experience with OSSs showed more appreciation for the Assistance level of OSS. In cases of inexperienced and professionals without prior OSS affinity, semiguidance level scores higher than the other two variants. At the level of overall scores, all variations in the scores are shown to be statistically significant. At the level of categories, while the majority of scores for SUS and NASA TLX categories have statistically significant variation, the same cannot be said about the customized categories. Pairwise comparison of different participant groups reveals that in the majority of cases (i.e., 14 out 18) the observed difference in scores of different groups is statistically significant. (See Table 6.) Fig. 15 present the result of the second questionnaire, which is about the usefulness of the VP platform. As the results indicate, the respondents thought that more realistic control and audio would considerably enhance the VP platform. This, together with the low score on the realism of the VP (2.84) can be construed as the need for further improvement of the VP platform from audio-graphical and control perspectives. In the open comment section, several respondents highlighted that the 3D model used for the roller belonged to a type of roller that is not familiar to them and this has distracted their attention. A strong suggestion was given to use a more realistic representation of the roller. Having said that, nearly half of the participants, i.e., 21/50 participants, scored VP adequacy 4 or 5 which indicates that they thought the VP platform was adequate for comparing various levels of support and provided them with a good evaluation basis. Combining this finding with the suggestion for improved graphics indicates that the VP platform with a more graphically realistic VR model can be very well used for future studies similar to this research.

Usefulness assessment
In addition, the participants appreciated the most the additional value of the VP platform as a tool that would allow them to investigate  their compaction strategies before actual operations, which was scored 3.5. Also, in line with the finding from the previous questionnaire, the ease of use of the VP platform, mainly from the hardware perspective, was acknowledged with a score of 3.46. A majority of the respondents mentioned that they would recommend the use of the VP platform for the evaluation of other technological advancements in the domain of compaction operations. Finally, it seems that the educational value of the VP platform in the current shape is controversial, looking at the scores for formative (2.88) and educational (2.9) values.

Discussion
Based on the results of the workshops presented in the previous section, a number of observations can be made. First, according to the results, it seems that the main characteristics that explain the difference between various levels of support are non-distractiveness, ease of use, assistiveness, comprehensibility, and non-overloadingness of the system. On the contrary, the lowest scored characteristics are shown to be the 3D visualization, effectiveness, informativeness, reusability, instructiveness. This observation can be used as a guideline on the system characteristics that need to be further taken into account in the design of compaction OSSs.
Second, the results clearly indicate that based on the aggregated opinion of the participants, the semi-guidance support system is preferred over other levels of support. On the basis of informal discussions with the participants, it can be posited that this type of support is seen to combine elements of descriptive (i.e., raw data) with prescriptive (i.e., processed data) support. Also, while it remains easy to use and not distracting, it does not take away the feeling of being in control of the operation. This can be a strong incentive for the operators to use semiguidance support systems. From the liability perspective, also, this is an important fact because a major reservation about the guidance system is that while prescriptive systems do not offer a justification about the recommended compaction strategy, and thus demanding "blind" compliance, the operators remain accountable for the quality of the  work they perform. This, to the eyes of operators, creates a strong barrier against the adoption of prescriptive systems. This indicates that the adoption of more advanced forms of assistance requires significant management support and calls for a major overhaul of the contractual and legal framework of pavement projects to ensure that accountability about the quality is commensurate with the support level of OSSs. However, before this state and industry-wide initiative can be mobilized, the usefulness of guidance systems, in form of their contributions to enhanced compaction quality, needs to be demonstrated in more case studies. Given that this could create a vicious circle of there is no trust without (examples of) successful implementation and there is no successful implementation without having operators trusts. It is important to consider the use of VP platforms, such as the one used in this study, to establish a safe and realistic environment where technological befits can be explored and demonstrated without consequences.
Third, it is evident from the results that prior experience with OSS and level of expertise have a significant impact on how these systems are perceived. As shown in Section 6, inexperienced operators, who happen to be from the younger generations and therefore more technologysympathetic, tend to be more optimistic about the use of different levels of support systems in their work. Combining this observation with the fact that operators with prior OSS experience showed a more positive attitude towards different types of compaction OSS suggests that familiarity with technology, and its contribution to enhanced work, can play a role in how the technology is being assessed and perceived. This is very much in line with the findings in the domain of autonomous vehicles that affinity with the usefulness of the technology contributes significantly to the successful adoption of the technology [100,101]. This can be construed as the need for offering periodic technology training sessions to both novice and experienced operators. Again in this regard, the VP platform developed and used in this research can be very instrumental since it can offer an environment where new technological ideas can be (1) easily demonstrated and (2) interactively experienced by the end-users.

Conclusions
This paper utilized a VP approach to assess the usability of various types of compaction OSSs. Three different levels of support that range from fully descriptive to fully prescriptive have been considered, tested, and modeled in a custom-made VR simulator. A series of workshops were held with 50 participants who had different backgrounds (i.e., novice learners, professionals without prior OSS experience, and professionals with prior OSS experience). A series of questions were used to assess the usability of different support levels of compaction OSSs and also to assess the usefulness of the VP platform for this investigation.
In the light of the results of this study, the following main conclusions can be made: (1) It is found that a semi-guidance compaction support system is more appreciated by all operators. Nevertheless, the authors would like to interpret this observation with caution. It is because, as discussed in Section 6, the low appreciation of more prescriptive support systems is likely affected by trust and perceived usefulness issues. As suggested by the results, more support of the OSS systems comes with the increased affinity and exposure to the technology. It can be cautiously argued that a combination of management  There are a number of limitations in this research that can be addressed in the future work of the authors: (1) in the current research, the objective impact of following different types of support on the process quality has not been taken into account. A comparison between how different levels of support can actually contribute to improving the Table 3 Results of aggregated usability assessment (the bold represents the highest score in each category, green shading suggests the category passed the significance test and the red shading suggest it failed it) .    process quality (i.e., in terms of compaction efficiency) and how they are being perceived and received by the operators (i.e., the usability aspect) can create an interesting insight into desirability vs. effectivity of compaction support system. The authors are already busy investigating this issue and will report the findings in the near future; (2) in this research, the VP platform was not assessed from the perspective of the user experience of the VR environment itself. Therefore, the VR UX questions were not used in this research. This is very important because the quality of the VR environment can have a significant impact on how realistically the user can experience the subject of the VP-based study. Going forward, this aspect can be taken into account and the contribution of the VR technology on conveying a realistic feel about the operation will be investigated; (3) Finally, in this research the focus was only placed on the comparison of three different levels of support from the usability perspective. As such, the optimization of the interfaces used for the presentation of each level of support (i.e., on the virtual display inside the cabin) was not taken into account. In the future, the authors intend to pay more attention to the interface design of each support level and repeat the same process to investigate the extent to which the enhancement of the interface can improve the system usability.