TakeoverQuality: Assessing the Effects of TimeBudget andTraffic Density with the Help of a Trajectory-Planning Method

In highly automated driving, the driver can engage in a nondriving task but sometimes has to take over control. We argue that current takeover quality measures, such as the maximum longitudinal acceleration, are insufficient because they ignore the criticality of the scenario. ,is paper proposes a novel method of quantifying how well the driver executed an automation-to-manual takeover by comparing human behaviour to optimised behaviour as computed using a trajectory planner. A human-in-the-loop study was carried out in a high-fidelity 6-DOF driving simulator with 25 participants. ,e takeover required a lane change to avoid roadworks on the ego-lane while taking other traffic into consideration. Each participant encountered six different takeover scenarios, with a different time budget (5 s, 7 s, or 20 s) and traffic density level (low or medium). Results showed that drivers exhibited a considerably higher longitudinal and lateral acceleration than the optimised behaviour, especially in the short time budget scenarios. In scenarios of medium traffic density, the trajectory planner showed a moderate deceleration to let a vehicle in the left lane pass; many participants, on the other hand, did not decelerate before making a lane change, resulting in a dangerous emergency brake of the leftlane vehicle. In conclusion, our results illustrate the value of assessing human takeover behaviour relative to optimised behaviour. Using the trajectory planner, we showed that human drivers are unable to behave optimally in urgent scenarios and that, in some conditions, a medium deceleration, as opposed to a maximal or minimal deceleration, is optimal.


Introduction
Over the past decades, improvements in sensor technology, artificial intelligence, and control systems have led to an increase in vehicle automation. Adaptive cruise control (ACC), a technology that was introduced in the 1990s, has the potential to increase driver comfort by automating the longitudinal control task [1][2][3][4]. e recent arrival of assistance systems that can perform lateral control has led to a situation where the driver is provided with the option to detach him-or herself from the control loop of the vehicle [5,6]. However, in current and near-future levels of automation (SAE levels 2/3), the human still has the role of a fallback agent because automated driving systems have limitations regarding their operational design domain and the handling of unforeseen road situations [7,8].
It has been shown that being out of the loop can lead to a degradation of situation awareness, mental underload, complacency, and mental overload if the automation reaches a system boundary and the operator has to take back control (e.g., [9]). e fact that the drivers of automated vehicles are susceptible to such detrimental effects, but have to take over control when needed, has sparked a wealth of research on automation-to-driver takeovers. In particular, an important topic concerns the quantification of how well the human driver takes over from the automated driving system.
A variety of takeover quality measures have been proposed in the literature (see [10]; for a review). Radlmayr et al. [11] used the maximum longitudinal acceleration as a takeover quality measure, whereas Zeeb et al. [12] used the maximum lateral acceleration and centerline deviation. In many other studies, takeover quality is quantified using the minimum time to collision (MTTC) [13][14][15][16].
e interpretation of these measures has been hampered by the fact that different scenarios impose different demands on the driver. Zhang et al. [17] surveyed 129 takeover studies and found that the time budget and the traffic density are among the most frequently used independent variables. Scenarios can differ from nonurgent, without an immediate need to take over on an empty road [18][19][20], to highly urgent, where the time budget to collision is 5 s or less or where multiple vehicles drive in the vicinity [21][22][23]. Urgent scenarios will generally result in "poor" scores on takeover quality measures, such as a low MTTC [24,25] and high accelerations [26]. Gold et al. [27] stated that, with shorter time budgets, driver reactions are "worse in quality" (p. 1938). e problem with this interpretation of Gold et al. is that, in urgent scenarios, high vehicle acceleration may be desirable or even necessary to avoid an accident and thus reflect "good" rather than "poor" driving behaviour. In summary, the type of scenario is expected to have a large influence on takeover quality measures, but the literature tends to ignore this fact in the interpretation of takeover quality. Takeover quality measures may be uninterpretable if not compared to an optimal reference behaviour.
is study aimed at assessing the merits of using a novel scenario-specific approach for defining takeover quality. We assessed the effects of time budget and traffic density on not only "traditional" takeover scores (e.g., maximal accelerations) but also in relation to the scores obtained from a trajectory planner. More specifically, we took scenario criticality into account by comparing the scores on the measures to a scenario-specific optimised behaviour, obtained via a trajectory planner developed at Volkswagen Group Research, Wolfsburg.
A human-in-the-loop driving simulator study was carried out using different driving scenarios. e independent variables of this study were the time budget and traffic density, as these two variables have been shown to strongly affect takeover quality in previous studies using traditional quality measures [11,28,29]. We used 5, 7, and 20 s time budgets. A time budget of 5 s is critical, and a 7 s time budget is the minimum for a safe takeover after being engaged in a secondary task [27,30,31]. e 20 s scenarios were included to investigate human takeover behaviour in a case where there is little urgency and the human is free to decide when and how to act.

Participants.
irty-one Porsche AG employees were recruited to participate in this study. ey were required to have a valid driver's license, as well as normal or correctedto-normal vision. A total of 6 participants had to be excluded due to incomplete data recordings. erefore, the analysis was performed on data from 25 participants, of which 13 were female and 12 were male. ere were 11, 9, 3, and 1 participants in the 21-30, 31-40, 41-50, and 51-60 age categories, respectively. Two participants indicated having participated in a simulator study regarding automated driving before. e research was approved by the Human Subjects Research Ethics Committee of the TU Delft, and all participants provided written informed consent.

Apparatus.
A hexapod driving simulator at the Porsche Research and Development Facility in Weissach, Germany, was used [32][33][34]. e hexapod was fitted with a fully functional mock-up of a Porsche Macan. e vehicle dynamics software was based on an all-electric Porsche Taycan. e 6-DOF moving base platform (eMove eM6-640-1800) has an actuator stroke of 640 mm. e motion cueing was according to a classical washout algorithm. e visualised field of view was 180 deg, achieved by projectors displaying 3840 × 2160 pixels on all three sides, as well as the ceiling. e side mirrors were not physically present but integrated into the simulation. e visualization was refreshed at a frequency of 60 Hz. e simulator during one of the takeover scenarios is shown in Figure 1. During the experiment, participants wore Dikablis eye-tracking glasses [35].

Road Environment.
e automated vehicle drove at 130 km/hr on a simulated two-lane motorway. e width of all vehicles, including the ego-vehicle, was 1.78 m. e lane widths were 3.88 m. During the approximately 30 min drive, six takeover scenarios occurred. ese takeovers were triggered by roadworks on the right lane. e roadworks were 361 m long. All takeovers took place on a straight stretch of road. ere was no hard shoulder on the right.

Human-Machine Interface.
e automation could be activated using a button on the steering wheel. Upon activation, a message was shown on the instrument cluster, indicating that the driver was allowed to take the hands off the steering wheel. Activation was confirmed by the grey steering wheel icon (automation available) turning green (automation active) (see Figure 2). e participants performed an audiovisual nondriving task in the form of watching a comedy television series (see also [36]). e video automatically started playing on the 10.9-inch centre-display when the automated driving function was enabled.
In each takeover scenario, the participants received a one-stage auditory signal with a fundamental tone of 300 Hz and peaks at 342.5 Hz, 432.5 Hz, 666.7 Hz, and 866.7 Hz as well as a visual warning on the instrument cluster. e steering wheel icon switched to red along with two hands holding it, as shown in Figure 2. Additionally, the centredisplay turned black and stated: "Limited functionality while driving manually." 2.5. Independent Variables. A 3 × 2 within-subject design was used, with time budget (3 levels) and traffic density (2 levels) as independent variables. Six takeover scenarios occurred per participant, comprised of each combination of time budget and traffic density, as shown in Table 1. e traffic density levels were "low" and "medium," as defined in the literature as 5 and 10 vehicles per km, respectively [37]. For the medium traffic density scenarios, a left-lane overtaking vehicle, which was driving at a constant speed of 140 km/hr, drove directly behind the ego-vehicle at the time of the takeover request, making an immediate lane change unsafe. In the low traffic density condition, an immediate lane change was possible and safe due to the absence of nearby vehicles at the time of the takeover request. Figure 3 shows the scenario layout for Scenario 1. Similar figures for Scenarios 2-6 are provided in the Supplementary Materials ( Figures S1-S5). e order in which the 6 scenarios appeared was counterbalanced using a Latin-square method. e automated driving time in between takeovers varied between 3 and 5.5 min, to counteract expectations. e total driving distance was 60.4 km, with a portion of automated driving availability of 50.1 km (23.1 min) per participant.

Dependent Variables.
We computed a variety of takeover measures that have been used in the literature before (e.g., [19,38,39]), as shown in Table 2. Four subjective dependent variables were included as well, based on Radlmayr et al. [10]. After each takeover, participants were asked to rate the scenario, on a 7-point scale rating for criticality (very noncritical-very critical), complexity (very uncomplicated-very complicated), discomfort (very comfortablevery uncomfortable), and subjective time budget (more than enough-way too little). Note that these are translations from German; the entire study was conducted in the German language. e effects of time budget and traffic density were assessed using a two-way repeated-measures analysis of variance (ANOVA).

Procedure.
e participants were scheduled into onehour time slots. After welcoming the participant and thanking him or her for participating, the participant was asked to read an information form. is form explained the capabilities of the highly automated driving system, icons on the dashboard, how to activate the automated driving system, the availability of the nondriving task (television series), the takeover requests, and the four questions that had to be completed on the tablet. e participant was informed that the automation cannot cope with every situation and that their priority should be safety and adherence to the traffic rules. Next, the participant signed a consent form, completed a demographics questionnaire, and was asked to take a seat inside the driving simulator.
While the simulator was still stationary, the experimenter in the passenger seat explained how to activate the automated driving function and how the subjective ratings should be completed on a tablet. Next, the eye-tracking glasses were handed over to the participant and calibrated using DLab behavioural research software (version 3.5). A safety briefing followed. e participant fastened the seatbelt, was informed that opening the car door would stop the dynamic simulator immediately, and was informed that radio communication would always be possible if necessary. A 5 min test drive followed during which the participant got used to the manual controls, automation, display icons, performing the nondriving task, and completing the questionnaires on the tablet. After the test drive, the assistant exited the simulator. Once the moving base platform was reactivated, the main study began.
At the start of the drive, participants engaged the automated driving function while driving in the right lane.  ere was no speed requirement for activation, although it was advised to activate the automated driving function when driving approximately 130 km/hr, as this was the automated driving system's target speed. e automated driving function would accelerate or decelerate to the target speed of 130 km/hr. If the automated driving function was engaged, a video automatically started playing on the centre-display. e participant's gaze was recorded using the eye-tracking cameras. In case the gaze was frequently directed at the road instead of the centre-display, the intercom was used to remind the participant to trust the automated driving system and attend to the video.
Once the automated driving function prompted a takeover request, the driver was required to regain manual control while taking into account any potential left-lane overtaking vehicles. After passing the roadworks, the participant returned to the right lane and reengaged the automated driving system. While the car was driving automatically, the participant completed the subjective rating of the previous takeover scenario. From the simulator control room, it was checked whether the form was received properly. If this was not the case, the participant was asked through the intercom to resubmit the rating form.
Once finished with the rating, the participant continued watching the video displayed on the centre-screen. is order of events was repeated for a total of six times (Table 1). After the sixth takeover scenario, the intercom was used to ask the participant to bring the car to a standstill. e moving base platform was lowered, and the participant exited the simulator. A postdrive questionnaire with regard to simulator fidelity was filled out. In this questionnaire, the participant also had the opportunity to give open feedback regarding the study. e participant was offered something to drink as well as a sweet. Once all potential questions from the participant regarding the study had been answered, the participant was once again thanked.

Trajectory Planner.
A trajectory planner was used to generate reference trajectories for the six scenarios. e look-ahead time of the trajectory planner was changed to correspond with the time budget of each scenario. is created an identical urgency level for the participants in the driving simulator study.
e trajectory planner relied on local motion planning and used a linear reward function that maximised route progress, comfort, and safety. More specifically, the trajectory planner maximised progress to the destination (i.e., the end of the roadworks) and distance to objects, while minimizing acceleration, jerk, wheel angle, and wheel angle changes.   e reward function has an impact on the driving style. e weights used in the reward function were determined by Volkswagen Group Research using a process similar to backtracking, where vehicles were driven manually, and for each time step and corresponding state within the world, the "human" weights were computed. More information about the trajectory planner is provided by Rosbach et al. [41]. e trajectory planner and the human-in-the-loop simulator used the same files describing the simulation environment (openDRIVE extensions .XODR and .XML, built using VIRES Virtual Test Drive (VTD) 2.1 software [42]), including the same simulated vehicles. e trajectory planner assumed a simple kinematic bicycle model. e vehicle parameters of the trajectory planner were those of a 2012-model Volkswagen Golf. Because the optimised reference trajectory was well within the limits of vehicle dynamics, this deviation from the vehicle model used in the driving simulator (Porsche Taycan) was thought to be unimportant for the goals of the present study.

Results
For Scenario 2 (time budget � 5 s and traffic density � low), driving data were missing for two participants due to a datalogging error. Figure 4 shows all 25 participants' trajectories for all six scenarios. Additionally, it shows the optimised trajectory as driven by the trajectory planner.
In Scenario 1 (time budget � 5 s and traffic density � medium), two participants crashed with the roadworks, and three participants crossed the road boundaries. In Scenario 2 (time budget � 5 s and traffic density � low), zero participants crashed with the roadworks, and three participants crossed the road boundaries. No crashes or road departures occurred in the 7 s and 20 s time budget scenarios. Figure 5 provides information about the speed pattern during the takeover manoeuvre. In Scenarios 1 and 3 (time budget � 5 or 7 s and traffic density � medium), the planner decelerated to let the fast vehicle on the left lane pass before initiating the lane change. Most participants did the same, although somewhat later. A small number of participants brought their vehicle to a full stop in Scenarios 1-3. Figure 5 further shows that a small number of participants in the medium-density scenarios sped up, presumably in an attempt to change lanes before the arrival of the fast-driving vehicle in the left lane.

Minimum Time to Collision (MTTC).
e participant's MTTC values for each scenario are shown in Figure 6, along with the values corresponding to the trajectory planner. In Scenarios 1 and 2, the MTTC was often smaller than 1 s, indicating a highly critical interaction (cf. [43][44][45]). In the noncritical scenarios (5 and 6), the MTTC showed large individual differences. e trajectory planner changed lane relatively late, resulting in an MTTC value that was lower than the MTTC value of most of the participants. According to a two-way repeated-measures ANOVA, the effects of traffic density and time budget on MTTC were statistically significant (Table 3). e means, standard deviations, and pairwise comparisons per dependent variable are provided in the Supplementary Materials (Tables S1-S53).

Acceleration Measures.
e results for the four acceleration measures are shown in Figure 7. Participants showed similar levels of maximum leftward and rightward lateral accelerations, a symmetric pattern that is characteristic of a normal lane change. e trajectory planner drove considerably more smoothly in lateral and longitudinal directions as compared to the participants. However, for the 5 s and 7 s time budget scenarios with medium traffic density (Scenarios 1 and 3), the trajectory planner's and participants' decelerations were relatively similar. is similarity can be explained by the fact that the trajectory planner slowed down to let the approaching vehicle in the left lane pass, as did most of the participants. For Scenario 5 (time budget � 20 s and traffic density � medium), the trajectory planner did not decelerate to let the left-lane vehicle pass, as there was enough time to wait until the left-lane vehicle had passed. Some human drivers, on the other hand, did decelerate for the left-lane vehicle, or they accelerated (see Figure 5) and subsequently had to brake for a leading vehicle. e two-way repeated-measures ANOVA (Table 3) showed that the effect of traffic density was statistically significant for decelerations and accelerations. e effect of time budget was most pronounced for lateral accelerations; that is, participants steered more abruptly when the time budget was shorter. Figure 8 shows the lane change duration for the participants and the trajectory planner. In the 5 s and 7 s scenarios, participants generally changed lanes faster than the trajectory planner. For the 20 s scenarios, the average participant lane change durations were similar to the trajectory planner. e effect of traffic density on lane change duration was not significant, but the effect of time budget was significant (Table 3). Figure 9 shows the participants' MTTC versus their maximum deceleration in Scenario 3 (time budget � 7 s and traffic density � medium). It can be seen that participants who showed a low maximum deceleration (which may normally be regarded as high takeover quality) and made an early lane change (indicated by high MTTC values) ended up in a highly dangerous situation, as the overtaking vehicle in the left lane had to initiate an emergency stop. e trajectory planner showed an optimised deceleration of around 4 m/s 2 , just sufficient to let the vehicle in the left lane pass. ese findings indicate that a low maximum deceleration and high MTTC are undesirable in this scenario.

Subjective Ratings.
e questionnaire results are shown in Figure 10. Generally, the experimental manipulations had the desired effect, with the urgent scenarios (time budget � 5 s) with medium traffic density being perceived as most critical, uncomfortable, and complex. e nonurgent scenario (time budget � 20 s) with low traffic density, on the other hand, was regarded as least critical, most comfortable, and least complex. e effects of traffic density and time budget were significant for all four questions (Table 3).

Driver Response Times.
Results showed that humans needed about 0.9 s for attending to the road, about 2.5 s to  attend to the rear-view mirror, and about 2 s to grab/touch the steering wheel ( Figure 11). e trajectory planner, on the other hand, was not susceptible to these delays in human information processing. Table 3 shows that the effects of time budget and traffic density were not significant for the eyeson-road time. is finding is sensible because only after the driver attends to the road, he or she can assess the time budget and traffic density. ere was a significant effect of time budget on eyes-in-side-mirror times and hands-onwheel times; participants took somewhat more time when the time budget was higher.

Discussion
In this study, we assessed human takeover quality for different time budgets and traffic density levels and compared the results to an optimised behaviour obtained via a trajectory planner. e acceleration measures showed that the trajectory planner drove extremely smoothly as compared to the human participants. e participants showed especially large lateral accelerations in the more time-critical scenarios, which may have been because participants were startled by the upcoming roadworks. e trajectory planner "knew" everything from the moment of the takeover request onward, whereas humans needed time to assess the situation, as indicated by the fact that the average eyes-on-road time was about 0.9 s. Another reason for the high accelerations is that participants may not have noticed these high accelerations due to the inherent physical limitations of the motion-base simulator [46,47]. Several participants stated after the experiment that the simulator motion did not feel realistic. In the 20 s time budget scenarios, on the other hand, participants' median lane change durations were close to the lane change durations of the optimised trajectory. In  Journal of Advanced Transportation summary, the results indicate that the differences between a human takeover manoeuvre and that of the trajectory planner are large in time-critical scenarios. at is, humans seem unable to behave optimally when temporal demands are high. A driving simulator study by Petermeijer et al. [19,36] showed that drivers become quicker and more fluent in taking over control with increasing takeover experience. Payre et al. [48] recommended that drivers should be trained in how to operate their automated driving system. Accordingly, it may be argued that it is unreasonable to compare untrained drivers with the results of a path planner.
However, it is noted that the differences between human behaviour and optimised behaviour were generally very large (see, e.g., Figure 7). It is unlikely that more experienced/trained drivers would perform on par with the trajectory planner in the urgent scenarios. e 20 s time budgets were included to examine human driving behaviour in nonurgent takeover scenarios in relation to an optimised trajectory. e participants exhibited large individual differences, which can be attributed to personal preferences regarding when to make a lane change.
ese findings are in line with a meta-analysis by Zhang et al. [17] which concluded that participants do not take over control as quickly as possible, but may delay their steering or braking response according to their discretion. e acceleration measures and lane change duration of the trajectory planner in the 20 s condition were similar to those of the more time-critical 5 and 7 s time budget scenarios. In other words, the trajectory planner tended to finish the lane change manoeuvre relatively quickly, rather than decrease lateral accelerations with larger time budgets. Figure 9 illustrates the fact that one should not rely on a single measure (peak deceleration) as an index of takeover quality. In the urgent scenarios of medium traffic density, the trajectory planner gently decelerated and let the vehicle in the left lane pass before executing a smooth lane change.
Here, the trajectory planner adopted a deceleration of 4 m/s 2 , which was sufficient to let the vehicle in the left lane pass while retaining high comfort and forward progress. e human participants, on the other hand, showed large individual differences. Some participants did not decelerate at all and made an immediate lane change, as a result of which the vehicle in the left lane had to initiate an emergency brake;  other participants initiated an emergency deceleration themselves in order to prevent colliding with the roadworks. In summary, it was shown that a low deceleration and an early lane change to avoid collision with the roadworks are not desirable in this scenario. What is desirable is to brake with an intermediate deceleration level, a type of behaviour not identified using traditional takeover measures, but captured by the trajectory planner. e optimised trajectory was computed offline. eoretically, this calculation could be performed in real time as the trajectory planner at Volkswagen Group Research is designed for real-time application as an integral part of an automated driving system. We do not claim that the trajectories generated by the trajectory planner are optimal. Instead, the trajectories are optimised for a specific set of variables, such as route progress, accelerations, and object proximity. Somewhat different conclusions may be obtained if selecting different weights for the trajectory planner. For example, it would be possible to assign a lower weight to acceleration and increase the weight for progress, leading the algorithm to drive with a more aggressive driving style.
In conclusion, this study showed benefits of assessing takeover quality in comparison with a reference behaviour. Humans deviate substantially from optimised acceleration values if the scenario is urgent. In other words, urgent scenarios are dangerous not only because of the impending collision (a physical factor) but also because humans have difficulty behaving optimally in cases of time pressure (a psychobehavioural factor). Additionally, we illustrated that takeover measures should not be used in isolation. Maximizing driver comfort (i.e., minimizing the deceleration) was found to be highly dangerous in scenarios of medium traffic density. e comparison with the trajectory planner showed that intermediate deceleration values are most desirable.
Recently, frameworks have been proposed that combine various measures into a single takeover quality score. e Takeover Controllability Rating (TOC) quantifies takeover quality through a coding sheet, which experts use to grade human takeover behaviour by inspection of video footage [49]. Similarly, the Takeover Performance Score (TOPS) combines three variables related to vehicle guidance, mental processing, and subjective rating. ese three variables can be combined into a single quality score [10]. ese approaches represent an improvement to traditional methods, but they rely on subjective judgments. e present trajectory-planning approach can contribute to a more objective quantification of takeover quality.