Driver intervention performance assessment as a key aspect of L3–L4 automated vehicles deployment

1 Introduction

Driving is commonly considered as a multifactorial process that requires significant cognitive resources. It necessitates continuous road monitoring, surroundings analysis, and quick decision-making [1]. The process involves three levels of control [2]:

• Operational – refers to the driver’s reaction to traffic conditions with limited decision information. This level takes advantage of sensorimotor abilities and occurs in short intervals of time, e.g. steering and braking due to sudden changes.

• Tactic – requires vehicle manoeuvring in response to normal road conditions, e.g. at junctions. This type of control occurs for a few seconds.

• Strategic – refers to route planning, including destination selection. Occurs at intervals of minutes to hours.

Technology development in the field of advanced sensors, software, and artificial intelligence encouraged car companies to develop a self-driving vehicle (e.g. refs.[3,4]). Automated driving systems (ADS) [5] in which perception and decision-making are being made by machine/artificial components are said to become a reality in the following decades. Automation requires drivers to relinquish control of the vehicle, while maintaining awareness to enable safe performance in case the system reaches its limits. ADS should have both automated and manual modes. According to the society of automotive engineers (SAE 2018 [6]), there is a six-level scale of driving automation – from 0 (no automation) to 5 (full automation):

• Level 0 – no automation, the driver performs the entire dynamic driving task (DDT), even when enhanced by active safety systems.

• Level 1 – driver assistance, the driver controls the vehicle, and the system makes adjustments to speed and direction (e.g. adaptive cruise control, lane keep assist). The system executes only one subtask – not both simultaneously.

• Level 2 – partial automation, the driver controls the vehicle, and the system makes simultaneous adjustments to both speed and direction (e.g. park assist, traffic jam assist).

• Level 3 – conditional automation, the system has full control over the vehicle (speed, direction, environment monitoring – e.g. Steering Collision Avoidance) but only under specific conditions (e.g. motorway, limited speed, no crossings). The driver has to be constantly ready to intervene if a dangerous situation is detected.

• Level 4 – high automation, the system has full control over the vehicle, and driver presence is not necessary, but only up to the system’s limits. The driver can undertake non-driving-related tasks. If the actual driving conditions exceed the system performance limits, it may ask the driver to intervene or decide to stop the journey. The system works under specific conditions.

• Level 5 – full automation, the human-driver is not necessary and considered as a passenger. The system works unconditionally. An ADS performs the entire DDT and DDT fallback without any expectation that a user will respond to a request to intervene (RtI).

There are two paths to achieve full automation. The first one involves designing the vehicle from scratch, while the second uses incremental development through all the above-mentioned levels of automation. In this article, the second path will be discussed, namely the L3 and L4 levels, where the vehicle can move independently, but only under certain conditions, and there may be a need to retake control. It is assumed that in L3–L4 vehicles, during periods of automated driving, human will be out-of-the-loop and may not have enough information to safely maintain control at the operational and tactic level. Attention shifting may result in reduced situation awareness, causing distraction or fatigue. A Driver Intervention Performance Assessment (DIPA) uses a variety of methods to ascertain that the driver is able to respond safely to dangerous traffic incidents. DIPA comprises time-based intervention performance measures (e.g. for assessing different time budget components), as well as objective (e.g. minimum time headway) and subjective (e.g. perceived controllability) take over quality measures. Owing to the risk of an accident, which varies depending on vehicle type, driving skills, age, gender, personal characteristics [7], and arousal level, the assessment should focus on the knowledge and attitude of the driver. It could identify the risk management capability and ensure that drivers are aware of vulnerable road users, in particular pedestrians, cyclists, and motorcyclists.

2 DIPA definition

DIPA involves the definition of relevant objective measures to assess the quality of intervention performance, such as driver take over time (TOT) from onset of RtI, driver intervention time, control stabilization time, remaining action time, as well as subjective measures for the quality of intervention performance.

From the ADS’s perspective, DIPA is implemented as a decision system, laying above modules intended to monitor driver’s state, behaviour, and reactions. DIPA module activates when a RtI appears. It gathers information from driver state monitoring (DSM) and vehicle monitoring. When system identified that driver took the control, DIPA module ascertains if the human is able to safely maintain manual driving, or a decision of performing a Minimal Risk Maneuver should be taken (otherwise).

Figure 1 shows a simplified flow diagram of the input information to the DIPA module.

Figure 1

DIPA system.

3 Control transition

The assessment of the driver is crucial primarily in relation to reacting to a sudden, unexpected event on a road. It necessitates the completion of a sequence of processes in a very short time. In level 3 conditional automation, the driver is not required to constantly monitor the environment and the automated system performance. That may result in shifting attentional resources to other tasks. However, the driver has to maintain readiness for taking over vehicle control in “situations that exceed the operational limits of the ADS” [8], detected and announced by the automation system. The transition from automated driving to manual mode may be human- or system-initiated.

DIPA refers to transition forced by ADS, which is started by a RtI – a critical event requiring active driver intervention. The driver has to relocate hands and feet to the driving position, regain situation awareness, and execute an adequate response to the system limit [9,10]. ADS can met with trust by end-users if the driver state is continuously monitored, his/her availability properly evaluated and sufficiently triggered (through tailored human-machine interfaces – HMIs) [11]. The human factor is essential for the safety deployment of L3–L4 vehicles, mainly due to the necessary driver-vehicle interaction in cases where the boundaries of the Operational Design Domain are being reached. ADS limits may be caused by sensor malfunction, extreme weather conditions, appearance of evolving accident scenes, unexpected roadblocks, hazardous traffic violations falling of goods, etc. The figure below (Figure 2) describes the processes during the transition from automated to manual driving initiated by ADS.

Figure 2

System-initiated vehicle control transition [8].

4 DIPA measures

While the capabilities of an automated system may be exceeded in some circumstances, the RtI signal may appear. Once a RtI occurred, a driver has limited time to react, to ensure safety. The time interval between the RtI and the system limit is defined as “total time budget.” It represents the maximum time window after a RtI for a successful resumption of manual control. In assessing the quality of driver intervention, selected factors should describe the driver’s behaviour and driving technique, meeting the following criteria [12,13]:

• suggesting an unambiguous method of measuring,

• measurement repeatability,

• can be applied both in a simulation environment and in a real vehicle,

• have one interpretation,

• can be easily measured and presented graphically,

• applicable to engineering procedures and calculations,

• applicable in various countries, vehicles, and conditions,

• used in research and described in the literature,

• use SI units.

There is still a lack of unanimity on how to ensure driver’s ability to take over the control and safely perform the manoeuvre. The monitoring and assessment of a driver should ensure that he/she is able to control the vehicle himself/herself. It includes monitoring of sensory, motoric, cognitive, arousal, and emotional states, as refers also to DSM, but expanding these factors by measures of driving performance. The take over phases should be measured ensuring the minimal safety level after turning the driver back-in-the-loop. It could be done using objective and subjective measures. Objective measures relate to speed, manoeuvres, or other forms of behavioural analysis (time-to-collision, time headway, time-to-lane-change), as well as driver reaction measures (TOT, driver intervention time, control stabilization time, remaining action time). Subjective ones are associated with perceived controllability [5].

The objective measures are time-based intervention performance and take over quality measures as follows:

• TOT – an interval between RtI occurring and user-initiated intervention or deactivation of an engaged automation function. The transition can be divided into further subprocesses, e.g.: time to first driver reaction, time to visually fixate RtI message (if visual HMI is involved), time to start to move (at least one) hand to wheel/feet to pedals, and time to start to operate relevant vehicle controls (e.g. blinker) [8]. TOT is a human-based measure and should not be considered in relation with system deactivation time.

• Reaction time – a subprocess of take over, it is defined as the time that elapses from the moment of the stimulus (nervous impulse) activation to the moment of movement initiation [14].

• Decision time – an interval between detection of a silent system failure and the decision to disengage the automation feature.

• Intervention time – a period of time required to handle the imminent take over situation by performing an appropriate driving manoeuver.

• Driving recovery time – a measure combined with TOT and intervention time. Comparison between total time budget and driving recovery time should be used to provide guidelines that take account of all the driver’s reactions to ensure a successful manoeuvre.

• Remaining action time – a comparison between total time budget and a successfully finished driver intervention time. It may be paraphrased as a remaining time to last intervention possibility at the time of intervention [3].

• Control stabilization time – a period of time until vehicle control performance is fully re-established. After the immediate driver intervention phase, manual vehicle control performance can be below the average performance of an individual driver [8].

• Headway – the space between the reference points of two vehicles. It may be defined in both time and distance domains. Headway can be defined as the space between the outer edges of the leading vehicle’s rear bumper and the following vehicle’s front bumper (Figure 3). The size of this parameter determines the safety margin that the driver of the following vehicle leaves to react to manoeuvres of the leading vehicle. Measuring this parameter necesitates meeting the following conditions [13]: vehicles move in the same lane, vehicles follow each other on the passage path, longitudinal parameter of the distance is measured (assuming movement on a straight road).

Figure 3

Definition of headway [12].

In the time domain, the headway is expressed by the length of time that elapses from the moment it is measured until the following vehicle’s reference point reaches the point that determined the location of the leading vehicle’s reference point, assuming that the speed of the following vehicle does not change.

• Time to collision (TTC) – indicates the time left until a collision occurs if both vehicles continue on converging trajectories at their current speed (Figure 4). Low values of minimum TTC indicate either a slow braking reaction time or an inadequately low braking force [15]. TTC is only calculated when the following vehicle has a higher speed than the vehicle in front [16].

• Brake application – an indicator of maximum deceleration between the moment of resuming manual control and the moment of lane change/stopping the vehicle. May be measured in m/s². Higher values indicate larger deceleration.

• Acceleration pedal steering – a parameter describing driver’s activities due to the location of the vehicle in its longitudinal axis indirectly indicating a number of vehicle control operations undertaken to stay behind the preceding vehicle.

• Lateral lane position – a parameter indicating the vehicle’s fixed point location in the longitudinal axis relative to the fixed point of the lane cross section [17]. It describes the driver’s ability to keep the vehicle on a straight (desired) trajectory.

• Steering wheel reversal rate – an indicator of steering wheel changes from clockwise to counter clockwise, or vice versa. The frequency and amplitude of these movements identify driver’s ability to maintain control over the vehicle.

• Time to lane crossing (TtLC) – an indicator of the uncontrolled vehicle’s approach to the lane border (Figure 5). TtLC determines the time it will take the vehicle to cross the lane if the driver does not intervene (e.g. the current steering angle will be maintained).

• Post-encroachment time (PET) – time difference between a moment when an offending road user leaves an area of potential collision and a moment of arrival of a conflicted road user possessing the right of way. The value indicates the extent to which they missed each other.

• Deceleration-to-safety time – the necessary deceleration to reach a positive PET value if the movement of the conflicting road users remains unchanged.

  Subjective take over quality measures:

• Intervention accuracy – the accuracy of choosing intervention type (every intervention should be carried out using the best possible method). The choice of intervention method is subjective and depends on the skills and experience, as well as the environmental and ethical context. Resuming manual control may be performed by: braking, accelerating, or steering intervention. The best intervention method may differ in various driving scenarios [18].

• Quality assessment of safety effects – a measure using subjective rating of criticality. Each level is set based on measurements, but categorization and intervention time division is done subjectively. The method may be designed with categories, e.g. uncontrollable situation (no time to react), dangerous, unpleasant, harmless, and nothing noticed (no danger). The assessment method may be based on time-metrics, acceleration values, or distance [16].

• Crash probability – a method similar to the previous one, but in addition to measuring time or motion-oriented values; it is also based on the likelihood of an accident. It can result in categorization or present results in percentages [10].

• Self-reported difficulty/danger – a survey carried out after performing a take over. The subjective driver’s rating of the difficulty/danger of the situation. The outcomes may be presented on the scale or converted to a percentage [19].

• Expert-based assessment – a method of integrating different timing and quality parameters into a global controllability measure [8]. Trained evaluators are required to integrate different aspects of driving performance by using a standardized rating scheme (Figure 6). The method proposed by Naujoks et al. [20] was developed in reference to automated vehicles, based on an existing assessment procedure that has been successfully applied to assess the criticality of driving situations in manual driving conditions (Kaussner’s driving behaviour assessment [21]). The TOC-Rating (Take over controllability rating) may be used in empirical studies on controllability in accordance with the Code of Practice [22].

Figure 4

Definition of TTC [12].

Figure 5

Definition of TtLC [12].

Figure 6

Overview of TOC-rating process [20].

5 DIPA framework

Trustonomy (trust + autonomy) is a project connecting stakeholders from different automotive industry, research, and transport areas. It aims to compare in terms of performance, ethics, and acceptability different technologies and approaches in different scenarios for automatic driving and requests to take control. The project investigates several ADS-related domains including DSM, performance assessment, and training. A DIPA framework is being under the development within the project [23]. Other assumptions, goals, and expected results are described in the article Bringing Trust to Autonomous Mobility, written in proceedings of the 5th AEIT International Conference of Electrical and Electronic Technologies for Automotive (18–20 November 2020) [24].

6 Conclusion and discussion

In assessing driver’s intervention performance, it is essential to present the overall safety effects of a transition in a particular traffic situation. Driving scenario context is crucial, as there is no intervention type suitable for all dangerous situations. Thus, different outcomes and measures of take over quality should always be compared carefully and with extensive attention to details. It is always recommended to use a combination of appropriate measures as single measures may not be sufficient enough to differentiate safe and unsafe events.

DIPA can be considered in various sensor configurations, involving different measures of driver state and behaviour. It can be also considered in multiple road situations, with different causes of RtI, which may differ driver behaviour and maximal available time for reaction (total time budget). Since it is not possible to consider all possible cases, in Trustonomy, several typical scenarios will be selected and used as a basis for the DIPA framework to be developed. All details presented in this paper are early estimations and may change in further work.

Although a driver is considered to be an end-user of the final design, the main outcomes of the project directly address the needs of the automotive industry. DIPA framework will provide methodology and measures that can be used to develop decision system for the RtI situation. The outcomes of the project may be useful for developers and integrators of Automated Vehicle systems and treated as a background for standardization activities or for test protocols development.