PRORETA 5 – building blocks for automated urban driving enhancing city road safety

1.1 Use case

The research in the project focuses on urban scenarios developing novel approaches for different functions of AD systems. Urban environments pose particular challenges for AD. They are highly dynamic, with pedestrians and bicyclists quickly changing directions. Unclear priority situations, for example because of narrow roads, demand situational understanding and cooperation. Occlusions by parked vehicles or other objects obstruct other traffic participants. The use case of PRORETA 5 covers these challenges by developing an urban pilot for narrow roads. Table 1 gives a brief overview of the operational design domain (ODD) of the system.

1.2 Contribution and overview

The PRORETA 5 system architecture is presented in Figure 1. The software architecture of the automated system is based on the sense-plan-act structure. In the sense layer, raw sensor data and an HD map are processed. A road model is created by localizing the vehicle inside the HD map using a GNSS reference system. By combining RADAR and LIDAR data, traffic participants of the current traffic situation are detected. In the planning layer, the movements of dynamic road users are predicted. Then, the trajectory planner combines all previously processed information into a future trajectory. In the act layer, the planned trajectory is safeguarded with the safety check module before it is executed by the feedback controller.

Figure 1:

PRORETA 5 system architecture. The yellow marked modules were developed in this project.

Several building blocks for AD in urban environments have been developed in this project. Modules that are outside the scope of this project have been provided by Continental’s AD software stack. An object tracking based on RADAR and LIDAR data estimates the state of other traffic participants in the vehicle’s surveillance area. The tracking is performed on a manifold and described in Section 2.1. In addition to the object detection based on LIDAR and RADAR data, in PRORETA 5, the usage of visual saliency models for the detection of traffic participants in images was investigated. In Section 2.2, an overview of the main results and challenges of this approach is presented for the prospective implementation of saliency-based detection in forthcoming projects.

The object tracking output is fed into a multimodal object prediction presented in Section 2.3. The prediction module is a neural network trained on simulated and real-world data. The object prediction is able to predict the traffic participants’ most likely trajectories into the future, which supports decision making of the trajectory planner. The trajectory planner is a tree-based anytime algorithm that outputs the current best trajectory based on the ego vehicle’s speed, steering angle, road network, and predicted trajectories of other traffic participants (see Section 2.4). Before sending the trajectory of the planner to the vehicle controller, the logic-based safety check ensures that the planned trajectory is safe and can be performed collision-free. If this is not the case, the safety check performs a risk minimizing emergency maneuver into standstill. The safety check, which is based on RADAR data and unprocessed LIDAR point cloud data, is presented in Section 2.5. Furthermore, a framework for traffic rule compliance generating behavior constraints in the context of AD is developed and presented in Section 2.6.

2.1 Extended object tracking

In the realm of AD, the capability to estimate the kinematics and extent of objects in the immediate surroundings is crucial for safe operation and collision avoidance. Extended object tracking addresses this challenge by estimating the state of pedestrians, byciclists, and vehicles. In PRORETA 5, the object tracking is based on RADAR and LIDAR, which play a pivotal role in AD. An overview of different tracking approaches for the individual sensors is given in [13]. RADAR-based solutions are especially effective for detecting objects with refelective surfaces and estimating their velocities. LIDAR relies on processing point cloud data to track entities with complex shapes and provide better estimates of the object’s position.

Due to the availability of two object lists provided by the Continental ARS430 RADAR and the Ibeo LIDAR ECU, a high level fusion is performed instead of processing the RADAR’s and LIDAR’s low level sensor information. The object’s tracking state is partially provided by the respective sensor object list, which is then probabilistically fused by means of one Extended Kalman Filter (EKF) per object. Our solution is akin to [14], where the authors track vehicles with infrastructure-based sensing systems and propose a generic sensor interface for the tracking approach. However, our approach tracks objects relative to a moving ego vehicle instead of stationary sensors. Our approach tracks a reference point of the object, where a similar approach using LIDAR only is presented in [15]. Both sensors are fused in [16], where the authors only track an object’s 2D position and velocity, while our algorithm also estimates the object’s extent, turn rate, and 2D acceleration. We also provide detailed information of the measurement functions and the Jacobian derivations. The presented approach takes advantage of each sensor’s strength and can be easily extended to additional sensing systems such as camera. Furthermore, the presented high level fusion can potentially benefit from vehicle-to-vehicle (V2V) communication, where vehicles share motion information like position, speed, acceleration, turn rate, and other useful quantities among each other [17].

The state of an object at time t is defined as

where the capital letter subscripts indicate the respective coordinate frame the variable is given in. There are three relevant coordinate frames: the ego vehicle frame E, the frame of the tracked object O, and the local odometry frame L. p _L and ψ _L are the 2D position and the yaw angle in the local odometry frame L, respectively. v _O and a _O are the body-fixed 2D velocity and acceleration in the object’s coordinate frame O, respectively. The object’s yaw rate is given by ψ ̇ . Finally, the state contains the object’s length and width given by s . All the relevant frames, variables, and measures for the object tracking are displayed in Figure 3. The ego vehicle on the left is tracking an object on the right. Before performing EKF updates at time t + 1, the object’s position and orientation are predicted with

where R ₂(◦) ∈ SO(2) is the 2D rotation matrix of the respective angle and Δt is the time between t + 1 and t. Although the object’s acceleration is part of the state vector, we refrain from predicting the velocity using v _O,t + a _O,t  Δt, since this may lead to instability in certain cases. Special care is needed for estimating the angle ψ _L ∈ [−π, π) with its discontinuity at 2 π k , k ∈ Z . In our case, we perform state estimation on a manifold and normalize the angle after each operation according to [18, eq. (24)].

Figure 3:

Relevant frames, variables, and measures for the object tracking.

The RADAR measurement used for Kalman filter updates is given by

The RADAR measurement offers the possibility to update all parts of the state vector (1) except for the object’s yaw rate. The measurement function results to

where the superscript ego indicates measures of the ego vehicle. The LIDAR measurement used for Kalman filter updates is given by

where the LIDAR measurement does not contain the object’s size, since the object’s size is estimated with RADAR measurements in our case. While the LIDAR cannot infer an object’s full dimensions when observing it from one specific side only, the RADAR is able to determine the size more precisely through reflections on the ground. In comparison to the RADAR measurement, the Ibeo Lux LIDAR ECU provides an estimate of the object’s yaw rate. The yaw rate is particularly helpful when predicting an object’s movement. The measurement function is given by

Tracking an object’s center with multiple sensor sources may lead to jumps in the estimated state, when not taking into account the different bounding box sizes [15]. This circumstance is visualized in Figure 4, where a vehicle is tracked from a rear view point and both bounding box centers exhibit a non negligible distance to each other. To alleviate this problem, a reference point of the bounding box, provided by the respective sensor object lists, is tracked instead. In Figure 3, the reference point is in the object’s center. When a measurement arrives with a different reference point that is being tracked, one has to change the position p _L using

where o is the offset between the previous and new reference point.

Figure 4:

Exemplary LIDAR and RADAR bounding boxes of the same object with a rear view point and possible reference points (circles).

The EKF’s covariance is propagated for the operations (2), (4), (6), (7) using Σ _t+1 = JΣ _t J ^⊤ , where J is the Jacobian. We leave out the definitions of the Jacobians G = ∂ g ∂ x , F = ∂ f ∂ x , H RADAR = ∂ h RADAR ∂ x , H LIDAR = ∂ h LIDAR ∂ x due to lack of space. However, these Jacobians can be computed using a symbolic computer algebra system.

The extended object tracking fuses processed information of the RADAR and LIDAR, which is sufficient to robustly track pedestrians, bicyclists, and vehicles. State estimation w.r.t. the extent of the object can be improved by also handling the width and length of the object on manifold as in [19].

2.2 Visual saliency

In general, a visual saliency model takes an input image and detects areas likely to contain whole, partial, or groups of objects based on salient features, such as size, color, and shape [20]. The detected areas are typically displayed using an intensity map, as illustrated in Figure 5. In the context of AD, saliency maps have been used for object detection applications. The salient areas in a map can serve as region proposals for object detection [21], [22]. Similarly, saliency maps also have been used to identify the areas of the scene [23] as well as the objects and traffic participants that attract the driver’s attention [24].

Figure 5:

Coverage and agreement trade-off of saliency maps. Traffic image and binary mask of traffic participants taken from the KITTI semantic segmentation dataset [25].

Two applications of the saliency maps were investigated in PRORETA 5: saliency-based traffic participant detection [26] and detection of driver attention over traffic participants, along with the assessment of drivers’ attentive state based on measures of gaze allocation [27].

By focusing on the overall detection results, previous works on saliency-based object detection lack an analysis of the quality of the saliency maps in terms of their correspondence between the salient areas and the traffic participants in the image. In order to fill this gap, a systematic evaluation of different saliency models was conducted [26]. The evaluation revealed that saliency models exhibit a trade-off between the coverage of the salient areas and the agreement with the image segments containing traffic participants [26]. This trade-off is illustrated in Figure 5 with a set of saliency maps computed using the Spectral Residual model [28] at different resolutions (width × height). A coarse resolution produces a map with large connected salient areas covering several traffic participants. This results in good coverage and bad agreement, as shown in the 64 × 19 map where the smaller traffic participants in the background are contained within a single salient area. At higher resolutions, the model produces maps with small and sparse salient areas. This results in bad coverage and bad agreement for the traffic participants in the foreground, as shown in the 256 × 77 map.

The coverage and agreement trade-off can be thus summarized as follows: while a large salient area might cover many traffic participants simultaneously (i.e., good coverage with bad agreement), small and sparse salient areas might only cover them partially (i.e., bad coverage and agreement for mid-size traffic participants and good coverage and agreement for small traffic participants). This trade-off hinders the reliability of saliency models for detection tasks in challenging situations, as in images containing large traffic participants in the foreground and small ones in the background. Furthermore, saliency models tend to produce a large number of false positives due to high-contrast areas of the image, such as the contours of trees and buildings against the sky [26]. In conclusion, the results show that saliency detection needs to be adjusted for traffic participants appearing in different dimensions and that false positives caused by elements of the surroundings and the landscape need to be minimized [26]. It is important to note that saliency-based object detection was not integrated in the PRORETA 5 system illustrated in Figure 1 and thus it is not featured in the system demonstration reported in Section 5. Nevertheless, the results are reported here in order to provide a complete overview of the methods investigated in PRORETA 5.

2.3 Multimodal object trajectory prediction

In order to predict the future behavior of various traffic participants, a multimodal trajectory prediction approach is used, which is explained in detail in [29]. Trajectory prediction plays an important role in the automated vehicle industry, helping to anticipate the behavior of other road users and helping the vehicle make informed decisions, such as avoiding collisions or adjusting speed.

Therefore, this area of investigation has been addressed by numerous researchers, particularly focusing on diverse deep learning prediction techniques. Recent papers that report the application of such methods are reviewed in [30]. Currently, several benchmark datasets are available, e.g. [31], [32], alongside active corresponding competitions such as Argoverse [33].

To calculate an accurate trajectory prediction, multiple inputs are considered, such as the observed movement of the traffic participant, the movement of other surrounding traffic participants, road structure, traffic signs, and other environmental information. An important aspect is to consider the interactions of road users, e.g. overtaking maneuvers, a pedestrian crossing the street that can lead to different future trajectories based on the decision made during the traffic situation.

The approach that is used in the PRORETA 5 project is defined by a model with multimodal trajectory prediction [29] for traffic participants like cars, buses, trucks, bicycles, and pedestrians. Understanding the environment and intentions of other traffic users can improve the quality of the planned trajectory and offer a more stable and comfortable experience for the passengers in an automated vehicle. The methodimplemented for prediction is using vectors and an original design of a neural network that combines information about the observed objects, group context, and road context. This network was developed using the in D dataset [34] and a dataset created from real-world scenarios using the PRORETA 5 vehicle.

Besides its real-time capabilities, that allow the model to run on a real car, another advantage of this approach is that it combines several types of data about the agents, road information, and context and is capable of learning the interactions between traffic participants.

As presented in Figure 6, the model has an encoding part in order to extract information from the observed object (past trajectory and the type of the object), the group context (information about the surrounding objects near the observed object), and the road context (information about the road structure), followed by the decoding part in order to predict multiple possible trajectories (modes), more details in [29]. The encoding part consists in three fully connected layers with a leaky ReLU activation function for each observed agent, for the group context and also for the road context. For the group and for the road context, a symmetric function must be used, because such a function is insensitive to the order in which its operands are considered. In our case, a max pooling component is applied to the embeddings of the other traffic participants and road structure. The decoding part of the network extracts the information for future trajectories using three fully connected layer with a leaky ReLU activation function and a second fully connected layer with a sigmoid (logistic) activation function. The outputs represent the future trajectories with a probability for each mode (trajectory). For this network, information about past, group context, and road context were extracted in order to learn different traffic interactions. In this way, the network can associate a past trajectory with surroundings information.

Figure 6:

Overview of the object trajectory prediction approach, based on [29].

The training of the model is based on a conditional loss function that consists of two terms. The first term is a general loss that is applied to all modes and aims to decrease the mean squared error (MSE) of the trajectories weighted by their mode probabilities. The second term is a specific loss that is used only for the best mode, i.e., the mode with the smallest MSE. In order to integrate the object prediction in the PRORETA 5 vehicle, the Frenet coordinates are used. This improves the precision of the predicted trajectories by transforming curved road segments to straight ones.

Furthermore, the object trajectory prediction module was verified on the test track for different types of traffic participants in various traffic scenes. The prediction results can be improved by including more variation (e.g. speed, traffic participant type, and behavior) in the dataset.

2.4 Anytime tree-based trajectory planning

To plan the future trajectory of the automated system, a novel approach, CarPre trajectory planning (Monte Carlo Model Predictive Trajectory Planner) is used, which is explained in detail in [35]. In the following, the motivation as well as a short overview of the planning approach is presented.

In the state of the art, the planning problem is divided into behavioral planning and motion planning as in [36], [37] to cope with the complex, non-convex planning problem. While this approach is often chosen (e. g. [38], [39]), it has its disadvantages in the urban domain, since the finite set of high-level behavior classes must be chosen such that all possible traffic scenarios can be handled. However, the number of possible traffic scenarios is very large due to the potential high number of other traffic participants as well as the limited driving area. Therefore, determining the set of discretized behavior classes is difficult, which limits the approach of division. The common alternative of determining a behavioral trajectory (e. g. [40]) has the problem of ensuring that the motion planning maps the behavioral trajectory and does not converge to another local optimum. Therefore, the combined planning problem of behavioral and motion planning is considered as in [41], [42]. However, since this is complex and a long planning horizon is needed to make foresighted decisions, efficient algorithms are needed. One promising approach, which is used in behavior planning, e. g. [43], is Monte Carlo tree search (MCTS) [44]. MCTS is known to solve complex decision problems efficiently and is therefore chosen as the basis for CarPre planning. In our approach, a conversion to Frenet coordinates as in, e. g. [45], is omitted since lateral accelerations can no longer be taken into account during planning. This results in problems with sharp curves [41], such as 90° curves which commonly appear in urban areas. Likewise, a path-velocity decomposition as in e. g. [46] is omitted, because the lateral vehicle movements depend directly on the vehicle speed [47].

In CarPre trajectory planning, a trajectory tree is iteratively created by MCTS, where each tree node represents a possible vehicle state x _ego and each edge a discretized action u _ego. Here, each action is a composite pair of an acceleration a and a steering rate value ω. Applying an input u _ego,t for a sampling time T _in, a new state x _ego,t+1 can be calculated from the previous state x _ego,t using the kinematic single track model of [35] with the reference point on the middle of the front axle as discussed in [48]. This creates an equitemporal graph to extract the planned trajectory. To discretize the action space, the acceleration values are equally distributed and limited by the minimum and maximum acceleration:

For the lateral movements, the steering angle δ is discretized by equally distributing δ between the limits defined by the speed-dependent transformation from [47]:

with the maximum drivable path curvature κ _max, the maximum lateral acceleration a _lat,max, and the wheelbase l of the vehicle. This can be seen in Figure 7, where the extracted boundaries from [47] (dashed red lines) are adapted for the CarPre planner (red lines) to increase the comfort of the system. The steering rate ω is then chosen so that no discretization errors arise.

Figure 7:

Discretization of the steering angle δ (orange lines) compared to extracted statistics of human drivers from [47] (2D histogram with a logarithmic scale). Figure taken from [35].

To estimate the long-term effect of each short-term action of the vehicle, Monte Carlo simulations combined with a heuristic are used. The heuristic, extracted from human driving data, models steering to follow the driving lane as well as minimizing the jerk (i.e. choosing the same acceleration value as before). With this, the estimated reward for each node is always available and is enhanced with more iterations of the algorithm. This enables real-time applications since a planned trajectory can be extracted at any time. The algorithm is put into operation on the PRORETA 5 vehicle and a first evaluation of the algorithm can be found in [35], which reveals comfortable and foresighted driving. For future work, the result of the Monte Carlo simulations can be approximated using neural networks to increase the speed of convergence, which enables longer planning horizons.

2.5 Logic-based safety check for AI functions

As Nascimento et al. [49] point out in their study, besides their great potential in various fields of AD, AI approaches also pose risks regarding safety. Since it is difficult to gain information about the reasoning of AI algorithms, approaches for enhancing safety of AI-driven automated vehicles are e.g. to increase the amount of training data and especially the coverage of critical scenarios [50]. However, completeness of training data representing the entirety of reality can hardly be reached. There are several approaches for safeguarding an automated vehicle, like [51] and [52], but they mostly assume ideally working sensors and thus only safeguard a part of the automated vehicle. The goal of the so called Safety Check (SC) concept, which was previously presented in [53], [54], is to safeguard the results of the AI algorithms during operation in a separate module before the resulting trajectory is performed by the vehicle. Consequently, the SC only uses conventional approaches without learning-based AI approaches in order not to have the same AI-caused safety issues as the modules of the normal operation behavior planner. Since another goal of the SC is to be easily integrable, no additional safeguarding hardware is used. The SC only has access to the same sensor data as all other modules in the AD system but uses different approaches of interpreting them.

To check the safety of the currently planned trajectory and the state of the AD system, several submodules perform different checks. Besides the system health check, which checks if all modules and sensors in the AD system are running and sending data, the plausibility of the sensor data as well as the object list from the perception module are checked. Thus, the SC can rely on the correctness and completeness of the available sensor data or object lists for the object criticality check, which is another SC submodule used to identify collision critical objects. Furthermore, the behavior space conformity (see Section 2.6), the physical feasibility and the temporal stability of the planned trajectory are checked. The latter means the consistency of successively planned trajectories. The architecture of the SC is depicted in Figure 8. Further details about the architecture and the functionality of the submodules can be found in the dissertation of Popp [54].

Figure 8:

Architecture of the SC module, based on [54].

The value of the safety flag is set by the mentioned submodules and can be set to safe or unsafe. If none of the submodules rates the state of the AD system or the currently planned trajectory as unsafe, the planned trajectory is forwarded to the motion controller. Otherwise, if at least one submodule detects a safety issue, an emergency trajectory is initiated. Since the considered ODD includes a speed limit of 30 km/h, braking to standstill is mostly preferable to evasion maneuvers in safety critical situations. Thus, the preferred strategy to reach a minimal risk condition is to decelerate to standstill along the path of the currently planned or the last safe trajectory, depending on the kind of detected unsafe state.

The functionality of the safety check was verified on the test track as well as in public traffic, focusing on object list plausibility and object criticality [54]. The validation of the safety check module is still pending.

2.6 Specifying and testing traffic rule compliance

One major aspect of road safety is the conformity to traffic regulations. As for all other traffic participants, an automated vehicle is required to behave according to the traffic rules. Thus, within PRORETA 5, the consideration of traffic rule compliance is incorporated in the development process. From the stated requirement, it follows that a holistic consideration of traffic rule compliance is indispensible. This means that first the behavior constraints for the vehicle based on traffic rules need to be specified. Then, necessary functionalities on system level within the targeted ODD need to be derived and finally, test criteria and methods need to be developed to verify these functionalities.

Current approaches in the field of behavior frameworks for automated vehicles consider traffic rules, but either the rules are only regarded on a high level within a rule hierarchy [55] or they are modeled for specific scenarios only [56], [57]. Apart from this, various approaches aim to formalize traffic rules for testing general traffic rules at runtime in the vehicle [58]–[61]. Also, there exist approaches for specific traffic areas (e.g. specific intersections) [62], [63]. Still, there is no approach known to the authors that is applicable to different traffic domains by using an universal description of behavior constraints. Thus, within PRORETA 5 methods have been developed to close this gap.

In order to specify the behavior constraints, it is necessary to delineate which behavior is rule compliant and which is not within the entire ODD. This directly depends on the local scenery,^[2] which represents the static traffic environment. Present elements (e.g. traffic signs) instantiate the rules that constrain the vehicle behavior. Current description approaches represent the scenery as a human perceives it, raising the questions: what is the actual necessary information from behavioral perspective and how may it be represented? These questions are answered with the usage of the Behavior-Semantic Scenery Description (BSSD), which was previously presented in [64] and [65]. This approach introduces a structure of attributes (speed limit, boundary, reservation, and overtake) that assigns the traffic-rule based behavioral information to the scenery and thus, directly specifying the behavioral constraints. Because these attributes directly describe behavioral rules and not elements of the traffic environments, they are applicable to any traffic area.

Figure 9 depicts one of the scenarios analyzed in PRORETA 5 with the BSSD of the local scenery. The green and red colored sections of the roadway depict lanes that are own-reserved and externally-reserved, respectively. The reservation describes the rules regarding priority and to reside in a certain space. The lane pointing in the driving direction of the ego vehicle is therefore own-reserved. The ego vehicle has priority within this lane and can stay permanently in this lane. Shifting over the dashed lane marking into the oncoming lane means moving into an externally-reserved space because priority must be given to oncoming vehicles. The ego vehicle is only allowed to stay in this lane as long as it does not hinder oncoming vehicles. This is expressed by the link (red arrow) of the reservation. As a consequence of the dashed lane marking, the boundary attribute is set to allowed. In both lanes, there exists a speed limit of 30 km/h due to the respective speed limit sign in the scenery. Because there are no further traffic signs, overtaking is allowed. In [66], a method to derive these attributes automatically from an HD-map is presented.

Figure 9:

Overview of shift out of lane scenario with underlying BSSD description.

With this representation, the constraints for vehicle behavior are described semantically. To be able to evaluate the vehicle behavior, it is necessary to quantify these constraints in order to receive test criteria. In [67], all behavioral attributes are formalized using predicate logic. These criteria were used within PRORETA 5 to evaluate the test cases. By using the systematic approaches of Schuldt [68] and Amersbach [69], a test plan for the validation of traffic rule compliance was derived. The test effort was thereby reduced by implementing equivalence partitioning [70, Part 4, p. 10], boundary value analysis [70, Part 4, p. 12] and combinatorial test design techniques [70, Part 4, p. 15]. A more detailed insight into this process is given in the dissertation of Glatzki [67].

For the introduced example in Figure 9, during development, insufficiencies of the vehicle behavior were found by applying the reservation attribute criterion. The criterion was violated because the ego vehicle was leaving the lane in front of the oncoming vehicle too late and thus, hindering it to continue in its lane.

Further research revealed that the ego vehicle had left the ego lane already in distance to the parked objects. Because the environmental perception had not yet picked up the oncoming object, the planner was unable to determine with sufficient reliability whether driving in the oncoming lane constitutes a violation. When the approaching of the oncoming object was eventually seen, though, the ego vehicle also failed to reposition itself in the ego lane. By modifying the cost function of the planner module, the ego vehicle moves out of the lane closer to the parked objects and the planner is thus able to better judge if it can move into the oncoming lane. With this adaption, the vehicle conformed to the reservation attribute. Therefore, the introduced method helped finding behavioral insufficiencies of the system and improving the system such that it conforms to the traffic rules.

Specifying and testing the traffic rule compliance uncovered various functional insufficiencies in the system. Since not only traffic rules are influencing the behavior, a clear hierarchy for all behavior specifications needs to be determined (e.g. [55]) in the future. A violation of the traffic rules might be reasonable if a collision is thereby avoided. This hierarchy needs to account for misbehavior of other traffic participants and possible compensation of the ego vehicle. For this, knowing and understanding the rules is essential, for which BSSD serves the basis for. An international agreement of the regulating legislative bodies on quantified limits for vehicle behavior will be needed.