Sensitivity and Performance Evaluation of Multiple-Model State Estimation Algorithms for Autonomous Vehicle Functions

Robust object tracking and maneuver estimation methods play significant role in the design of advanced driver assistant systems and self-driving cars. As an input to situation understanding and awareness, the performance of such algorithms influences the overall effectiveness of motion planning and plays high role in safety. The paper examines the suitability of different probabilistic state estimation methods, namely, the Extended Kalman Filter (EKF) and the more general Particle Filter (PF) with the addition of the Interacting Multiple Model (IMM) approach. These algorithms are not capable of predicting motion for long term in road traffic conditions, though their robustness and model classification capability are essential for the overall system.The performance is evaluated in road traffic scenarios where the tracked object imitates the motion characteristics of a road vehicle and is observed from a stationary sensor. Themeasurements are generated according to standard automotive radar models.The analysis conducted along two aspects emphasizes the different performance and scaling properties of the examined state estimation algorithms. The presented evaluation framework serves as a customizable method to test and develop advanced autonomous functions.


Introduction
Nowadays, highly automated driver assistance systems and autonomous vehicles are in focus of attention and pose many different challenges that need to be solved.The architecture of the motion planning has multiple layers from route planning, through behavioral and tactical planning to local control [1].The perception subsystem consists of vehicle and environment sensors, data fusion and tracking layers, behavioral classification and prediction tasks [2].These layers need to work together to fulfill the original task of the vehicle, namely, to reach its destination with the possible minimization of journey time though with respect to traffic rules, passenger comfort, and, most importantly, to safety.The local and short term decisions require the autonomous vehicle to have some ability to reason about the future motion of surrounding vehicles [3].This leads to the problem of behavior prediction, where the ego vehicle needs to predict the possible future trajectories of the surrounding traffic participants, such as vehicles or pedestrians [4].Classic object tracking algorithms are not suitable for mid-term motion prediction because they cannot consider the interaction of participants, though their robustness is essential for the proper input generation for these algorithms.The classification ability of multiple model (MM) systems can also enhance the efficiency of the prediction [5].
A maneuvering vehicle can be effectively tracked by using a multimodel state estimator [6].The interacting multiple model (IMM) estimator is an approximate solution of the general multimodel problem [7,8], which is computationally tractable with linear scaling in the number of the considered models.The IMM estimator was originally proposed with Kalman filters [9]; however, it can be realized by particle filters [10,11] or fused with random set filtering methods.In [12,13] IMM estimation, realized as a particle filter, was presented which is able to work in a cooperative road traffic situation.A random finite set based particle filtering in the IMM framework for cooperative road traffic application was presented in [14].The choice of coordinate system for maneuvering target tracking affects the structure of the estimation algorithm and the achieved performance [15,16].Plenty of research deals with the problem that which object tracking, or in more general, probabilistic state estimation algorithm is suitable for these purposes.One approach is to examine to what extent the process and measurement noise affect the position estimates.The authors of [17] concluded that if the effect of the process noise to measurement noise ratio is above 0.5, the IMM offers better estimates.While the study in [18] states that the performance differences are highly dependent on the maneuvers of the target.This solution assumed a so-called perfect IMM (PIMM), which is a lower bound error estimation, since it never fails to choose the appropriate model.Naturally the PIMM is an ideal model and was generated artificially.Using nonideal IMM, our study examines the effects of the hyperparameter tuning of the IMM, in a road vehicle-like environment, where the model for the sensor imitates the capabilities and performance of radar sensing.
The paper is organized as follows.Section 2 introduces the probabilistic state estimation problem, and the possible solutions, such as Kalman filter, Extended Kalman Filter, Particle Filtering, and IMM methods.Section 3 gives an overview on the scenario and models used for the evaluation of the algorithms, and, finally, Section 4 evaluates the performance of the above-mentioned methods.

Problem Statement
In probabilistic estimation the state is considered as a random variable and its distribution is approximated.The approximation can be performed with various precision; however, the underlying mathematical structure, the Bayes theorem, can give a unified description of the different methods.The general problem is to estimate the state x given measurements z and the model of the considered dynamic system.The model incorporates the time evolution of the system, which is the motion model and the measurement model, which accounts for how we get information from the system through sensors.Formally the estimation at timestep  involves the evaluation of the formula where the conditional probability (x  | z 1: ) is the posterior density of the state in question.The likelihood   (z  | x  ) can be constructed in the knowledge of the measurement model.The denominator equals the marginal Probability density function (PDF) (z  | z 1:−1 ) which is called the evidence and serves as a normalizing factor.The prior term (x  | z 1:−1 ) can be expressed with the Chapman-Kolmogorov equation where is the state transition density constructed from the motion model.The estimation can be Initialize variables: carried out in a recursive manner; however, there are no general analytic solution because of the involved integrals.In case of a linear system model and additive Gaussian noise (1) can be shown to reduce to the Kalman filter [19].
Beside the fact that the Kalman filter is the optimal minimum mean squared error filter for linear-Gaussian systems, its usage is limited by the presuppositions of the model.Most real-life systems show nonlinearities and the Gaussian presumption cannot always be held.An approximate method is the Extended Kalman Filter (EKF) which, to handle nonlinearities, linearizes the system equations by first order Taylor expansion around the current state.
A general discrete time state space system model with additive noise has the form where  is the state transition function, ℎ is the measurement function, and w and k are noise vectors.If the nonlinearities in  and ℎ are not too strong, the linearization approximates the functions well and the EKF gives good results; convergence however cannot be guaranteed.The structure of the EKF is shown in Algorithm 1.
A more general approach to handle nonlinearities or even non-Gaussian PDFs is to use the particle filter (PF), which is an umbrella term.It corresponds to a family of Monte Carlo sampling based sequential algorithms.It is a numerical method that approximates a function, in our context the posterior PDF by particles.The particles are weighted samples, drawn from the distribution; hence, the higher the particle number, the more accurate the approximation.The PDF of a state vector x  is approximated by   particles as where x () is the -th sample,  () is its weight, and  x () stands for the Dirac measure centred at x () .The particle filter can be considered a Bayesian-type filter where the PDFs are numerically handled.At the predict stage samples are drawn from a pool, called the importance density.The choice of the importance density is crucial.An obvious choice is to use the state transitional density whose method is referred to as the bootstrap particle filter.The update stage involves the computation of the likelihoods for every particle which is realized by the evaluation of the PDF associated with the measurement model.The normed likelihood values give the updated particle weights and the point estimate for the current timestep can be obtained by the weighted sum of the particle states.The main steps of the bootstrap particle filter are summarized in Algorithm 2.

Multimodel Estimation.
The purpose of multimodel filtering is twofold.On one hand it helps giving a more precise state estimation if the correct system model is used and on the other hand it provides information on the actual mode of operation.To be effective at both at the same time, as will be seen, can be a contradictory requirement.
System model is the collective term for the process or motion model and the measurement or sensor model.The mode of operation or system mode refers to a certain kind of behavior that we identify, e.g., accelerating or turning.A given process model, if general enough, can account for multiple modes of operation.Mode change is the switching between two modes of operation.The mode history is the time sequence of the actual system modes.Mode uncertainty refers to the circumstance that we are not aware of the actual system mode.
One way of dealing with model uncertainties in an estimation problem is to use a number of plausible system model, compare their performances, and choose one result or combine several.The output of a filter associated with a certain mode is referred as the mode-conditioned estimate.There are numerous multiple model algorithms at hand [6].The static multiple model approach has the supposition that the system does not change its behavior during the observation period and the filter selects the most likely mode and outputs a weighted estimate that is a combination of the individual filters.The dynamic multiple model estimator considers mode switching and uses mode transition probabilities that are predefined parameters.The exact solution to the dynamic multiple model problem, the Generalized Pseudo-Bayesian (GPB) estimator, has complexity that exponentially grows with time because it covers all the possible combinations of mode histories with a filter.Since the exact solution is intractable, one has to approximate and take into account only the last or the last two modes of operation resulting in the first (GPB1) and second order (GPB2) estimators.If  mode of operation is considered, the GPB1 runs  filters while the GPB2 runs  2 filters, because of deeper memory of the algorithm.Therefore the computational requirement of the GPB1 and GPB2 filters scales are linearly and quadratically respectively in the number of modes.

The Interacting Multiple Model estimator.
The interacting multiple model method, proposed in [7], is another type of approximate estimator to the dynamic multiple model problem.It uses only the output of the last step and creates a unique mix from the mode-conditioned estimates for every filter.Being a first order approximation it has a performance near the GPB2 but computationally only intense as the GPB1 [9].
The structure of the IMM estimator is depicted in Figure 1.The inputs to the recursive algorithm at timestep  are the mode-conditioned state estimates and the mixing weights as a matrix ] (,) for ,  = 1 . . ..The state estimates consist of a mean value and a covariance matrix, describing a Gaussian distribution.At the mixing stage the input for each filter is computed as a weighted sum of Gaussians: These values and the measurement vector z  are passed to the filters.Beside the state estimation the filters also produce the residual and the associated covariance matrices.From these a model likelihood is computed as  The mode probabilities are not purely the likelihoods because mode switching dynamics are implemented with the help of the mode transition probability matrix  through which the prior values are derived: The updated mode probabilities are An overall estimate can be computed by weighting each filter output by the mode probabilities: The associated covariance matrix is The mixing coefficients for the next step in the recursion is given by 2.2.IMM Particle Filter.Originally the IMM algorithm was working with Kalman filters [7].The realization of the IMM with extended Kalman filters is trivial, because the output of an EKF is consistent with the KF.The same does not hold for the particle filter since its output consists of weighted samples and cannot be directly integrated into the IMM framework.One method is to distribute the particles between the modeconditioned estimators.The more likely a mode estimate is, the more particles will belong to that estimator.We used another approach, which is closer to the original structure of the algorithm [20,21].Every filter has a unique Gaussian input parametrized by x() The PF draws   samples from this distribution The weights associated with the samples have equal values: 1/  .The particles are then propagated through the motion model: The likelihood of a particle is A point estimate is computed from the particles by a weighted sum: (,) | (17) with covariance The particle filter does not produce residual covariance like the EKF.To be integrated in the IMM framework, a model likelihood has to be derived.The estimated measurement is The model likelihood has an analogous form as in the EKF Here, With L ()  (z  ) the mode probability is computed as in (10).If particle degeneration is an issue, one will perform some form of resampling after the update step [22].In our case, as we sample from the mixed distribution at every timestep, no particle degeneration is possible; hence, we implemented the PF without resampling.The individual particles are not preserved; only the weighted point estimate, its covariance, and the model likelihood are utilized.

Evaluation Framework
To evaluate the filter performances a simulated environment is used.The tracked object is assumed to be a road vehicle with appropriate motion characteristics.The maneuvering vehicle is observed from a stationary sensor at the origin.The measurements are generated according to standard automotive radar models.

Models.
In this study the considered modes of operation are moving with constant velocity and turning with constant speed along a circle or a clothoid segment.
The CV and CT model both use the state vector x = [, , , V, ] and the motion model has the form: where   either stands for the CV or the CT model.With the given the state vector the CV model is which after linearization reads as The constant turn rate (CT) model is angular velocity dependent: Linearizing ( 25) around x  yields where It would also be possible to use a simple CV motion model with states [, ẋ , , ẏ ], system matrix and perform the mixing in the IMM algorithm with state vectors of different length and components.In such case, depending on the actual components of the state vectors, one has to either augment one state vector with the missing element or transform the state vector to the base of the other one.These kinds of computations, detailed in [23,24], are out of scope of the current study.For this reason the same state vector was used for both the CV and CT motion.
The noise acting on the system is modeled as an additive Gaussian term.For the CV and CT model it is a white noise linear acceleration and white noise linear and angular acceleration, respectively. where and   is a zero mean Gaussian with covariance In (36) the diagonal elements stand for the linear ( 1 ) and angular ( 2 ) acceleration.The factor  scales the noise intensity.The PDF associated with the motion model has the form The measurement vector comes from simulated radar observations with components z = [ 1 ,  2 ] ⊤ , where  1 is the bearing angle and  2 is the distance.The measurements originate from the nonlinear sensor model ℎ(x): In linearized form the sensor model reads as where  = √( 2  +  2  ).With the help of (38) the likelihood function is given by where the covariance matrix is ] . (41)

System Setup.
As it was pointed out in [25], narrow beam range finders like long range radar and especially lidar have a possibility of failing to detect thin objects, for example, lampposts.An additional problem is that even if the sensor correctly detects an obstacle, a filtering algorithm might not recognize its existence.Sensors that are practically noise free lead to a measurement likelihood function that is spike-like.The support of a likelihood function of this kind, due to limited numerical precision, can be a very small region in the state space.If the estimator algorithm does not provide a prior in that small region, then the likelihood function will be zero out of other priors.At the filtering level the narrow likelihood problem can be handled by applying a Gaussian smoother.Low level sensor fusion techniques and track-before-detect approaches can help to overcome typical sensor drawbacks [26].At the detector level, new, emerging technologies are expected to overcome this deficiency [27][28][29].
The vehicle is initialized with state vector  0 = [0, 0, 30, /4, 0].The noise acting on the system is described by Sensor noise values are chosen according to common automotive radar devices [30].

Scenario.
The vehicle moves along a road segment and its trajectory is segmented into straight lines and circular or clothoid arcs.Two trajectories were designed with different characteristics to help emphasize the nature of the filters.Trajectory1 (T1), as outlined in Table 1, has more turns and Trajectory2 (T2) has more straight parts, as indicated by Table 2.In the clothoid segment in both cases the angular velocity ( 1 and  2 ) changes linearly with time.For T1  1 starts from zero and increases to 16 deg /s in 20 steps and then decreases to −12 deg /s in 20 steps.The same is true for  2 for T2 except that 10 -10 steps are used in this case.
The speed is constant throughout the whole simulation.The duration of the simulation is  = 100 sec with time step   = 1 sec.The steps for  1 and  2 are also   .
The filter performances were examined along two parameters, the mode transition probability matrix  and the process noise intensity .The performance is also measured along two dimensions: the position error and the estimated mode.
The mode transition matrix is adopted as where  is the scaling parameter.The values taken by  and the process noise intensity  are shown in Figure 2.

Results
The performance of the filters was measured from two aspects of interest.First, the position error is calculated as the Euclidean distance between the real and the estimated coordinates.Second, as the measure of maneuver detection, the estimated mode is regarded and two measures are defined: from which the first is the ratio of time steps when the correct mode was estimated versus the total time; and the second method is area based and indicates the robustness of the output.The area under the estimated mode probability curve is divided by the length of time that mode was active.
A ratio to measure the effect of relative motion to measurement uncertainty on the position is defined in [17] as the maneuvering index.In our case it can be quantified as which takes values between 0.2 and 0.5.According to [17] an IMM is preferred to a single model KF if  is higher than 0.5.This limit is challenged in [18] with the conclusion that the IMM can outperform the KF if the model with the lower process noise is active for enough time.Numerical complexity of the compared algorithms has the following order.The single model EKF has the lowest requirements.The state space is of low dimension, the involved matrix operations can be computed directly, without sophisticated numerical methods.The IMM-EKF needs to run a filter for every considered mode of operation.In addition, the IMM algorithm poses overhead mainly in terms of creating Gaussian mixtures and computing covariance matrices.The IMM-PF is the most computationally expensive.The most demanding steps are sampling, evaluation of the likelihood function, and sample covariance computation.The particle number was held constant 1000 which can be considered as a mediocre quantity.The current implementation used parallelization wherever it was possible at the particle level.
Simulations were conducted for both trajectories.The IMM is implemented with EKF and PF and for comparison single model CV/CT EKF were used.100 Monte Carlo runs were averaged out for every parameter setting meaning 75000 runs in total.Figures 3,7,11,and 13 show IMM filter performance across the whole parameter range.Figures 4,8,12, and 14 sum up the efficiency of mode estimation and RMSE for parameter  = 2.5 and for every .
Figures 5 and 9 show the results of the 100 MC averaged runs for parameters  = 2.5 and  = 0.3.The estimation of one particular run from 100 is presented in Figures 6 and 10.
The estimated mode at each timestep is acquired by selecting the most probable mode: A general property of the IMM filters in all cases is that the position RMSE is lower for greater  values while the dependence on  shows variety.The effect of a smaller  value is that the diagonal elements (1 − ) in  become large and mode transitions are harder to achieve; resulting in a slower multimodel filter and a lower ratio in terms of modecorrectness.In contrary with a higher  value mode transitions can be realized with low latency after measurements start to deviate from the expected state.The downside of using values near 0.5 is that the robustness of the mode estimation in terms of the area based measure degrades as the correct mode gets lower confidence.This effect is demonstrated in Figures 4, 8, 12, and 14.
The position RMSE of IMM-EKF exhibits negligible dependence on  for both T1 and T2.The accuracy of mode estimation on Figure 7 shows a connection of linear nature between the hyperparameters for T2.In case of T1 the best model estimation efficiency can be achieved with lower noise and higher  parameters (Figure 3).
The single model EKF with constant turn rate can be considered as an option in certain situations.If the tracked vehicle performs a great amount of maneuvering, the EKF can outperform the IMM in terms of position RMSE at the cost of providing no mode estimates.The EKF, in scenario T1, estimates the state significantly better than the IMM; on the contrary in T2 the IMM performs comparably while the EKF is worse.The particle filter implementations of the IMM have different characteristics.The position RMSE shows dependence on both hyperparameters.The mode of operation can be estimated with greater efficiency for certain parameters compared to the IMM-EKF.In scenario T1 the dependence of the RMSE on  is weak for lower  values indicating a greater freedom at the filter design process.The mode can be estimated slightly more efficiently with the IMM-PF than the IMM-EKF in both scenario T1 and T2.

Conclusion
The continuously increasing level of automation in road traffic and vehicles poses new requirements towards data processing algorithms [31].Road vehicles with autonomous functions need the ability to sense the environment and faithfully capture and interpret the traffic situation.Algorithms for advanced driver assistance systems have to run realtime which greatly constrains the level of analysis of acquired data.This study covers performance evaluation of multimodel object tracking and maneuver identification methods of great   importance for the automotive industry.The scenario was provided by a simulated environment where the sensor model had the characteristics of common automotive radar which performed measurements on the bearing angle and distance of the tracked object.The implemented estimators are the IMM-EKF, IMM-PF, and the single model EKF.The filters were evaluated along two aspects, the RMSE of the estimated position and the performance of the mode estimates.The varying parameters of the simulation were the mode transition probability matrix and the intensity of the process noise.The robustness of estimates of the actual mode of operation was measured by two methods.The single and multimodel filters were compared in terms of position RMSE.The IMM-KF and IMM-PF have different scaling properties regarding the simulation parameters which imply different design approaches.
It is essential to fine tune or, during operation, reconfigure the filters to help adaption to the actual conditions.The presented work can serve as a guideline creating a customized framework to analyze and help the development of multiple-model state and maneuver estimating algorithms for advanced vehicular functions.In the knowledge of the scaling properties and behavior of the algorithms parameter ranges for the optimal performance can be identified.Residual covariance of mode  :

List of Notations
Mode transition probability matrix : Intensity of process noise : Process noise covariance : Measurement noise covariance Γ: Process noise input matrix  Mixing matrix.

Figure 1 :
Figure 1: Structure of the interacting multiple model algorithm.

Figure 2 :
Figure 2: Parameter set of the simulation.50  values marked with circles and 15  values marked with squares.

Figure 5 :Figure 6 :Figure 7 :Figure 8 :
Figure 5: MC averaged result of the estimated mode probabilities (left) and the position error (right).The vertical dotted lines indicate mode changings.The dashed line is the actual angular velocity of the tracked vehicle.Case: EKF -T1.

Figure 9 :Figure 10 :
Figure 9: MC averaged result of the estimated mode probabilities (left) and the position error (right).The vertical dotted lines indicate mode changings.The dashed line is the actual angular velocity of the tracked vehicle.Case: EKF -T2.
b a b i l i t yo f m o d e  ] (,)  :

Table 1 :
Motion of the tracked object in the simulation: Trajectory1.

Table 2 :
Motion of the tracked object in the simulation: Trajectory2.
: Dimension of state space   : Dimension of measurement space (x  ): P D Fo fs t a t ev e c t o rx  x  : State vector at time   w  ∈ R   : P r o c e s sn o i s ev e ct o r z  : Measurement vector at time   k  ∈ R   : Measurement noise vector  |−1 (x  | x −1 ): State transition density   (z  | x  ): :