Performance Comparison of the IMMPDAF Algorithm Using Real and Simulated Radar Measurements

: Performance indexes obtained in idealized simulated scenarios are the primary source of data for evaluating different target tracking algorithms in most researches presented in the literature. Despite the convenience of simulation, ultimate evaluation of a tracking algorithm must be made in real scenarios. Unfortunately, real radar measurements as well as accurate aircraft position, necessary for calculating tracking errors, are not easily available. In this paper, we present an evaluation of the well-known Interacting Multiple-Model with Probabilistic Data Association Filtering algorithm using data obtained from a flight inspection of a Brazilian Air Force ground-based long-range surveillance radar. The presented results show that, in this scenario the Interacting Multiple-Model with Probabilistic Data Association Filtering algorithm performance using real data is worse compared to simulation. Statistical properties of the real radar measurements are also investigated, and some evidence is found that embedded noise is not well modeled as perfectly white.

Probabilistic Data Association Filter (PDAF) (Bar-Shalom et al. 2009) and its extension to maneuvering targets, the Interacting Multiple-Model with Probabilistic Data Association Filtering (IMMPDAF) (Kirubarajan et al. 1998).Besides the zero-mean white Gaussian measurement noise assumption, these algorithms are also rooted on simplifying assumptions regarding radar performance characteristics and spurious detections, such as constant probability of detection and uniformly-distributed clutter (Bar- Shalom et al. 2011).
Despite being mathematically convenient, such assumptions may be far away from conditions presented by real scenarios of application.Therefore it is reasonable to say that only when an algorithm is deployed and evaluated in these scenarios the designer can truly assess its performance.This question has been addressed in some recent researches such as Hess et al. (2014), where efforts have been made to verify the performance of tracking systems in real conditions of inspection flights.
Following this approach, the current paper uses data obtained in a flight inspection of a Brazilian Air Force ground-based long-range radar to evaluate the performance of the IMMPDAF algorithm.The results obtained with real measurements are compared to those obtained with simulated data, but the generation of simulated measurements differs from previous studies in the literature.Instead of emulating target dynamics with simple kinematic models, radar measurements are generated by adding white Gaussian noise to the actual target trajectory obtained from the inspection aircraft.The main contribution of this paper is to show that the IMMPDAF can yield larger estimation error in a real environment compared to simulation possibly due to different statistical properties of noise in simulated and real environments.
The paper is organized as follows.In the next section, we present basics of target tracking and the IMMPDAF algorithm.Next, data collection and its preparation are described.Afterwards, some numerical results and conclusions are presented.

IMMPdAF And tArGEt trAcKInG BASIcS
Under the multiple-model approach, the maneuvering target is modeled as a stochastic dynamic system whose state and observation equations are usually given by: Considering measurement uncertainty for handling targets in clutter, association hypotheses are defined to express the choice of a particular measurement-to-track association (Bar- Shalom et al. 2011).In order to avoid searching the entire measurement space to construct measurement-to-track associations, a multidimensional gate, or validation region, is defined.The set of validated measurements (i.e.those that fall inside the validation region or gate) at time k is denoted as Z k = {z k,i } i=1 , n k being the number of validations at that time instant.An association hypothesis at time k is defined as Θ k = θ q , where θ q , q ∈ {0 … n k }, is the event in which the q-th measurement was originated from the target of interest, and q = 0 is used for the hypothesis that no measurement came from the target (the target was not detected or its measurement was not validated).The set of all validated measurements up to time k is denoted as The posterior density of the target state vector at time k is written considering all model and association sequences (Bar- Shalom et al. 2011), as follows: where L k = ∏ l=1 (n l + 1).M j and Θ i represent M k = ψ j and Θ k = θ i , respectively.
The density in Eq. 3 is a Gaussian mixture with exponentially increasing number of terms, and the evaluation of the a posteriori mean of x k (the Minimum Mean Square Error estimate, referred to as MMSE estimate) cannot be feasibly realized.Numerous sub-optimal algorithms have been proposed in the literature (1) Performance Comparison of the IMMPDAF Algorithm Using Real and Simulated Radar Measurements to approximate Eq. 3 and provide good approximations of the MMSE estimate at a feasible computational effort (Blackman and Popoli 1999;Rong Li and Jilkov 2005).
The IMM algorithm has been one of the most successful tools for tracking maneuvering targets without assuming measurement origin uncertainty (Bar-Shalom et al. 2001).On the other hand, the Probabilistic Data Association (PDA) Filter (Bar- Shalom et al. 2009), in particular, handles measurement origin uncertainty for a single flight-model by combining the Gaussian mixture representing all measurement associations at time k into a single Gaussian.
The basic idea of the IMMPDAF algorithm is to combine IMM and PDA to track a single maneuvering target in clutter.It first merges all measurement associations conditioned to the same model at time k into a single Gaussian (PDAF step) and then uses the IMM framework to propagate a Gaussian mixture with r terms.
A detailed description of the IMMPDAF can be found in Bar-Shalom et al. (2011), Kirubarajan et al. (1998) and Sinha et al. (2006).It is worthy to notice that to implement this algorithm in practice some parameters assumed to be known must be chosen by the designer.This is the case of the model-set and the model transition probability matrix.Guidelines for choosing these parameters can be found in Bar-Shalom et al. (2001).Parameter values needed in the measurement-to-track association procedure are the gate probability, i.e. the probability that the true measurement from the target falls into the validation region, and the sensor probability of detection, used in the association probability update.Details about the choice of these parameters can also be found in Bar-Shalom et al. (2011).

dAtA coLLEctIon
In this section we describe the flight inspection data and simulation experiments conducted for evaluating the IMMPDAF algorithm.

Flight inspection
The Brazilian Air Force usually carries out flight inspections to evaluate the performance of radar and tracking systems by comparing information from aircraft avionics with those displayed by ground systems.One of these flights was selected for the evaluation presented in this paper.The radar in this inspection is a ground-based long-range surveillance radar, with antenna rotation period of 10 s.This radar is suitable for air traffic control as well as for defense applications.
The flight was performed by a Hawker EU-93A aircraft of the Flight Inspection Group (GEIV) of the Brazilian Air Force (DECEA 2015).Its 2-D trajectory obtained onboard from GPS data is depicted in Fig. 1.Axis coordinates are given in nautical miles (1 NM = 1,852 m), and the radar was at coordinate (0, 0).The flight started at coordinate (80, 0) and comprised a sequence of maneuvers performed by the aircraft.Longitudinal acceleration/deceleration as well as vertical ascending/descending maneuvers (not explicitly shown in the 2-D plot of Fig. 1) have also been executed.
GPS position points were available every second, and the radar produced target measurements every 10 s (antenna rotation period).Consequently, radar data had to be linked to the corresponding GPS position point.This was achieved by analyzing the timestamp of the radar plot and selecting the two nearest GPS position points.These two points were then linearly interpolated to match the radar measurement timestamp.
Furthermore, when dealing with real data one has to ensure that biases in measurements are properly corrected to produce zero-mean measurement noise, as it is usually assumed in the evaluation of tracking algorithms.The SASS-C software (Zeebroek 2010), developed by EUROCONTROL and licensed to the Brazilian Air Force, was employed to obtain radar measurement bias values that were used in the bias compensation procedure.SASS-C uses trajectory reconstruction algorithms based on Renes et al. (1985) to calculate sensor systematic errors.
Estimates of noise standard deviations used for range and azimuth coordinates were obtained by the Plot Accuracy Analysis tool (PAA) of SASS-C using real data from a recording of approximately 3 h.This tool calculates differences between radar measurements and points of the reconstructed trajectory to produce accuracy measures (Zeebroek 2010).These procedures are summarized in Fig. 2.

nuMErIcAL rESuLtS setup
The IMMPDAF algorithm was implemented in MATLAB using the IMMPDAF equations presented in Bar-Shalom et al. (2011).This implementation had three flight models: a nearly-constant velocity model with process noise standard deviation of 0.1 m/s 2 , a discrete Wiener acceleration model with process noise of 3 g, where g = 9.8 m/s 2 is the acceleration of gravity, and a constant acceleration model (zero process noise).These are similar to the models implemented in Bar-Shalom et al. (2001), Heidger and Mathias ( 2008) and Kirubarajan et al. (1998).As the state vector is described in Cartesian coordinates, standard conversion from polar to Cartesian was applied (Bar- Shalom et al. 2011).
The model transition probability matrix was fixed as:

simulAtion setup
For simulation, radar measurements were artificially generated by adding zero-mean white Gaussian noise to the trajectory depicted in Fig. 1.Missed detections observed in the real scenario were considered.
Radar measurements were given in polar coordinates, and noise samples were generated independently in range and azimuth (Bar- Shalom et al. 2011).
Figure 3 illustrates the simulation setup.It is similar to Fig. 2, differing only that measurements were artificially generated from aircraft GPS position points.Since no bias was introduced during measurement generation, there was no need for the IMMPDAF to perform bias correction.The estimates of noise standard deviation were obtained by using SASS-C as above described.where the lines and columns correspond, respectively, to the nearly-constant velocity, discrete Wiener acceleration and constant acceleration models.
Additional choices were the gate probability P G = 0.99 and probability of detection P D = 0.80.This value of gate probability is a common choice (Bar- Shalom et al. 2011), and detection probability of 80% is the minimum required for a primary surveillance radar according to the current Brazilian legislation (DECEA 2014).
The error measure adopted to evaluate the IMMPDAF algorithm was the difference between the aircraft reported position (GPS position) and the position estimate.For radar measurement error analysis, the difference between aircraft GPS position and radar measurements was used.
Since the aircraft GPS generated position reports in geocentric Cartesian coordinates, equations presented in Engel (2005) were applied to transform these reports to a 2-D coordinate system centered on the radar, referred to as local coordinate system.Geocentric aircraft reports were this way transformed to local Cartesian coordinates (ξ k GPS , η k GPS ) and local polar coordinates (ρ k GPS , θ k

GPS
).For the sake of notational simplicity, the superscript GPS will be suppressed in the following.

immpdAF perFormAnce with reAl And simulAted dAtA
To characterize the position error of the IMMPDAF algorithm at time k, differences in each coordinate were initially computed as: where Є ξ,k and Є η,k are the errors at time k in coordinates ξ and η, for the i th realization, and N ENS is the total number of realizations.
Figure 4 shows the results obtained for Є ξ,k and Є η,k with N ENS = 4,000 independent simulations.It can be observed in this figure that the mean of errors in both coordinates varies significantly with time.It is worth noticing that the most significant variations may be associated to intense maneuvers, since it has been recognized that Kalman filter innovations are not zero-mean during these maneuvers (Bar- Shalom et al. 2001).
The results obtained after averaging over 4,000 independent simulation runs are shown in Fig. 5, where it can be observed that the errors in coordinate η are more intense than those in ξ.This is explained by the fact that the flight was carried out mostly eastwards from the radar, where conversion of measurement noise from polar to Cartesian coordinates yielded higher variance in the η axis (Bar- Shalom et al. 2011).Besides, significant variations of the standard deviation in both coordinates are exhibited in this figure .In short, the estimates shown in Figs. 4 and 5 indicate that the IMMPDAF errors are highly non-stationary in both firstand second-order statistical moments.This is an important fact to be taken into account in the analysis of errors obtained with real data (radar measurements).It is important to remark that a natural difficulty in comparing performances using real and simulated measurements arises from the fact that real data is usually rare, so a limited number of samples are available and it is not possible to obtain reliable estimates of statistical moments such as those shown in Figs. 4 and 5.
The strategy here adopted to circumvent this difficulty was to verify, through a statistical analysis, if the samples of errors obtained in real conditions of radar operation can be considered as having the same order of magnitude of those obtained with simulated data.
To carry out such verification we first calculated the normalized zero-mean sample of errors obtained with simulated data as follows: and combined them to form a single normalized error measure q k defined as: For the sake of probabilistic modeling it was assumed that q ξ,k and q η,k are independent zero-mean Gaussian random variables with unit variance.Under this assumption, the normalized error amplitude q k has a Rayleigh probability density and mean √π/2.This is adopted here as a reference probability density for the error amplitude, denoted by p r (.).
So, given a tail probability ∏ arbitrarily set, we are able to evaluate an acceptance interval [0, λ] such that: Now, with the aim of comparing the IMMPDAF performance subjected to the two sources of measurements, we further define the variable q R , which is the normalized error measure obtained with real data.Accordingly, this variable is evaluated using Eq. 13 and error values obtained with real measurements in a similar way of q ξ,k and q η,k in Eqs.11 and 12, respectively.
The data obtained in 4,000 independent simulation runs and two values for tail probability ∏ (0.01 and 0.001, resulting in λ = 3.03 and 3.71, respectively) have been considered to plot Fig. 6, which shows the sample function of q R × k and the thresholds (λ) associated to the acceptance intervals.A large number of values of q R above these thresholds, i.e. outside the acceptance interval, may be observed.
On the other hand, a very different behavior is observed in Fig. 7, where samples of normalized errors obtained from four simulation runs are shown.The same thresholds included in the previous figures are used as references.It can be observed that in these cases the thresholds are rarely surpassed, which is an indication of the compliance with the assumed statistical framework.
A quantitative indication of the differences observed in the IMMPDAF performance under the two sources of measurements is given in Table 1, in which the percentages of instantaneous values of q R observed above several threshold (λ) values are presented, including the two thresholds shown in Fig. 6.It is shown both in Fig. 6 and Table 1 that the number of instantaneous samples of q R obtained with radar measurement data that surpass the chosen thresholds is considerably larger than what should be expected if the IMMPDAF errors with real radar data followed the reference probability distribution.
For further investigation and comparison of the IMMPDAF errors obtained from simulated and real radar data, the RMS position error along with a given (say the i th ) simulation run has been calculated as follows:  The average and standard deviation of the RMS error samples obtained after 4,000 simulation runs have been calculated and are denoted here as Є RMS and σ RMS , respectively.A comparison with Є RMS is shown in Table 2.It can be seen that Є RMS is 6.56 standard deviations greater than Є RMS , a significant difference of performance.To statistically quantify this difference, we regard the RMS error value as a random variable with first and second moments Є RMS and σ RMS , respectively.By using the Chebyshev's inequality, we find that the probability of observing values of RMS error above Є RMS is lower than 0.0232.
Therefore, this comparison of RMS errors leads us to observe once again that the IMMPDAF performance indexes obtained using real radar data seem not to be statistically consistent with those obtained from simulations.

noise AnAlysis
Radar measurement errors in distance and azimuth were obtained according to: Considering that the target trajectory was the same for the evaluation of both types of measurements, i.e. simulated and real, we may infer that the difference in IMMPDAF performance using real and simulated data cannot be caused by the target dynamics, but rather resides in the difference of noise characteristics in the two scenarios.
Since bias was compensated, the real measurement noise is assumed to be zero-mean.To further investigate its characteristics, Fig. 8 shows estimates of normalized autocorrelation and crosscorrelation functions for radar errors in distance (Є ρ,k ) and azimuth (Є θ,k ), obtained from a single realization of simulation and from real measurements.
Figure 8 shows that the estimates of autocorrelation functions generated from simulated measurements present typical white noise behavior (Figs.8a and 8c), as expected.It also confirms that simulated measurements in distance ( 16) (17) where: x k , F[M k ], z k and H[M k ] are, respectively, the state vector, state transition matrix, measurement vector and observation matrix at time k.M k represents the flight model in effect at time k and can be any element of the model-set M = {ψ j } j=1 , being r the number of models.The random sequences {υ [k, M k ]} and {w [k, M k ]} are Gaussian, zero-mean, white and mutually independent.At time k, υ [k, M k ] and w [k, M k ] are random vectors with covariance matrices Q[M k ] and R[M k ], respectively.Given M k , the values of F[M k ], H[M k ], Q[M k ] and R[M k ] are assumed to be known.The initial state x 0 is a Gaussian random vector with known mean and covariance matrix.Model transitions are Markovian with probabilities P{M k = ψ j | M k-1 = ψ i } = p ij .Both the model-set and transition probabilities p ij are assumed constant and known.

Figure 2 .
Figure 2. Performance evaluation with real data.

Figure 3 .
Figure 3. Performance evaluation with simulated data.

Figure 4 .
Figure 4.Estimated averages of IMMPDAF errors from simulation.

Figure 5 .
Figure 5.Estimated standard deviation of IMMPDAF errors from simulation.

Figure 6 .
Figure 6.Normalized amplitude of IMMPDAF errors obtained with real measurements.
the number of points in the trajectory.In a similar way, the RMS error along with the single sample function obtained with real data has been evaluated.It is denoted by Є RMS .

Figure 7 .
Figure 7.Samples of normalized error amplitude of IMMPDAF obtained with simulated measurements.
,k and z θ,k are radar measurements at time k in distance and azimuth, respectively.

table 1 .
Quantitative evaluation of samples outside acceptance intervals.