Exclusive Association Sampling to Improve Bayesian Multi-Target Tracking

Multi-target tracking has been studied for many years, yet it remains a challenging problem, particularly in terms of implementing data association when tracking targets over several time steps. To achieve robustness, probabilistic approaches have been proposed, including Bayesian multi-target tracking methods. However, these approaches involve high calculation costs, which are incompatible with real-time applications. We propose exclusive association sampling (EAS) to improve the efficiency of Bayesian multi-target tracking methods. Although EAS is a simple procedure composed of random sorting and sampling based on the observation probability, it can be employed to increase the efficiency of association candidate generation and the calculation of statistical values. In this study, we proposed two Bayesian multi-target tracking methods based on EAS: stochastic joint probabilistic data association (SJPDA) and Rao-Blackwellized particle filter with EAS (RBPF with EAS). Evaluation of these methods with simulated data shows that integrating EAS into these methods can enhance their speed and accuracy. Moreover, evaluation on open datasets used for pedestrian tracking on camera sequences shows that the proposed methods achieve significantly better performance on some important metrics compared with representative methods.


I. INTRODUCTION
Multi-target tracking has been studied for many years in the context of various applications using sensors, such as pedestrian tracking using cameras or LIDAR. The main barrier to successful multi-target tracking has been the management of data associations between the tracked targets and the observations. Various approaches to achieving robustness have been proposed. Bayesian multi-target tracking methods, such as joint probabilistic data association (JPDA) [1] and multiple hypothesis tracking (MHT) [2], are classic and successful techniques. These approaches have introduced multiple candidates or probability-weighted association to achieve robustness. Unfortunately, the calculation cost of Bayesian multi-target tracking is high; therefore, its application is difficult when tracking a large number of targets.
Recently, some vision studies have revisited traditional Bayesian filtering-based approaches and enhanced them using recently developed fast optimization techniques [3], [4]. These methods have achieved performance comparable with that of state-of-the-art approaches for pedestrian The associate editor coordinating the review of this manuscript and approving it for publication was Wei Wang . tracking with single cameras. Nevertheless, the calculation cost remains high when applying them in real-time applications. The Bayesian filtering-based multi-target tracking approach has many advantages because it can be performed as a sequential method and can integrate a wide variety of data types, such as motion and appearance data. Furthermore, the robustness of Bayesian multi-target tracking is critical for sensors such as radar sensors [5], which do not provide as many information about targets as a camera does. Therefore, improving the calculation speed and accuracy of the Bayesian filtering-based method would represent a valuable advance in multi-target tracking.
In this study, we propose a novel exclusive and probabilistic sampling method for sampling data associations to improve the performance of various Bayesian filtering-based multi-target tracking methods. This sampling method, called exclusive association sampling (EAS), is composed of the random sorting of observations and sampling based on the observation probability. Despite its simplicity, EAS can approximate the joint probabilities of observations by applying appropriate weights. The samples generated by EAS are therefore useful for approximating probabilistic variables and generating associations for hypotheses in Bayesian FIGURE 1. How EAS works. We propose EAS, which can be applied to the data association problem to generate several candidates for association. EAS provides efficient samples of associations for calculating the weighted association for JPDA or for associating samples in the RBPF.
multi-target tracking, as a result they can improve the tracking efficiency (see Fig. 1).
We integrate the proposed method with two Bayesian filtering-based multi-target tracking methods: JPDA [1] and Rao-Blackwellized particle filtering (RBPF) [6]. The JPDA-based method, called stochastic JPDA (SJPDA), applies EAS to approximate joint probabilistic scores, and the simulation results show that this is achieved faster than with a recently developed improved JPDA variant [3]. The RBPF-based method, called RBPF with EAS, utilizes EAS to generate data associations for individual particles, and it achieves considerably higher accuracy than the conventional RBPF method [6]. We evaluate the proposed methods using open datasets on pedestrian tracking with a single camera. The results demonstrate that the proposed methods are considerably better than the original methods and that they achieve better performance on some important metrics than representative methods.
The remainder of this paper is organized as follows. In Section II, we review work related to this study. In Section III, we explain the data association problem. In Section IV, we explain EAS for improving Bayesian multi-target tracking. In Section V, we propose two Bayesian multi-target tracking methods using EAS. The experimental results are presented in Section VI. Finally, we conclude this paper in Section VII.

A. BAYESIAN MULTI-TARGET TRACKING IN EARLY TIME
The primitive approach to multi-target tracking involves the use of an optimization method to implement data associations within one-step observations. This association process can be modeled as an assignment problem and solved using a combinatorial optimization algorithm, such as the Hungarian method [7]. Each target is tracked using a Bayesian filter, e.g., a Kalman [8] or particle filter [9]. However, the one-step optimization approach is not robust because the association with the highest joint probability is not always correct. To achieve robustness, various probabilistic approaches that use multiple candidates for data associations or probability-weighted data associations have been proposed based on Bayesian filtering.
Multiple hypothesis tracking (MHT) [2], which is the most successful multiple-candidate approach, creates a tree of tracking hypotheses for each candidate target, with individual branches representing respective data association candidates. The solution is obtained by finding the most likely combination of tracks without conflicts. As MHT is essentially a breadth-first search algorithm, its performance strongly depends on its ability to prune branches within the search tree.
JPDA [1] is the most successful approach based on probability-weighted data association. It performs probabilistic data association by calculating joint probabilistic scores. However, naïve JPDA suffers from high combinatorial complexity because it considers all possible assignments of observations to targets when calculating the joint probabilistic scores. Correspondingly, as the number of targets increases, the technique becomes intractable in nearly all practical applications. To alleviate the computational burden, some heuristic approximations for JPDA have been proposed [10], [11]. Nevertheless, these methods sacrifice tracking accuracy to make the algorithms computationally tractable.
RBPF [6] is a multiple candidate-based approach in which candidates are generated for each data association set instead of for each candidate target, as in the case of MHT. However, in situations with high numbers of cluttered detections, the RBPF sampling approach does not perform well because it does not consider data association exclusivity.
Certain types of sensors (such as radar and LIDAR) provide no information to identify objects beyond their positions. This has led to the development of tracking methods that do not distinguish individual objects, such as filtering methods based on random set theory [12], e.g., Gaussian mixture probability hypothesis density (GMPHD) filters [13], which estimate the states of multiple targets as a probability density. Because these methods do not provide IDs for individual targets, they are not suited to track individual targets.

B. RELATED STUDIES IN COMPUTER VISION
Multiple target tracking has also been extensively studied in the context of computer vision. Successful approaches are based on offline optimization [14]- [17], which follows a different strategy from the Bayesian filtering-based approaches described above. Sequences of frames are grouped together and the states and data associations of all targets are jointly inferred by optimizing a predefined objective. These approaches introduce respective assumptions to simplify conditional dependences and reduce complexity through the use of, for example, the min-cost flow [14]- [16] or k-shortest paths algorithm [17]. Improvement of optimization techniques and the usefulness of exploiting appearance models have led to a revisiting of Bayesian filtering methods, as exemplified by Kim et al. [4], who applied MHT to develop an appearance model. The Bayesian filtering framework can easily integrate different models, such as motion and appearance models, suggesting that a classical MHT implementation has come surprisingly close to the performance of state-of-the-art methods. Rezatofighi et al. [3] applied JPDA as an approximation, using the m-best solutions (JPDA m ), which calculates joint probabilistic scores from the m-best assignments instead of from all assignments. Furthermore, a recently developed integer linear programming algorithm [18] can quickly provide the m-best solutions and has been applied to significantly reduce the JPDA calculation time. Our SJPDA was inspired by JPDA m [3] and demonstrates improved calculation speed by introducing a sampling strategy, rather than calculating the m-best solutions. Random-finite-set-based approaches have also been studied, as they are Bayesian filterings for multiple object tracking that can operate in real time without any special modifications. In paticular, some GMPHD-based approaches have been proposed [19]- [22]. However, GMPHD-based approaches suffer from many ID switches, as they track probability density instead of individual objects. Almost all of these studies employ appearance information to identify individual objects. Some of these methods [20], [21] are evaluated in Sec. VI.
More recently, deep learning approaches using the appearance information have been intensively studied [23]- [25]. Milan et al. [23] proposed recurrent neural network based approach. Leal-Taixé et al. [24] employ a Siamese CNN to learn local appearance features to associate targets. Sadeghian et al. [25] propose an appearance model using the LSTM, which takes images in the tracklet step-by-step and predicts the similarity score. However, these approaches are highly dependent on appearance information; therefore they cannot be applied to sensors that do not provide appearance information about the target.

III. DATA ASSOCIATION PROBLEM
We start with a discussion of the data association problem, which is the main problem in multi-target tracking. Data association is the task of associating multiple observations with multiple estimated tracks. The data association problem is illustrated schematically in Fig. 2, where x i,k is the state of the i-th track at time step k, and y j,k is the j-th observation at time step k.
The connection matrix θ k between the M states x k = {x 1,k , · · · , x M ,k } and the N observations y k = {y 1,k , · · · , y N ,k } is unknown and must be estimated at each step. This process is called data association. The primitive approach to connection estimation involves optimization of the joint observation probability p(y k |x k , θ k ). The Hungarian method [7] is the most popular optimization method for tackling this problem. However, this one-best approach is not very robust because the association with the highest joint probability at each step is not always correct for the time-series observation. More robust probabilistic approaches introduce multiple candidates for data associations or probability-weighted data associations. Here, EAS is a technique for improving the efficiency of such probabilistic data associations.

IV. EXCLUSIVE ASSOCIATION SAMPLING
In this section, we present a novel exclusive and probabilistic sampling method for data association, called exclusive association sampling (EAS). In this context, exclusivity refers to a strict one-to-one correspondence between observations and targets: each target is associated with only one observation, and each observation is associated with only one target. The EAS is not an association algorithm; rather, it is an algorithm that is used for generating association samples. The main difference from the Hungarian method is the generation of weighted association samples according to the joint probability. EAS is not a stand-alone association algorithm; rather, it is a module used to improve Bayesian multi-target tracking methods such as JPDA and RBPF.
The EAS process applies the following procedure during each time-step k (with y j,k written as y j ): 1) Randomly sort observations {y 1 , · · · , y N }.
2) Associate the first observation y j with the target x i based on the probability p(y j |x i ). 3) Exclude the associated target from subsequent associations. 4) Repeat steps 2 and 3 for all observations.
Using this procedure, a sample θ of the associations is obtained. The procedure is illustrated schematically in Fig. 3.
EAS can approximate the joint probability p(y|x, θ) by introducing an appropriate weight w θ for each sample. This joint probability p(y|x, θ) is written as follows: where θ ij represents element i,j of the association matrix θ. If x i and y j are associated, then θ ij = 1. Otherwise, θ ij = 0. The sampling probability q(θ) is determined through the following procedure. First, an order of observations is generated from the N ! possible orders by random sorting (see Fig. 3(b)). Then, N -times sampling is executed. The sampling probability q(θ) is therefore: where χ j is the candidate set of targets that are available for association, and | · | denotes the number of elements. d j is the j-th index of the random sorted observations. We introduce the weight w θ to approximate the joint probability p(y|x, θ): where · denotes multiplication. This equation can be transformed as follows: where θ id j is 1 at only one i in i = 1, · · · , |χ j |, otherwise 0. Therefore (4) can be transformed as follows: When θ is fixed, an observation j and its corresponding target i are always included in χ j ; therefore, the following relation holds: Substituting (6) into (5), the suitable weight w θ can be obtained as follows: By introducing the weight w θ , EAS can approximate the joint probability p(y|x, θ). The complete algorithm for EAS is presented in Algorithm 1. The computational complexity of EAS is O (MN ). However, because multiple sampling is generally necessary for applying multi-target tracking, the time complexity becomes O(SMN ), where S is the number of samples. EAS can be calculated on an order close to that of the Hungarian method. When the number of targets and observations exceeds the number of samples, EAS is faster than the Hungarian method. In this algorithm, false detection and new targets are not considered. To address false detection and new targets, it is necessary to introduce corresponding dummy targets. Because the exclusion rule does not apply to a dummy target, we do not exclude it from the available set χ, even if an observation has already been assigned to it.

V. MULTI-TARGET TRACKING METHODS USING EAS
In this section, we propose applications of EAS to two multi-target tracking methods: stochastic JPDA and RBPF with EAS.
A. STOCHASTIC JPDA Stochastic JPDA (SJPDA) is a JPDA approximation method. The original JPDA is a multi-target tracking method using probability-weighted associations based on joint probabilities, which are calculated for each available association. The probabilistic distribution of the i-th target p(x i,k |y 1:k ), where y 1:k denotes {y 1 , · · · y k }, is calculated by updating the previous distribution p(x i |y 1:k−1 ) using the current observation y k : where p(x i,k |y 1:k−1 y j,k ) is the posterior distribution using an associated observation y j,k , which can be calculated using a Kalman filter. The association probability p(θ ij ) is calculated θ id j ← 1 14: w θ ← w θ · p 15: Remove x i from χ 16: end for 17: where is the set of all associations. p(x i,k |y 1:k ) can be calculated by adding up the posterior distributions p(x i,k |y 1:k−1 , y j,k ) calculated using each observation with the associated probability p(θ ij ).
However, p(θ ij ) is difficult to calculate because doing so requires calculations for all associations. As the number of targets or observations increases, the number of associations becomes explosively large, making the calculation of p(θ ij ) practically impossible. Recently, Rezatofighi et al. [3] proposed JPDA m , an approximation of JPDA that uses the m-best solutions calculated by an integer linear program. In the technique presented in this paper, JPDA is approximated by using the expected value calculated by EAS instead of the m-best solutions: where EAS is the set of samples of the association given by EAS. By introducing EAS, we can approximate the association probability p(θ ij ) using any number of samples of associations.

B. RBPF WITH EAS
Another proposed multi-target tracking method using EAS is the RBPF-based method. RBPF is a particle filter method that tracks states using multiple hypotheses of associations. Although RBPF-based multi-target tracking was previously proposed by Särkkä et al. [6], their sampling method does not consider the exclusivity of associations, resulting in some sampled associations being invalid as two or more observations are associated with one target. EAS efficiently removes such invalid associations.
In the EAS-based RBPF, the probability p(x k |y 1:k ) is represented as follows: where S is the number of particles and p(x (s) k |y 1:k ) is the s-th particle of the RBPF. Note that each particle contains all tracked states in the Kalman filter form. EAS is applied to update each particle as follows: where θ (s) is a sample of associations produced by EAS. This sample of associations is generated for each particle and is used to update all of the tracked states for that particle. The operation of RBPF with EAS is illustrated schematically in Fig. 4, where x (s) k represents the s-th particle at time-step k, which includes all tracked states. In the prediction step, tracked states in each particle are predicted by the system model. The data associations are then constructed through sampling, and each state is updated by the associated observation. The proposed method employs EAS to sample the associations and updates each tracked state based on the EAS-associated observations, with the probability of each particle updated using the weight produced by the EAS algorithm. Finally, resampling is performed to proceed to the next step.

VI. EXPERIMENTAL RESULTS
We evaluated the two proposed multi-target tracking methods first on simulated data and then on real camera sequences. Specifically, we used single camera sequences from open datasets [26]- [28] to evaluate the methods for pedestrian tracking.

A. EVALUATION ON SIMULATIONS
To evaluate the speed and accuracy of the proposed tracking methods, we applied them to a simulated scenario, in which three moving targets cross in front of each other. The simulated state space model is given as where τ = 0.1 is the sampling interval, u k is the system error, and v k is the observation error.  In the evaluation, the initial i-th estimated state waŝ To evaluate the performance of the two proposed methods, we compared them with several existing methods, namely the Hungarian method, original JPDA [1], JPDA m [3], and the original RBPF [6]. The JPDA m method was applied as an online method, in which the branch and bound algorithm was employed to calculate the m-best solutions. However, if outputting the results online, the accuracy of both RBPF and RBPF with EAS would be reduced because of the loss of particle continuity. Therefore, we processed the results offline by particle smoothing, which traces particles backwards from the last maximum likelihood particle. All of the algorithms (JPDA m , SJPDA, RBPF, and RPBF with EAS) used 100 candidates. The observation probability p(y j |x i ) was modeled as follows: where V is the volume of the detection area, which in this case was set to V = 10. Because false detections are introduced in this evaluation, but the number of targets does not change, we introduced only one dummy target for false detections x 0 . The evaluation was performed by a single core of a computer with an Intel Core i7-4770K CPU running at 3.50 GHz with 32 GB of RAM. A sample of detections and tracking results are shown in Fig.5, where it can be observed that the Hungarian method and RBPF produced numerous missed tracks as a result of the large number of cluttered detections. The Hungarian method tended to switch targets, while the RBPF tracked noise. In contrast, JPDA, JPDA m , and the two proposed methods were all able to track targets in the presence of cluttered detections.
We compared the run times and accuracy of the target positions; the results are listed in Table 1, from which it can be observed that both of the proposed methods achieved significantly improved performance.
SJPDA was faster than and almost as accurate as JPDA m . Although both methods are approximations of JPDA, they are more accurate than the original JPDA. This may be because these methods exclude improbable associations during the approximation process. Even though RBPF with EAS had a similar run time as conventional RBPF, its accuracy was considerably higher. These results suggest that applying exclusivity to the data associations improved the robustness of RBPF against cluttered detection. The accuracy of both proposed methods was also considerably higher than that of the Hungarian method. The JPDA and RBPF perform probability-weighted and multi-candidate-based tracking, which provides the robustness with crossed targets. The EAS can improve the efficiency of the JPDA and RBPF. This causes the increase in speed of the JPDA, and the improvement of the accuracy of the RBPF with the same number of candidates. Although JPDA was originally more accurate than the Hungarian method, it required more calculation time, however the EAS improved the calculation speed. The improved RBPF achieves higher accuracy than that of the Hungarian method. Overall, the results validate the proposed approach in terms of its ability to enhance the performance of Bayesian multi-target tracking methods.

B. EVALUATION ON OPEN DATASET OF PEDESTRIAN TRACKING WITH A SINGLE CAMERA 1) MOT16 DATASET
Next, we evaluated the proposed methods using real-world data by applying them to the MOT Challenge, a recently developed multiple object tracking (MOT) benchmark [26]. The MOT Challenge provides various video sequences and detections of pedestrians. Only trackers can be evaluated using these detections. We employed the MOT 16 datasets [29] and conducted an evaluation based on the popular CLEAR MOT performance metrics of the MOT Challenge [30]. These include multiple-object tracking accuracy (MOTA), which combines errors such as false positives (FP), false negatives (FN), and identity switches (ID Sw.) into a single number. Next, the multiple-object tracking precision (MOTP) is the overlapped ratio between the tracking result and ground truth rectangles. In addition, the mostly tracked (MT) and mostly lost (ML) scores [31] represent how many targets are tracked for more than 80% and less than 20% of their life-spans, respectively, based on ground-truth trajectories.
Neither SJPDA nor RBPF with EAS includes a mechanism for tracking a time-varying number of targets. We therefore introduced a special assignment to generate new targets and an existence probability that increases when an observation is assigned and decreases at the prediction step, as follows: where p(e i,k ) is the existence probability of the i-th target; p d is the decrease in the existence probability in one step and is set as 4.0 t, where t is a sampling interval; and p p is the detection precision, which is set as 0.8. Targets that have an existence probability below 0.1 after prediction are removed. The observation probability is modeled to be dependent on the existence probability and the detection precision as follows: where V is defined by the size of the camera image. The MOT challenge datasets contain low-confidence detections. We utilized detections with a higher confidence than the threshold value c = 0.0.
The MOT 16 evaluation results are listed in Table 2. In this study, the focus is only on position tracking and appearance is not considered. Representative trackers (as of 2020/06/04) that do not use appearance are shown at the top. The two proposed methods are compared with the original methods in the bottom half. SJPDA and RBPF with EAS are confirmed to have higher performance than the original methods. SJPDA is considerably faster than JPDA m and performs better on the main metric, MOTA. Although this is partly due to the different parameter settings, SJPDA achieves a higher recall rate than JPDA m , although it has more false positives than JPDA m . In contrast, JPDA m has a low false positive rate but VOLUME 8, 2020   [29]. The tracking performances of representative trackers that do not use the appearance are shown at the top. The performances of the two proposed methods and the conventional methods are listed in the lower half. The proposed methods achieved better performance than the conventional methods. Moreover, SJPDA is significantly faster than the representative methods and has comparable accuracy, and the MT and ML of RBPF with EAS are the best among the considered methods.

TABLE 3.
Tracking performance evaluated using the PETS2009-S2L2 [27]. The tracking performances of representative trackers that do not use the appearance are shown at the top. The performances of the two proposed methods and the conventional methods are listed in the lower half.

TABLE 4.
Tracking performance evaluated using the TUD-Crossing [28]. The tracking performances of representative trackers that do not use appearance are shown at the top. The performances of the two proposed methods and the conventional methods are presented in the lower half. a high number of missed detections. SJPDA is also faster than all the other methods and performs as well as the representative methods. RBPF with EAS also performs better than the original RBPF in terms of the MOTA and achieves a higher recall rate than the original RBPF. These results suggest that the exclusive association is effective for improving the performance on these metrics. Furthermore, the MT and ML of RBPF with EAS are the best among the considered methods. RBPF with EAS requires offline processing, and the computation time is not fast. However, it performs well for continuous tracking. These results confirm that EAS can improve Bayesian multi-target tracking for real scenes and that the proposed methods achieve performances comparable to or better than the representative methods. The result shows that although the proposed methods have heuristic procedures concerning false detections and new targets, they work well in a real situation with false alarms, miss-detection, and new targets. In paticular, the proposed method is superior than a random-finite-set-based approach, GMPHDN1T [20], which suffers from many ID switches.
Images of the tracking results obtained using SJPDA and RBPF with EAS are presented in Fig. 7. Successful tracking of pedestrians is shown in the same color frame. To some extent, both methods can track pedestrians in image sequences without any appearance information. Because the proposed methods are based on Bayesian filtering, they can exploit higher-order information such as appearance information, like MHT-DAM [4]. To create an application that focuses on tracking images, such enhancements should be included to improve the performance.

2) PETS2009 AND TUD DATASET
We also evaluated the proposed methods using the PETS2009 [27] and TUD [28] benchmarks. These datasets are included VOLUME 8, 2020 in the 2DMOT15 [35] benchmark, which is a previous benchmark of MOT16 and provides detections to evaluate the pure tracking performance. 2DMOT15 contains the same sequences as MOT16; however, the data, PETS2009-S2L2 and TUD-Crossing, are not included in MOT16. We apply the proposed method by using the detection of 2DMOT15 and evaluate its performance using the ground-truth data provided in [36]. The results of PETS2009-S2L2 and TUD-Crossing are listed in Tables 3 and 4, respectively. Representative trackers (as of 2020/09/25) that do not use appearance are presented at the top. The two proposed methods are compared with the conventional methods in the bottom half. The performance results of competitors are also calculated from the raw results of 2DMOT15 provided in [26] and the ground-truth data provided in [36]. These results do not contain speed information; therefore, only the speed of the proposed methods and the RBPF are evaluated. The MOTA of SJPDA is more accurate than that of JPDA m , and most of the other metrics of SJPDA are also better than those of JPDA. Although the speed of JPDA m has not been evaluated in these datasets, SJPDA can be expected to be faster than JPDAm based on previous results. The RBPF with EAS achieves better performance than the original RBPF on most metrics. In addition, the proposed methods achieved as good a performance as the representative methods. SJPDA achieves the highest MOTA on PETS2009-S2L2. RBPF with EAS achieves the best MT and ML on both datasets. In these datasets, GMPHD_OGM [21], which is a GMPHD-based tracking method, achieves good performance. This may be due to the fact that this method uses information about overlaps of detection boxes to improve accuracy.
Images of the tracking results obtained using SJPDA and RBPF with EAS are presented in Fig. 7. Successful tracking of pedestrians is shown in the same color frame. Both methods can track pedestrians in image sequences without any appearance information. These results reveal that the SJPDA has more misdetections, and RBPF has more false positives, which may be responsible for the differences in accuracy in each dataset. In these experiments, we employed the same parameters for both methods to ensure a fair evaluation; however, in practice, it is better to choose the appropriate parameter for each method separately.

VII. CONCLUSION
We presented a novel sampling method to enhance probabilistic data association methods. The proposed method, called EAS, applies probabilistic and exclusive sampling of data associations. EAS can approximate joint probabilities through the application of easily obtained weights. Based on the EAS scheme, we proposed two multi-target tracking methods: SJPDA and RBPF with EAS. The simulation results demonstrate that SJPDA is faster than JPDA m [3], while RBPF with EAS achieves considerably higher accuracy than the conventional method [6]. Furthermore, the results of the evaluation using open datasets on pedestrian tracking show that the proposed methods using EAS achieve higher performance than the original methods. The proposed methods achieved significantly better performance than the representative methods on some important metrics.