Constructing a meta-tracker using Dropout to imitate the behavior of an arbitrary black-box tracker

Abstract

Imitating the behaviors of an arbitrary visual tracking algorithm enables many higher level tasks such as tracker identification and efficient tracker-fusion. It is also useful for discovering the features essential in a black-box tracker or learning from several trackers to form a super-tracker. In this study, we propose a non-linear feature fusion framework, “MIMIC” that imitates many popular trackers by mixing a pool of heterogeneous features. The MIMIC framework consists of two subtasks, feature selection and feature weight tuning. These subtasks, however, tended to suffer from an overfitting problem when the number of videos available for training is limited. To address this issue, we incorporated Dropout algorithm into the training, which grants the trained MIMIC tracker a high degree of generalization. Extensive experiments testified the effectiveness of the proposed framework so that its applications would be promoted into different related tasks in visual tracking.

Introduction

Every year many off-the-shelf trackers are introduced to address various visual tracking challenges. Typically, those studies intend to handle a single or a few of these challenges: appearance changes (Babenko et al., 2009, Han and Davis, 2005, Henriques et al., 2012, Ross et al., 2008, Zhang et al., 2012, Zhong et al., 2012), abrupt/fast motion (Kwon and Lee, 2008, Zhou et al., 2012), target/non-target confusion (Dinh, Vo, & Medioni, 2011), long-term tracking (Kalal, Mikolajczyk, & Matas, 2012), background clutter (Avidan, 2007, Nguyen and Smeulders, 2004), irregular scale changes (Ning, Zhang, Zhang, & Wu, 2012), occlusions (Bao, Wu, Ling, & Ji, 2012) and moving cameras. However, large-scale benchmarks (Smeulders et al., 2015, Wu et al., 2015) suggested that TLD (Kalal et al., 2012), STRUCK (Hare, Saffari, & Torr, 2011), MIL (Babenko et al., 2009), FBT (Chu & Smeulders, 2012) and SCM (Zhong et al., 2012) have overall superior performance dealing with different challenges. Many of these trackers have solved the issues by which other methods are troubled. While mastering all of these domains to construct the holy-grail of trackers is difficult, fusing these trackers to integrate their merits into a unified framework has not been very successful (Martín & Martínez, 2014). The biggest obstacles to constructing such holy-grail trackers from currently realized ideas are the complexity of integrating mechanisms and contradictory objectives pursued by each of them.

In this study, we propose a framework, MIMIC, to imitate the behaviors of an arbitrary black-box tracker, with a relatively simple non-linear tracker benefiting from a pool of various features. This study is based on a premise that a linear combination of sufficiently expressive features along with a flexible but still simple non-linear observation model can roughly approximate the behaviors of many popular trackers.

Employing multiple features to track objects has been investigated considerably in visual tracking literature. Different frameworks like particle filters accommodate the feature fusion seamlessly (Perez, Vermaak, & Blake, 2004). Although automatic adjustment of the feature’s weights has been focused in some studies (Chau, Thonnat, Bremond, & Corvee, 2014), many studies opt to select an effective subset of implemented features to track with, yet the performance drastically depends on the way they approach to feature extraction and selection. Different features may be obtained by applying various pre-defined templates on a single image patch (Shi and Tomasi, 1994, Viola and Jones, 2004), be constructed by applying a particular function (e.g., scale) on a single image patch with different parameters (Kwon & Lee, 2010), or a “feature pool” may consist of several heterogeneous features (Chen, Liu, & Fuh, 2004).

Each employed feature lends its characteristics to the tracker so that the collective behaviors of the tracker are affected directly by its active features and/or their corresponding weights. Some features have prominent effects on the tracker, which grants it a certain degree of invariance against environmental changes. For instance, incremental PCA brings IVT (Ross et al., 2008) illumination invariance and sparse-coding-based features render the tracker robust against partial occlusions (Bao et al., 2012). Thus, it is natural to attribute the behaviors of a specific tracker to its employed features. Moreover, complicated trackers behave in a way that seems to be closely emulated using several features fused together.

The behaviors of many existing trackers can be imitated, by selecting the right set of features and/or proper weights for them. Without proper features, the observation model does not deal with the occlusion or sudden illumination changes, for instance, hence fails to track the objects. In this study, we propose a unified framework to emulate the behaviors of these trackers by a proper mixture of feature responses. The features when combined in a tracking-by-detection tracker (such as Ensemble tracker   Avidan, 2007) or in an optimization framework (such as Mean-shift tracker   Comaniciu, Ramesh, & Meer, 2003) lose the flexibility to mimic other trackers. Imitating trackers’ behavior, while they are dealing with different tracking challenges such as occlusions and illumination variations, complicates the imitation problem even more. Such challenges are managed by a combination of specialized features and stochastic sampling in some trackers (e.g., in IVT   Ross et al., 2008, MIL   Babenko et al., 2009 or L1APG   Bao et al., 2012), so deterministic feature fusion methods do not work well on imitating their behaviors. Sophisticated trackers (e.g.,  SCM   Zhong et al., 2012), on the other hand, cannot be approximated with linear motion models or dense sampling. Linear observation models (e.g., in Kalman Filter tracker   Čehovin, Kristan, & Leonardis, 2011) cannot emulate the non-stationarity of the scene observation, and moreover, a more complicated sampling and feature fusion are required to imitate highly adaptive trackers such as STRUCK (Hare et al., 2011). Dense sampling in effect acts like the sliding window scheme in tracking-by-detection methods, hence cannot reproduce well the behaviors of probabilistic trackers. A non-linear non-Gaussian probabilistic framework such as the multi-cue Particle Filter tracker (Perez et al., 2004), is a natural choice to address these issues.

To validate our hypothesis–i.e.,  a linear combination of proper features in a non-linear observation model emulates most of the complicated trackers–a rich pool of features is provided to a particle filter tracker with a few hundreds of particles and a random walk motion model. The goal of this tracker is to imitate the behaviors of a specific target tracker closely, in the sense of tracking overlap on the unseen test videos. To achieve this, our tracker adjusts the weights of the features of the feature pool, based on the outputs of the target tracker while it operates in several videos used as the training data.

Our tracker, MIMIC, extracts a variety of features from the image frame and weighs them in the observation model in a way to minimize the mismatch between its estimation of the object location and that estimated by the target tracker. These weights constitute the parameters to be tuned through supervised learning. Considering the high non-linearity of the objective function representing the mismatch to be minimized, this supervised learning was performed by an evolutionary algorithm which searches the high-dimensional parameter space for the best combination of feature weights. To select a subset of effective features from the feature pool, the optimization should set the weights of many unnecessary features to zero, which resembles the feature selection in effect. This is done by imposing an L1 regularization term to the objective function.

To suppress over-fitting in this high-dimensional learning task, we employed “Dropout”, as the success of this regularization mechanism has been demonstrated in artificial neural networks (Hinton, Srivastava, Krizhevsky, Sutskever, & Salakhutdinov, 2012). By emphasizing on optimization over the model ensemble, this algorithm investigates different combinations of the features and prevents over-fitting by approximating the training dataset by means of a virtual ensemble of the models employing a variety of feature subsets. The proper combination of the L1 regularization and the Dropout algorithm is expected to work well in the simultaneous task of feature selection and feature weighting, by avoiding over-fitting even when the number of available videos is limited (Fig. 1).

In summary, our contributions in this study are two-fold:

• We proposed a unified framework to approximate the behaviors of an arbitrary tracker by employing a weighted subset of features selected from a feature pool in the observation model of our MIMIC tracker.

• We introduced several applications of this framework including “tracker identification” in which a black-box tracker is identified among other trackers based on its tracking results, and also a couple of novel approaches to performing tracker fusion.

Following this introduction, Section  2 elaborates the proposed framework, and Section  3 demonstrates the wide range of applications of this framework. After the proposed framework is discussed in Section  4, our article is concluded in Section  5 with the outcome of this study and some future directions to improve it.