Object fusion tracking based on visible and infrared images: A comprehensive review

Highlights

• A review of fusion tracking methods via visible and infrared images is presented.

• Main RGB-infrared trackers are summarized and categorized into several groups.

• Public RGB-infrared datasets are summarized and compared.

• Main results on public datasets are summarized and analyzed in detail.

• Future prospects of RGB-infrared fusion tracking are discussed and suggested.

Abstract

Visual object tracking has attracted widespread interests recently. Due to the complementary features provided by visible and infrared images, fusion tracking based on visible and infrared images can boost the tracking performance under adverse challenging conditions. RGB-infrared fusion tracking has become an active research topic and various algorithms have been proposed in recent years. In this paper, we present a review on RGB-infrared fusion tracking. We summarize all major RGB-infrared trackers in the literature and categorize them into several major groups for better understanding. We also discuss the development of RGB-infrared datasets, and analyze the main results on public datasets. We observe that deep learning-based methodsachieve the state-of-the-art performances. Besides, the graph-based and correlation filter-based methods give a bit worse but still competitive performances. In conclusion, we give some suggestions on future research directions of fusion tracking based on our observations. This review can serve as a reference for researchers in RGB-infrared fusion tracking, image fusion, and related fields.

Introduction

Visual object tracking has received significant attention in recent years due to its wide applications in many areas, such as robotics [1], autonomous vehicles [2], human-computer interface [3] and video surveillance [4]. According to the type of images included, it can be roughly classified into tracking based on visible images, tracking based on infrared images, and RGB-infrared fusion tracking. Among these types, the most popular is tracking based on visible images. Note that in this paper we do not distinguish between visible and RGB (Red-Green-Blue) images, although the visible images also contain gray-scale images.

Currently, two main kinds of methods in visual object tracking are deep learning (DL)-based methods [5] and correlation filter (CF)-based approaches [6]. Tracking methods based on deep learning mainly utilize its strong feature representation ability to extract better features than handcrafted ones, thus these approaches can achieve good tracking results in many cases. Here handcrafted ones means the features that are designed manually, such as histogram of oriented gradients (HOG) [7] and scale invariant feature transform (SIFT) [8]. Previously, deep learning-based methods suffer from slow speed severely [9]. However, with the application of fully convolutional Siamese networks in tracking [5], recent deep learning-based trackers achieve high performance tracking results while maintaining real-time speeds [10], [11], [12]. In CF-based tracking algorithms, the model can be updated in real-time as the correlation operation can be efficiently implemented via the Fast Fourier Transform (FFT). Therefore, during tracking process, CF-based methods utilizing shallow features can run in real-time. However, recently some CF-based trackers use raw deep convolutional features which are of high dimensionality [13], [14], [15]. These trackers become slower and slower because the computational time for the correlation filters increases with the feature dimensionality [16].

However, due to the limitation of the imaging mechanism of visible images, tracking algorithms based on visible images may fail as they may be unreliable in certain circumstances. For example, when the illumination conditions are poor or change significantly. The infrared images detect thermal information of objects and are insensitive to these factors. They can provide complementary information to visible images, as shown in Fig. 1a. In recent years, researchers also explore performing object tracking with infrared images [18], [19], [20], [21]. However, the infrared images typically have low resolutions and poor textures, and are also unreliable in certain conditions as shown in Fig. 1b. Therefore, more researchers begin to investigate object tracking method based on the fusion of visible and infrared images to overcome the inherent shortcomings of the methods based on single-modal images. By fusing complementary information from visible and infrared images, the robustness of tracking algorithms can be greatly enhanced. As a result, in recent years, object tracking based on RGB and infrared images have become a hot research topic. An increasing number of researches have been published in high quality journals or well-known conferences [22], [23], [24], [25], [26], [27], [28], [29], [30]. As a consequence, the well-known visual object tracking challenge (VOT) started a new RGB-infrared subchallenge in 2019¹, aiming to attract researchers to evaluate the performances on provided video sequences. Note that since the appearance of tracking based on visible and infrared images, it did not have a consistent name. A large part of researchers used fusion tracking [31], [32] or tracking by fusion [33], [34], [35]. It was until 2017 that some researchers started using RGBT tracking [27]. In this review, we denote the object tracking based on the fusion of visible and infrared images as RGB-infrared fusion tracking, because we think it can cover this kind of methods better and is thus more suitable for a comprehensive review. Besides, by using this name we aim to emphasize the importance of fusion in this kind of methods.

The research on RGB-infrared fusion tracking has begun in 2000s, as indicated by the development timeline of this field given in Fig. 2. RGB-infrared fusion tracking can be categorized into different categories.  According to the primary modality utilized during fusion tracking, there are infrared-assisted RGB tracking and RGB-assisted infrared tracking. In infrared-assisted RGB tracking, visible image is the primary modality. Infrared images are employed for assisting RGB tracking, especially when the visible images are not reliable [36], [37]. In these works, the evaluation metrics are evaluated based on the ground truth of visible images. In contrast, in RGB-assisted infrared tracking, infrared image is the primary modality and all evaluation metrics need to be computed based on the infrared ground truth [38]. In this paper, we broadly divide the RGB-infrared fusion tracking methods into five categories according to their adopted theories, namely traditional methods, sparse representation (SR)-based, graph-based, correlation filter-based and deep learning-based approaches. It is well known that effective and robust feature representation is crucial for tracking algorithms. Before sparse representation-, graph-, correlation filter- and deep learning-based methods, researchers performed fusion tracking using traditional techniques such as mean shift, Camshift, Kalman filter, and particle filter. Traditional methods utilize handcrafted features to represent the target. Sparse representation-based methods work on the basis of possible representation of the target with linear combinations of bases in overcomplete dictionaries. Graph-based approaches firstly divide the bounding box around the target to non-overlapping patches, and then build the relationship among these patches to work out a feature representation of the target. CF-based trackers learn correlation filters online efficiently to adapt to variation of the target. Deep learning-based methods leverage the strong feature representation ability of deep neural networks to learn robust feature representation of the target from a large amount of images. In all these methods, a key point of achieving good fusion tracking performance is the effective combination of visible and infrared features.

As can be seen from Fig. 2, RGB-infrared fusion tracking is developing very fast. However, to the best of our knowledge, there is a lack of review on RGB-infrared fusion tracking in the literature that gives a comparison and evaluates the performance of these different techniques. This paper tries to fill this gap. The main contributions of this review are in several aspects. First, to the best of our knowledge, this is the first review on RGB-infrared fusion tracking. This manuscript systematically investigates the RGB-infrared fusion tracking methods, benchmark datasets, and evaluation metrics. Main RGB-infrared tracking algorithms are grouped into several types according to their corresponding theories and each kind is introduced in detail, including the main principles, representative methods as well as pros and cons. Second, main results on public datasets are presented and analyzed in this review to provide an objective comparison of the existing approaches. Third, based on the systematically review of main RGB-infrared fusion tracking methods and the performance comparison of different trackers, we give detailed discussions on future prospects and provide suggestions on promising research directions of this field.

The structure of this review is schematically illustrated in Fig. 3. Section 2 gives some background information. In Section 3, RGB-infrared fusion tracking methods are discussed in detail, including key points in implementation and different fusion levels. In Section 4, we summarize the development of RGB-infrared datasets. Section 5 introduces the evaluation metrics. Section 6 presents experimental results and gives an analysis on the performances. Sections 7 discusses the future prospects. Finally, Section 8 concludes the paper.