Transformer-Based Visual Object Tracking with Global Feature Enhancement

Abstract

With the rise of general models, transformers have been adopted in visual object tracking algorithms as feature fusion networks. In these trackers, self-attention is used for global feature enhancement. Cross-attention is applied to fuse the features of the template and the search regions to capture the global information of the object. However, studies have found that the feature information fused by cross-attention does not pay enough attention to the object region. In order to enhance cross-attention for the object region, an enhanced cross-attention (ECA) module is proposed for global feature enhancement. By calculating the average attention score for each position in the fused feature sequence and assigning higher weights to the positions with higher attention scores, the proposed ECA module can improve the feature information in the object region and further enhance the matching accuracy. In addition, to reduce the computational complexity of self-attention, orthogonal random features are introduced to implement a fast attention operation. This decomposes the attention matrix into the product of a random non-linear function between the original query and key. This module can reduce the spatial complexity and improve the inference speed by avoiding the explicit construction of a quadratic attention matrix. Finally, a tracking method named GFETrack is proposed, which comprises a Siamese backbone network and an enhanced attention mechanism. Experimental results show that the proposed GFETrack achieves competitive results on four challenging datasets.

1. Introduction

Single-object tracking is pivotal in computer vision [1,2,3] and is tasked with estimating an object’s position and motion trajectory across frames. Overcoming challenges like size variations, occlusions, and deformations are persistent issues [4,5].

In the field of computer vision, advancements have been made in CNN-based tracking algorithms, such as the Siamese networks [6,7,8], particularly in addressing challenges like size variations, occlusions, and deformations. SiamFC [6] is a classic example that employs cross-correlation operations to compute a score map, determining the target’s bounding box by convolving the feature maps of the template and search regions. However, this approach, due to significant differences between the score map and the feature map, overlooks the semantic information of the target, hindering the subsequent classification and regression operations. To solve this problem, many transformer-based trackers [9,10,11,12] are used for feature fusion. Mixformer [10] proposes a mixed attention module (MAM), which applies the attention mechanism to simultaneously extract features and exchange information. OSTrack [11] integrates the target information more widely into the search area to better capture their correlation. In [12], the learning and memory capabilities of the transformer are utilized by encoding the information required for tracking into multiple tokens. Through cross-attention operations in the cross-feature augment (CFA) module, TransT [9] obtains fused features containing rich semantic information from templates and search regions. This fusion feature can be directly used for the subsequent classification and regression operations to estimate the state of the target. By introducing transformer operations [13], our proposed GFETrack replaces the cross-correlation operations adopted in TransT. Although the attention score map of TransT may exhibit some dispersion, as shown in Figure 1a, the proposed GFETrack exhibits a more concentrated pattern, as shown in Figure 1b. Considering the limitations of TransT’s CFA module in extracting target features and position information, the proposed GFETrack with global feature enhancement can focus more on the object itself and make it more suitable for the subsequent regression operations.

With the rise of Vision Transformer [14,15], self-attention techniques show promise in visual tasks. However, their quadratic computational complexity has prompted research into mitigating this challenge. Sparse global attention patterns [16,17] and smaller attention windows [18,19] can reduce costs, but at the risk of overlooking information or sacrificing long-term dependency modeling. In Fast re-OBJ [20], the authors efficiently leverage the intermediate outputs of the instance segmentation backbone (ISB) for triplet-based training, avoiding redundant feature extraction during both ISB and the embedding generation module (EGM). In contrast, our method employs a distinct strategy. Initially, we pass the Q and K matrices through a $\phi$ map to create new matrices $\tilde{Q}$ and $\tilde{K}$, ensuring that the attention matrix A is equal to or approximately equal to the product of $\tilde{Q}$ and $\tilde{K}$. This approach offers computational advantages by first calculating the product of $\tilde{K}$ and V after decomposing matrix A, followed by multiplying the result by $\tilde{Q}$, reducing the spatial complexity of regular attention from $O(N^2+Nd)$ to $O(Nr+Nd+rd)$. The main contributions of this paper are as follows:

• An enhanced cross-attention module is proposed to boost object region feature information in cross-attention. It is primarily achieved by computing the average attention weights and giving higher weights to the highly ranked positions to enhance matching accuracy.
• Orthogonal random features are introduced to implement fast self-attention within self-attention. The attention matrix is decomposed into the product of random nonlinear functions of the original query and key, which can effectively reduce the model’s spatial complexity and enhance the inference speed.
• On publicly available datasets, such as OTB, VOT, LaSOT, etc., the proposed algorithm has demonstrated improvements in tracking accuracy and success rates compared to the baseline algorithm. In comparison to some other state-of-the-art methods, the proposed algorithm has achieved comparable results.

2. Related Work

In recent years, convolutional networks have been widely used in computer vision tasks [21], including object detection, segmentation, and tracking. SiamFC [6] was the first to apply Siamese networks for visual object tracking, which predicts the object states by calculating the distance between the current frame and the template. SiamFC++ [8] is an improved version of SiamFC that introduces a new network architecture and training method, enabling object tracking in different scenarios. SiamMask [22] uses Siamese networks for both object tracking and segmentation simultaneously. SA-Sia [23] utilizes feature mapping to extract spatial information leading to more accurate and robust object tracking. SAT [7] employs a deep Siamese network to assess the similarity between objects in each frame, offering a novel solution for evaluating and addressing the association challenges in consecutive frames. However, the local matching strategy, which is based on cross-correlation, could lead to sub-optimal results, especially when the object is occluded or partially visible. Moreover, semantic information about the object may be lost during the correlation operation, resulting in imprecise object boundaries. Thus, an improved transformer and attention mechanism are proposed to replace traditional correlation operations in object tracking. It can effectively extract global contextual information while preserving an object’s semantic information, resulting in more robust and accurate tracking results.

The transformer is a popular neural network architecture that is widely used in natural language processing (NLP) tasks [13]. It consists of attention-based encoders and decoders. Self-attention, which is the main module of a transformer, can compute representations of input sequences. It allows each position in the input sequence to focus on the other positions and calculate the weighted averages of their values. DERT [24] and ViT [14] are early methods for introducing transformer models into the field of computer vision. In visual object tracking fields, transformer-based tracking methods achieve significant improvements compared to Siamese network-based trackers. The correlation operation in the Siamese network is replaced by a self-attention module from the transformer in TransT [9] to fuse the information between the template and search region. Stark [25] applies an encoder–decoder structure in tracking, where the encoder models the global spatiotemporal feature dependencies between the object and search region, while the decoder learns embedded queries to predict the spatial location of the object. SwinTrack [26] introduced a fully attention-based transformer algorithm for feature extraction and feature fusion, enabling the complete interaction between an object and a search region in the tracking process. The encoder–decoder framework based on transformers is widely applied to sequence prediction tasks, as self-attention models the interaction between elements in the sequence. Based on these works, we find that traditional transformer methods struggle to distinguish between the object to be tracked and similar interfering objects. To enhance the object information within the fused features, the ECA module is proposed to boost the attention on tracking the object. The ECA module can calculate the average attention level for each position in the fused feature sequence and weigh the positions with higher attention levels. This enhances the feature information in the object region and improves matching accuracy.

In recent years, a series of new attention mechanisms, such as fast attention [27], have garnered research interest. Sparse attention [28] reduces computational costs by constraining attention weights to consider only relationships within a neighborhood. Shaw et al. [29] proposed an attention mechanism to embed relative positional information into attention calculations, allowing the model to better handle dependencies in long sequences. This mechanism has shown excellent performance in tasks involving long sequences, such as text generation. In this work, orthogonal random features are proposed to achieve fast attention operations. It can decompose the attention matrix into the product of random nonlinear functions between the original query and key, avoiding the explicit construction of a quadratic-sized attention matrix.

3. The Proposed Method

A transformer-based tracker with global feature enhancement, named GFETrack, is proposed. The pipeline of GFETrack is shown in Figure 2.

GFETrack mainly consists of three components: the backbone network, the attention-enhanced fusion network, and the prediction head network. Similar to a Siamese network, the backbone network can extract features from both the template and the search region by sharing weights. Then, the attention-enhanced fusion network is proposed to enhance and fuse these features. Finally, the prediction head estimates the states of the objects and obtains the final results by binary classification and bounding box regression. Firstly, detailed information about each component of GFETrack is presented in this section. Then, two crucial modules, fast self-attention (FSA) and enhanced cross-attention (ECA), are proposed and presented in the attention-enhancement fusion network. Finally, some training details are described.

3.1. Overall Architecture

The overall architecture of the proposed GFETrack can be mainly divided into three parts: the feature extraction backbone network, the attention enhancement feature fusion network, and the prediction head network. The attention enhancement feature fusion network comprises the FSA module and the ECA module.

3.2. Feature Extraction Backbone Network

In alignment with the transformer method [6,8], we utilize features extracted from the template image and search region in initial frames with annotations as inputs to the feature extraction network, commonly referred to as the backbone network. In the initial frame, a template patch, $T\in R^{H_{z0}\times W_{z0}\times 3}$ centered around the object’s coordinates and with dimensions twice the length of the object’s surroundings, captures both the object’s appearance and surrounding features. Simultaneously, the search region, $S\in R^{H_{x0}\times W_{x0}\times 3}$ extends from the previous frame’s object center coordinates, with dimensions four times the length of the potential object movement range. These patches are reshaped into a square format and fed into the backbone network for feature extraction. The ultimate feature map is derived from the fourth stage of the ResNet50 network [21]. Notably, we removed the last stage of ResNet50 and adopted the outputs of the fourth stage as the final outputs. By changing the convolution stride of the downsampling unit in the fourth stage from 2 to 1, we aimed to achieve a larger feature resolution. Additionally, to increase the receptive field, we replaced the $3\times 3$ convolution in the fourth stage with a dilation convolution with a stride of 2, inspired by [9]. In general, the backbone network could extract the feature maps of the template, $F_T\in R^{H_T\times W_T\times C}$, and search region, $F_S\in R^{H_S\times W_S\times C}$. While, $H_T=\frac{H_{T0}}{8}$, $W_T=\frac{W_{T0}}{8}$, $H_S=\frac{H_{S0}}{8}$, $W_S=\frac{W_{S0}}{8}$, and $C=1024$.

3.3. Attention Enhancement Feature Fusion Network

The attention enhancement feature fusion network is designed to fuse the $F_t$ and $F_S$ features. Firstly, the channel dimensions of $F_t$ and $F_S$ features are reduced through a $1\times 1$ convolution, resulting in two low-dimension feature maps: $F_{T0}\in R^{H_T\times W_T\times d}$ and $F_{S0}\in R^{H_S\times W_S\times d}$, where d is set to be 256. Then, $F_{T0}$ and $F_{S0}$ are flattened along the spatial dimensions to obtain $F_{T1}\in R^{H_T{W}_T\times d}$ and $F_{S1}\in R^{H_S{W}_S\times d}$, which can be viewed as sets of feature vectors with a length of d. As shown in Figure 2, $F_{T1}$ and $F_{S1}$ are taken as inputs for the template branch and search region branch in attention-enhancement fusion networks, respectively. Two FSA modules are used to enhance input features by making these features adaptively focus on useful information through multi-head fast self-attention. Two ECA modules receive these feature maps simultaneously and fuse these feature maps with multi-head ECA. The attention-enhanced fusion layer is formed by these two FSA and ECA modules, as shown by the dashed box in Figure 2. The attention-enhanced fusion layer is repeated N times, and then an additional ECA module is used to fuse these feature maps from the two branches, resulting in the fused feature map, $F\in R^{d\times H_S{W}_S}$. N is set to be 4 in this paper.

3.3.1. Attention Mechanism

Attention is the fundamental component in feature fusion networks. N represents the size of the input feature sequence. The attention mechanism can be viewed as a mapping function that takes $Q,K,V\in R^{N\times d}$ as inputs, Q and K as two feature vector sets with dimensions $d_k$, and V as a feature vector set with dimensions $d_v$. The process can be interpreted as queries, keys, and values in a continuous dictionary. Thus, the attention scores between Q and K are obtained through scaled dot-product operations, while the attention map can be generated through softmax operations. The proposed GFETrack can reweight V based on the attention map. Thus, it can adaptively adjust the weights of features in V by leveraging the correlation between Q and K, allowing the algorithm to better focus on the features at useful positions within V. The traditional form of attention is as shown in Equation (1):

$$Attn(Q,K,V)=softmax\left(A\right)V,A=exp\left(\frac{Q{K}^T}{\sqrt{d_k}}\right),$$

(1)

where $A\in R^{N\times N}$ is the attention matrix. Because the attention matrix A must be saved explicitly, the time complexity and space complexity of Equation (1) are $O\left(N^{2}d\right)$ and $O(N^2+Nd)$, respectively. When extending to the multi-head attention, the formula is as shown in Equation (2):

$$\begin{array}{c}MultiHead(Q,K,V)=Concat(H_1,\dots H_{nh})W^o,\\ H_i=Attn(Q{W}_i^Q,K{W}_i^K,V{W}_i^V),\end{array}$$

(2)

where $W_i^Q\in R^{d_m\times d_k}$, $W_i^K\in R^{d_m\times d_k}$, $W_i^V\in R^{d_m\times d_k}$ and $W^0\in R^{d_m\times d_v{n}_h}$ are parameter matrices. $H_i$ represents the attention matrix for different heads. In this paper, $n_h=8$, $d_m=256$, $d_k=d_v=\frac{d_m}{n_h}=32$.

3.3.2. Fast Self-Attention Modules

The architecture of the proposed fast self-attention is shown in Figure 3. It applies a residual form of multi-head fast self-attention to adaptively adjust the weights from different positions in the feature map. As shown in Equation (1), spatial position encoding has to be introduced to distinguish the positional information of the input feature sequence. The sine function is applied to generate the spatial positional encoding. In the conventional approach, the attention matrix is multiplied with the value input to compute the final results. However, with the introduction of FSA, the matrix multiplication can be rearranged to approximate the outcomes of traditional attention mechanisms. Importantly, this approximation is achieved without the necessity of explicitly constructing a quadratic-size attention matrix. In essence, FSA optimizes the computational efficiency of attention mechanisms, offering a more streamlined and scalable solution.

Usually, the attention matrix $A(i,j)=K(q_i^T,k_i^T)$, where $q_i$ and $k_i$ are the i th query and key vector, respectively. $K(\ast )$ is the kernel function, which can be calculated by Equation (3):

$$K(x,y)=\mathbb{E}\left[\phi {\left(x\right)}^T\phi \left(y\right)\right],$$

(3)

where $\phi \left(u\right)$ represents the random feature map of $u\in R^d$. For $\tilde{Q},\tilde{K}\in R^{N\times R}$, the effective attention mechanism can be computed by Equation (4):

$$Att{(Q,K,V)}_F=\left(\tilde{Q}\left({\tilde{K}}^{T}V\right)\right),$$

(4)

where each row of $\tilde{Q}$ and $\tilde{K}$ are $\phi {\left(q_i^T\right)}^T$ and $\phi {\left(k_i^T\right)}^T$, respectively. $Att{(Q,K,V)}_F$ represents the fast self-attention, which is the approximation of traditional self-attention. As shown in Figure 3, the space complexity and time complexity of the proposed fast self-attention are $O^{\prime }(Nr+Nd+rd)$ and $O^{\prime }\left(Nrd\right)$, which are lower than the space and time complexity of traditional self-attention ($O(N^2+Nd)$ and $N^{2}d$, respectively). N is the number of patches, and d represents the dimension of the patches. Unlike the traditional self-attention [19], r is the hyperparameter. In this work, $N=64,d=128,r=8$. Thus, the spatial computational complexity is 79% of the traditional self-attention. The FSA module can be computed by Equation (5):

$$X_{FSA}=X+MultiHead(X+P_x,X+P_x,X),$$

(5)

where $P_x\in R^{N_x\times d}$ is the spatial position encoding, $X_{FSA}$ is the output of the multi-head fast self-attention, $X_{FSA}\in R^{N_x\times d}$.

3.3.3. Enhanced Cross-Attention Module

The enhanced cross-attention module is proposed to boost the feature information of the object area and improve the matching accuracy by reweighting the attention weights. As shown in Figure 4, $X_{FSA}$ from fast self-attention are taken as the inputs of the ECA module. The inputs $X_{kv}$ and $X_q$ are fused by a multi-head cross-attention with a residual form. Similar to the self-attention module, spatial position encoding $P_x$ is also used to enhance the cross-attention module. Additionally, a feed-forward network (FFN) is applied to enhance the fitting capability of the ECA module. The FFN can be computed by Equation (6):

$$FFN\left(x\right)=max(0,x{W}_1+b_1)W_2+b_2,$$

(6)

where x is the input feature vector, $W_1$ and $W_2$ denote the weight matrix of the first and second layers, respectively. Thus, $b_1$ and $b_2$ are the corresponding bias vectors. The overall computation formulas for the enhance cross-attention modules are as follows:

$$X_{ECA}={\tilde{X}}_{ECA}+FFN\left({\tilde{X}}_{ECA}\right),$$

(7)

$${\tilde{X}}_{ECA}=X_q+Output,$$

(8)

$$Output=\left\{\begin{array}{c}softmax(\frac{(X_q+P_q){(X_{kv}+P_{kv})}^T}{\sqrt{d_k}})X_{kv}\times rate,\\ ifsoftmax(\frac{(X_q+P_q){(X_{kv}+P_{kv})}^T}{\sqrt{d_k}})X_{kv}>0.7;\\ softmax(\frac{(X_q+P_q){(X_{kv}+P_{kv})}^T}{\sqrt{d_k}})X_{kv},\\ ifsoftmax(\frac{(X_q+P_q){(X_{kv}+P_{kv})}^T}{\sqrt{d_k}})X_{kv}\le 0.7;\end{array}\right.$$

(9)

where $X_q$ and $X_{kv}$ are the two inputs of the ECA module, $P_q$ and $P_{kv}$ are the corresponding spatial position encoding. The attention scores of ECA can be calculated by the element-wise product of $X_{kv}$ and $X_q$, as shown in Equation (9). When the scores in the attention map exceed 0.7, the scores are multiplied by a pre-defined coefficient to accomplish the enhancement of cross-attention. Firstly, the average attention level for each position in the search feature sequence is calculated, resulting in average attention weights. Then, the positions in the attention map are sorted based on their attention scores, and positions with high attention scores (exceeding 0.7) are reweighted by multiplying a rate to increase the weights of highly attentive positions by an additional 30%. The weights of other positions in the attention map are kept unchanged. Finally, the reweighted feature sequences are concatenated together to enhance the matching accuracy. Thus, $X_{kv}$ is reweighted based on the reweighted attention scores, and the results are added to $X_q$ to enhance the representation capability of the feature map. $X_{ECA}\in R^{N_q\times d}$ is the output of the enhance cross-attention module.

3.4. Prediction Head Network

The prediction head network comprises both a classification branch and a regression branch. Each branch is constructed with a multi-layer perception (MLP) with three hidden layers and a ReLU activation function. For each fused feature, the prediction head generates predictions, computing foreground/background classification results and normalizing coordinates in relation to the size of the search region. Notably, the proposed GFETrack is capable of directly predicting the states of objects without the need for additional post-processing.

3.5. The Training Loss

The overall training loss of the proposed GFETrack can be divided into classification loss and regression prediction loss. The prediction head receives the feature vectors of $H_s{W}_s$ and outputs binary classification and regression results. The feature vectors within the ground truth are viewed as positive samples, while the rest of the feature vectors are viewed as negative samples. All samples contribute to the classification loss, but only positive samples contribute to the regression loss. In order to mitigate the imbalance between positive and negative samples, the loss caused by negative samples has been reduced by a factor of 16. The standard binary cross-entropy loss is used for the classification, as shown in Equation (10):

$$L_{cls}=-\sum _i[y_{i}log\left(p_i\right)+(1-y_i)log(1-p_i)],$$

(10)

where $y_i$ is the label of the ith sample, $y_i=1$ represents the foreground, while $y_i=0$ indicates the background. $p_i$ is the probability predicted by the learning model that belongs to the foreground. For regression, as described in [9], the $L_1$ loss and the GIoU loss [30] are utilized in this paper. Thus, the regression loss can be expressed as:

$$L_{reg}=\sum _i[{\lambda }_g{L}_{GIoU}(b_i,\widehat{b})+{\lambda }_1{L}_1(b_i,\widehat{b})],$$

(11)

where $b_i$ is the ith prediction bounding box, $\widehat{b}$ is normalized ground truth bounding box, $L_1$ and $L_{GIoU}$ represent the $L_1$ loss and the generalized IoU loss functions, respectively, and ${\lambda }_1$$L_1$ and ${\lambda }_g{L}_{GIoU}$ are the hyperparameters determining the relative impact of the respective loss functions. We set it up in the experiment as ${\lambda }_g=2$ and ${\lambda }_1=5$. Finally, the overall loss is defined as:

$$L={\lambda }_c{L}_{cls}+L_{reg}$$

(12)

where ${\lambda }_c=8.3$ is the weight factor, and $L_{reg}$ and $L_{cls}$ are the losses of regression and classification, respectively.

4. The Experiments

The proposed GFETrack is implemented using Python 3.7 and PyTorch 1.7.1. The training procedure is implemented on two GeForce GTX 4090 GPUs. The entire training process requires approximately 180 h on two NVIDIA RTX 4090 GPUs, while the inference procedure of GFETrack is performed on an RTX 2070 GPU. Specifically, this section will describe the performance and implementation of GFETrack from four aspects, including the implementation details, some analysis of GFETrack, and quantitative and qualitative experiments on public datasets.

4.1. Implementation Details

In order to enhance the discrimination power of the fused feature and reduce the overfitting problem, the proposed model has been trained on the COCO2017 [31], TrackingNet [32], LaSOT [33], and GOT10K [34] datasets, respectively. We directly sampled images from video sequences to construct training samples. Image transformation skills are employed to generate image pairs for the COCO dataset. Additionally, data augmentation techniques such as rotating and color jittering are also applied to augment the training data. The size of the search region and template region are set at $256\times 256$ and $128\times 128$, respectively. The pre-trained ResNet-50 is utilized to initialize the backbone network. The AdamW [35] optimizer is employed in the training process, the initial learning rate is set at ${10}^{-5}$ and is reduced by a factor of 10 after 600 epochs. Two NVIDIA RTX 4090 GPUs are used for training, with a batch size of 32 on each GPU. In general, we perform 1000 epochs, and each epoch consists of 1000 sample pairs.

4.2. Analysis of GFETrack

4.2.1. The Ablation Experiments

The impact of different components on the success rate of the OTB100 dataset is analyzed to gain insights into the proposed tracker’s performance. As presented in

Table 1. The ablation experiments on OTB100 dataset. “Yes” indicates the corresponding proposed module is used, while “No” indicates the corresponding proposed module is not used.

Methods	FSA	ECA	Pos	Success
M1	Yes	Yes	Yes	70.8
M2	No	Yes	Yes	70.3
M3	Yes	No	Yes	69.7
M4	Yes	Yes	No	69.5
M5	No	No	No	69.3

, $M1$ represents the proposed GFETrack, while $M2$, $M3$, and $M4$ indicate variations in the proposed tracker without the FSA module, ECA module, and positional encoding, respectively. $M5$ represents the baseline tracker.

Notably, when the ECA module is removed (comparing $M1$ and $M3$), the success rate experiences a significant drop of 1.1%. This decrease underscores the critical role played by the ECA module in focusing on the most salient subset of the object, thereby contributing to the overall tracking accuracy. The ECA module facilitates enhanced feature discrimination, allowing the tracker to prioritize and attend to crucial object-related information. Additionally, comparing $M1$ and $M2$ reveals that the FSA module leads to a modest improvement in the success rate of 0.5%. Contrary to initial assumptions, this improvement is not driven by a direct enhancement of tracking accuracy. Instead, the primary purpose of the FSA module is to reduce the number of parameters and computational complexity. By achieving parameter efficiency, the FSA module indirectly enhances the tracking speed without compromising the success rate. This nuanced understanding emphasizes that certain modules, while not directly contributing to accuracy improvement, play vital roles in optimizing the overall performance and efficiency of the tracker.

4.2.2. The Speed and Number of Parameters

To analyze the number of parameters and running speed of the proposed GFETrack, we compared the tracking speed and the number of parameters of the proposed tracker with three similar algorithms, TransT [9], STARK-ST50 [25], and SiamRPN++ [36].

As illustrated in

Table 2. The comparison experiments of tracking speed and the number of parameters.

Trackers	Speed (fps)	Params (M)
GFETrack	55	20.1
TransT	54	23.0
STARK-ST50	42.2	23.3
SiamRPN++	35.0	54.0

, the running speed of GFETrack is measured at 55 fps, surpassing the threshold of 25 fps. This indicates that our proposed method is well-suited for real-time object tracking. Additionally, GFETrack achieves a notable reduction of approximately 20% in the number of parameters compared to TransT. The reduction in parameters is attributed to our innovative FSA module, which employs orthogonal random features to implement fast self-attention within self-attention. This involves decomposing the attention matrix into the product of random non-linear functions of the original query and key. By doing so, the FSA module effectively reduces the model’s spatial complexity. This reduction in spatial complexity not only contributes to a more streamlined model but also enhances inference speed. The trade-off between speed and accuracy is a critical consideration in tracking scenarios. In the case of GFETrack, the FSA module achieves a delicate balance by reducing the number of parameters without compromising tracking accuracy. Orthogonal random features enable efficient self-attention computations, allowing the model to maintain a high level of accuracy while ensuring rapid inference. Generally, reducing the size and computational complexity of the model will result in an increase in fps, However, for accuracy, it may have the opposite effect. The effect in our article, such as Vot2018, is not very good, and it may also have this impact.

4.2.3. Analysis of the ECA Module

To analyze how the ECA module operates within GFETrack, cross-attention feature maps of different layers are visualized in Figure 1. From the four feature maps of different layers, we observe that the background interference is progressively suppressed, layer by layer, demonstrating the effectiveness of the ECA module in improving object recognition capability. As the number of layers deepens, the ECA module becomes more focused on the object itself. Compared to the TransT tracker, GFETrack achieves better discrimination between the object and background in the first two layers. This also demonstrates the effectiveness of our ECA module.

4.3. Quantitative Experiments on Public Datasets

Four widely used tracking benchmarks, VOT2018 [37], LaSOT [33], UAV123 [38], and OTB100 [39], are used to evaluate the performance of the proposed GFETrack.

VOT2018:

This dataset consists of 60 challenging video sequences. The accuracy (A), robustness (R), and expected average overlap (EAO) are used as the evaluation criteria for VOT2018. An excellent tracker typically has high A and EAO scores, but low R scores.

Table 3. The comparison experiments of different trackers on VOT2018 dataset. The red, green, and blue fonts indicate the top 3 trackers.

	ECO [40]	ATOM [41]	SiamRPN++ [36]	DiMP [42]	Ocean-Online [43]	TransT [9]	GFETrack
EAO ↑	0.280	0.401	0.414	0.440	0.489	0.266	0.362
A ↑	0.484	0.590	0.600	0.597	0.592	0.587	0.637
R ↓	0.276	0.204	0.234	0.153	0.117	0.384	0.281

reveals that GFETrack excels in accuracy with a score of 0.637, surpassing the baseline tracker (TransT). However, it is noteworthy that our proposed tracker demonstrates relatively average performance in terms of EAO and R scores. We attribute this observation primarily to a deliberate choice made during the feature extraction stage, wherein we removed the last layer of ResNet50. While this enhances accuracy by focusing on specific features, it inadvertently results in the loss of some local information from input images. This loss of local information likely contributes to the observed dip in both robustness (R) and expected average overlap (EAO) scores. The intentional removal of the last ResNet50 layer was aimed at optimizing the tracker for accuracy-focused scenarios. However, we acknowledge that this design choice comes at the cost of reduced robustness, as evidenced by the comparatively lower R and EAO scores. Future work may explore strategies to balance this trade-off, perhaps through the integration of additional contextual information or the use of alternative feature extraction techniques.

LaSOT:

LaSOT is a large-scale, long-term, visual object tracking dataset that includes over 10000 video sequences covering 142 different object categories. In Figure 5, the success and precision plots are presented. The proposed GFETrack outperformed other trackers, achieving the highest success rate of 67.5%, which is 2.6% higher than the baseline tracker, TransT. It also reached the highest precision rate of 73.2%, surpassing TransT by 4.2%.

To further analyze the proposed GFETrack, Figure 6 also presents the radar chart for various tracking challenges on the LaSOT dataset. As shown in Figure 6, compared to some state-of-the-art trackers, such as DiMP [42], DaSiaRPN [44], ATOM [41], TransT [9], SiamRPN++ [36], TrSiam [45], and PrDiMP [46], the proposed trackers demonstrate better performance in most of the tracking challenges, especially in viewpoint change, low resolution, and camera motion.

The detailed precision plots of different tracking challenges are shown in Figure 7. The proposed tracker performs well in various scenarios, such as viewpoint change, camera clutter, and low resolution. This indicates that the proposed method has significant potential in handling challenging scenarios. We believe the primary reason is that the proposed tracker focuses more on the object itself, leading to increased precision and robustness.

OTB100:

This dataset contains 100 sequences for visual object tracking with 11 tracking challenges. Figure 8 displays the results of the proposed tracker of precision and success rates. GFETrack achieved a promising performance on the OTB100 dataset. The success rate of GFETrack is 0.708, which is 1.9% higher than TransT, and the precision of GFETrack is 0.911, which is 2.5% higher than TransT.

In Figure 9, we compared the success plots of the proposed GFETrack with some state-of-the-art trackers under 11 tracking challenges. The results demonstrate that in most tracking challenges, the proposed GFETrack outperforms other trackers, such as TransT [9], SiamBAN [47], and SiamCAR [48]. Especially in the aspects of deformation and in-plane rotation, the proposed tracker achieved significant improvements when compared to the baseline TransT, which was higher by 3.4% and 2.6%, respectively. This also demonstrates that the proposed ECA module possesses strong discriminative capabilities, allowing us to focus more on the object itself.

UAV123:

UAV123 is one of the most widely used datasets in the fields of visual object tracking and unmanned aerial vehicles. The AUC and precision are the important evaluation metrics in the UAV123 dataset.

Table 4. The comparison experiments of different trackers on UAV123 dataset. The top 2 results are marked in red and blue, respectively.

Method	AUC(%)	P(%)
GFETrack	67.3	87.0
TransT [9]	68.1	87.6
ToMP-101 [49]	66.9	-
TREG [50]	66.9	88.4
TrDiMP [45]	67.5	-
SiamAttn [51]	65.0	84.5
SiamCAR [48]	61.4	76.0
DiMP [42]	65.4	-
ATOM [41]	64.3	-
SiamFC [6]	48.5	69.3

shows the results of the proposed tracker on the UAV123 dataset. The AUC score of GFETrack is 67.3, which is 0.9 lower than TransT. We believe the reason is that the scale variations in objects in the UAV123 dataset are relatively small. This may lead to inaccuracies in object tracking since transformer-based models are less sensitive to objects that are small and have small-scale variations.

4.4. Qualitative Experiments on Public Datasets

To qualitatively compare the proposed tracker with other trackers, Figure 10 also shows the tracking results of six tracking algorithms, including TransT [9], MDNet [1], DaSiaRPN [52], SiamRPN [44], Staple [35], and Ocean [43], on five challenging sequences. The five sequences, from top to bottom, are Ironman, Shaking, Bird1, Singer2, and Soccer, respectively. These five sequences encompass nearly 11 different tracking challenges, and some conventional trackers show poor performance in these five sequences, making them prone to losing the object and resulting in tracking failures.

From Figure 10, we can see that, compared to the bounding boxes of other colors, the red bounding boxes and green bounding boxes have a higher degree of overlap. This also indicates that the proposed tracker, compared to other methods, can achieve better tracking results, and the ECA module can better direct the fused features to pay attention to the object’s region. Other trackers, such as TransT [9] and Ocean [43] in the last row, have indirectly demonstrated the robustness of our algorithm under different challenges.

Although the proposed method could handle most of the tracking challenges, we notice that the ECA modules may face some challenges and lead to a decrease in matching accuracy under some specific conditions, such as complex backgrounds or when the appearance of the object changes fast. As the proposed GFETrack is still in its theoretical stage, it may encounter hurdles in the real world. Thus, research on the better feature fusion and re-detection methods would be the next task in GFETrack.

5. Conclusions

GFETrack, which is an enhanced transformer-based tracking method aimed at augmenting the extraction and fusion capabilities of object features, is proposed in this paper. It leverages a transformer as a feature fusion network, which can aggregate global information from object features through self-attention and cross-attention mechanisms. To bolster the effectiveness of cross-attention, the ECA module is proposed, which plays a pivotal role in refining feature information within the object area by reweighting the highest attention–rank positions. It could enhance matching accuracy and contribute to overall robustness. To address the computational complexity inherent in self-attention, the FSA module is proposed, which achieves complexity reduction by introducing orthogonal random features. Thereby, it reduces spatial complexity from $O(N^2+Nd)$ to $O(Nr+Nd+rd)$, which not only ensures computational efficiency but also preserves high-quality representations of object features for further optimizing the overall tracking process. The experiments on some challenging datasets confirm the effectiveness and practicality of the proposed GFETrack.