Parameter-Efficient Tuning for Object Tracking by Migrating Pre-Trained Decoders

Abstract

Video object tracking has taken advantage of pre-trained weights on large-scale datasets. However, most trackers fully fine-tune all the backbone’s parameters for adjusting to tracking-specific representations, where the utilization rate of parameter adjustment is inefficient. In this paper, we aim to explore whether a better balance can be achieved between parameter efficiency and tracking performance, and fully utilize the weight advantage of training on large-scale datasets. There are two main differences from a normal tracking paradigm: (i) We freeze the pre-trained weights of the backbone and add a dynamic adapter structure for every transformer block for tuning. (ii) We migrate the pre-trained decoder blocks to the tracking head for better generalization and localization. Extensive experiments are conducted on both mainstream challenging datasets and datasets for special scenarios or targets such as night-time and transparent objects. With the full utilization of pre-training knowledge, we found through experiments that a few tuned parameters can compensate for the gap between the pre-trained representation and the tracking-specific representation, especially for large backbones. Even better performance and generalization can be achieved. For instance, our AdaDe-B256 tracker achieves 49.5 AUC on the LaSOT${}_{\mathrm{ext}}$ which contains 150 sequences.

1. Introduction

Visual object tracking is a fundamental visual task that tracks the target specified in the initial frame in subsequent frames. In recent years, great improvement has been achieved in the field of tracking, specifically with the further development of pre-trained models [1,2,3] and the application of transformer structures [4] in visual tasks.

Early backbones for tracking, such as AlexNet [5] and ResNet [6], are usually pre-trained with the ImageNet dataset with a supervised training method. The tracking pipeline can be modeled as a two-stream matching process, between search and template regions. With the introduction of Vision Transformer (ViT), many works benefit from the transformer model in vision tasks [7,8]. Tracking can enjoy one-stream modeling which promotes the extracting representations and relation interactions benefiting from the transformer model. Moreover, various self-supervised training methods have been explored for ViT, prompting generalizing representation learning with richer training data. For instance, the contrastive learning model CLIP [1] and masked image modeling model MAE [2] have shown their excellent performance as the pre-trained weights [9,10]. DropMAE [11] improves MAE pre-training with video data rather than static image data, exploiting prior temporal correspondence information and making pre-trained weights more suitable for transferring to the downstream tracking task. Tracking specified representation is learned further in the masked image modeling method with rich tracking video data in MAT [12]. Although pre-trained weights have a significant impact on tracking performance [10], they only exert their influence on a tracking model as an initialization method. Full parameter fine-tuning is still necessary, which results in very low adjusting efficiency of parameters for the downstream tracking task and consumes more computing resources.

Since pre-trained models may come into contact with richer training data than tracking data, and even multi-modal data [1], especially large models that may benefit more from large amounts of pre-training data, does the learned pre-task representation and tracking specified representation contain only a small fraction of the task differences, such as priors of temporal relations? As parameter-efficient transfer learning (PETL) has attracted attention as an alternative to full fine-tuning, we also resort to that method for exploring that issue. In this paper, we freeze the pre-trained weights of the backbone and add a dynamic adapter structure for every transformer block for tuning. Inspired by [13], we also migrate the pre-trained decoder blocks in MAE pre-training ahead of the tracking head for better generalization. This can be regarded as an extension of the traditional tracking head, which only encoder features of the search region are fed into.

Our significant contributions are summarized below:

• We leverage adapter adjusting parameters to improve transfer learning in tracking. Attached to pre-trained models, with only a few additional parameters trained, tracking performance comparable to full fine-tuning can be achieved, dramatically increasing parameter efficiency.
• We migrate pre-trained transformer decoders in MAE pre-training to enhance the tracking head, increasing the robustness and generalization of tracking.

2. Related Work

Visual Object Tracking. We focus on single-object tracking rather than multiple-object tracking [14,15,16,17,18]. Recently, there have been significant developments in visual object tracking technology, especially with the support of the transformer structure. The transformer structure can promote the relation modeling between the template and search region features such as a neck module [19,20,21,22], and its attention modeling mechanism facilitates the one-stream pipeline [9,10] as well. The one-stream tracking pipeline makes tracking-specific representation learning and information interaction between template–search pairs optimized simultaneously, achieving strong discriminative power and a better performance–speed trade-off.

Inspired by development in pre-training methods, some works focus on the optimization of tracking-specified representation learning [11,12]. DropMAE [11] introduces masked autoencoder (MAE) pre-training in videos as an alternative to MAE pre-training in static images, ImageNet, and the latter is regarded as a sub-optimal method for a video object tracking task. MAT [12] can learn to track specified representations through a simple encoder–decoder pipeline via a masked appearance transfer technique. However, these works may require newly designed pre-training or additional fine-tuning of all parameters, consuming more computational resources. Currently, new pre-training methods are constantly emerging [1,2,3], and pre-trained models can learn more robust and generalized representations from large-scale datasets. Given this situation, in this work, we draw support from the recent parameter-efficient transfer learning method to explore whether it is possible to make slight adjustments on the basis of pre-trained models for obtaining better tracking-specified representations.

Parameter-Efficient Transfer Learning. Due to the increasing size of pre-trained networks, fine-tuning all network parameters in downstream tasks has become challenging in terms of computational resources and storage. Parameter-efficient transfer learning (PETL) is then presented to tackle with that issue in the NLP field [23], and has become popular in the field of computer vision as well. Common tuning methods include prompt tuning, adapter tuning and LoRA tuning. VPT [24] is a typical prompt-tuning work in vision. A set of learnable parameters is added to a pre-trained model, resulting in better performance than full fine-tuning on downstream tasks. Adapter tuning [25,26,27] always inserts a lightweight module into the original backbone structure for single or multiple downstream tasks. LoRA [28] tuning further presents low-rank matrices to approximate weight updating.

Recently, parameter-efficient transfer learning has received attention in the visual tracking field as well. By leveraging the power of RGB tracking foundation models, multi-modal tracking methods [29,30,31,32,33] can perform efficient training with a small number of parameters, and achieve comparable or better performance than the fully fine-tuned paradigm. These works demonstrate that RGB tracking models can adapt to multi-modal tracking with only a small number of parameters’ adjustments. Since multi-modal tracking representations and RGB-specific tracking representations can be bridged via a few parameters tuned, we are curious: is there a similar conclusion about RGB-specific tracking representations and pre-trained representations? We also investigated the effectiveness of PETL under different pre-training tasks, including contrastive learning such as CLIP [1] and masked image modeling such as MAE [2].

3. Proposed Approach

3.1. Preliminary

The one-stream tracking pipeline has shown its strong discriminative power and become the mainstream pipeline applied in many works [9,10,34,35,36]. With the attention mechanism of the transformer, the template and the search region are embedded in the same space for similarity calculation, and all-layer information interaction can better promote the template-guided feature extraction of search regions.

The one-stream pipeline first embeds the template region $I_z\in {\mathbb{R}}^{3\ast H_z\ast W_z}$ and search region $I_x\in {\mathbb{R}}^{3\ast H_x\ast W_x}$ into a series of patch embeddings $F_{xz}\in {\mathbb{R}}^{(N_x+N_z)\ast d}$. $H_z$ is the height of the template region; $W_z$ is the width of the template region. $H_x$ is the height of the search region; $W_x$ is the width of the search region. $N_x$ is the patch number of search region features under the patch size of P; $N_z$ is the patch number of template region features similarly. d is the channel dimension of embedded features. Then, two learnable positional embeddings, $P_x\in {\mathbb{R}}^{N_x\ast d}$ and $P_z\in {\mathbb{R}}^{N_z\ast d}$, are concatenated and added to the sequence of patch embeddings $F_{xz}$:

$$\begin{array}{c}\hfill F_{xz}^{{}^{\prime }}=F_{xz}+concat(P_x,P_z)\end{array}$$

(1)

Finally, patch embeddings $F_{xz}^{{}^{\prime }}$ will be fed into the encoder composed of transformer layers for representation learning and relation modeling. The i-th block of the transformer layers can be written as follows:

$$\begin{array}{c}\hfill F_{xz}^i=Attention\left(LayerNorm\left(F_{xz}^i\right)\right)+F_{xz}^i\\ \hfill F_{xz}^{i+1}=MLP\left(LayerNorm\left(F_{xz}^i\right)\right)+F_{xz}^i\end{array}$$

(2)

Finally, the search region part of the last layer’s output of features will be sent to the tracking head, which is used for target bounding box estimation. In general, the tracking head is the only part for random initialization rather than initialization inherited from pre-trained models.

3.2. Adapter Tuning

In most existing trackers, such as OSTrack [10] and KeepTrack [37], the full fine-tuning method is utilized for training. All the parameters are tuned, resulting in high costs during training. The adapter tuning technique can achieve equally effective training to full fine-tuning by training only a small number of parameters and freezing the remaining model parameters. There are many forms of adapter applied [25,26,32]. As illustrated in Figure 1, adapter tuning in our work is a bottleneck module for residual addition after MLP (Multi-layer Perceptron) operation in every transformer block. That bottleneck module includes two light MLP layers, one for downward projection and the other for upward projection. Every light MLP layer contains two linear projections which are connected by an activation function, GELU. With F as the input, our inserted adapter module can be expressed as follows:

$$\begin{array}{c}{F}_{down}=GELU(F{W}_{down})W_1\\ F_{up}=GELU(F_{down}{W}_2)W_{up}\\ F_{adapter}=F_{up}+F\end{array}$$

(3)

$W_2\in {\mathbb{R}}^{k\ast k},W_{up}\in {\mathbb{R}}^{k\ast d},W_1\in {\mathbb{R}}^{k\ast k},W_{down}\in {\mathbb{R}}^{d\ast k}$.

Figure 1. Details of our tracking framework with parameter-efficient transfer learning. “…” represents the several omitted transformer layers which are the same as others.

Figure 1. Details of our tracking framework with parameter-efficient transfer learning. “…” represents the several omitted transformer layers which are the same as others.

In our work, F represents the output of every transformer block; then, $F_{adapter}$ is the output of every transformer block after adjusting with the adapter module. d is the channel size of the transformer backbone and k is the compressed size, $k<d$. The details are also illustrated in Figure 1. Only a very small number of parameters is introduced in that module. By tuning these parameters, the pre-training representations of the pretext task can be well transferred to the downstream tracking task.

3.3. Migrating Pre-Trained Decoders

Motivation. Recently, some works have pointed out that discarding part of the pre-trained network may cause information loss [38] and taking full advantage of pre-trained models can contribute to improving the generalization of detectors [13] in the MAE pre-training method. To verify whether the pre-trained decoder contributes to downstream tracking tasks, we designed a set of comparative experiments. As illustrated in Figure 2, we insert the decoder module in the pre-training stage for predicting the image pixels between the encoder and tracking head. Only the features of the search region are imported into the decoder, so it can be regarded as an enhanced part of the tracking head. We adopt the corner head [19] without output embedding weighting as the tracking head for all variants. For variants with a decoder structure, the tracking head is reduced to only two convolution layers for the top-left and bottom-right score maps’ estimation. We also set a variant of random decoder for comparing the effectiveness of the pre-trained weights, as in Figure 2c. Experiments are conducted on LaSOT [39] and LaSOT${}_{\mathrm{ext}}$ [40]. Full fine-tuning is employed for all and the training settings are mostly the same as for OSTrack [10]. Details are in Section 4.1.

As shown in

Table 1. Comparison of three variants for exploration of pre-trained decoder. AUC scores are reported.

Tracker	LaSOT	Improvement	LaSOT${}_{\mathbf{ext}}$	Improvement
Baseline	68.9	-	48.7	-
Pre-trained Decoder	69.3	+0.4%	48.5	−0.2%
Random Decoder	68.6	−0.3%	47.5	−1.2%

, with the pre-trained decoder as the main part of the tracking head, just two additional convolution layers random initialized can achieve an even better performance than the baseline tracker. However, the decoder which is random initialized shows a decline in performance especially for the LaSOT${}_{\mathrm{ext}}$ benchmark. We think that the pre-trained decoder module indeed has benefits for tracking.

Decoder with Adapter Tuning. To taking full advantage of the pre-trained models, we apply the adapter tuning in decoder blocks as in Section 3.2 as well. Decoder blocks are expected to enhance the generalization ability and robustness when transferring to tracking.

4. Experiments

4.1. Implementation Details

Our models are trained with the AdamW optimizer [41] on two NVIDIA A100 GPUs (NVIDIA, Santa Clara, CA, USA), with each GPU holding a batchsize of 64. Training data contain training slices of COCO-2017 [42] and LaSOT [39]. Aligned with most trackers, data augmentations contains brightness jitter and horizontal flip. A total of 300 epochs are conducted with $6\times {10}^4$ template–search pairs each epoch, and each template–search pair is sampled among all the frames of a sequence. The learning rate decreases by a factor of 0.1 after the first 240 epochs. The learning rate is set as $2\times {10}^{-4}$ for both the encoder and decoder and $2\times {10}^{-5}$ for the remaining parameters. The compressed dimension k of the adapter structure is 128.

We present two variants for our model: B256 and L256. For B256, we adopt the ViT-Base model pre-trained with MAE [2]; for L256, we adopt the ViT-Large model pre-trained with MAE [2]. The ViT-Base model contains 12 layers in the encoder part and 8 layers in the decoder part, with an embedding dimension of 768. The ViT-Large model contains 24 layers in the encoder part and 8 layers in the decoder part, with an embedding dimension of 1024. For the B256 and L256 variants, the search region is resized to $256\times 256$ pixels and cropped with 4${}^2$ times the region of the target size; the template region is resized to $128\times 128$ pixels and cropped with 2${}^2$ times the region of the target size. The channel dimension k of the adapter module is 128 for all variants of model.

4.2. Mainstream Benchmarks

LaSOT [39]. A total of 280 videos constitute the LaSOT testing dataset. That benchmark is popular with its large scale and relatively long average duration. As reported in Table 2, our AdaDe-B256 achieves a 68.7 AUC score, which does not reflect a performance improvement or advantage, while AdaDe-L256 achieves a 71.2 AUC score, which exceeds the corresponding baseline tracker with a 3.1% performance increase. For larger pre-trained models, adapter tuning shows better performance advantages compared with full fine-tuning. That phenomenon may indicate that larger models are more prone to weaken in terms of feature representation ability during downstream fine-tuning training compared to adapter tuning.

LaSOT${}_{\mathrm{ext}}$ [40]. The LaSOT${}_{\mathrm{ext}}$ benchmark is an expansion of LaSOT, and consists of 150 videos among 15 categories. It adopts the one-shot evaluation protocol, which ensures that there is no intersection between categories in the testing dataset with the categories in the LaSOT’s training dataset. Rather than the MATLAB evaluation toolkit, we take the Python toolkit available from OSTrack [10] for a fair comparison with other trackers’ reported results. Our AdaDe-B256 achieves a 49.5 AUC score and AdaDe-L256 achieves a 50.3 AUC score. Compared with the baseline tracker, our trackers also outperform by 0.8% and 1.3% performance increases, respectively.

TrackingNet [43]. The TrackingNet benchmark is a massive short-term tracking benchmark that includes 511 videos for evaluation. From Table 2, we can also see that our tracker works well compared with other methods. On the basis of the adapter tuning of the pre-trained backbones, we also migrate pre-trained transformer decoders into MAE pre-training to enhance the tracking head. This technique can enhance the expressive power of the model to a certain extent, resulting in improved tracking accuracy. The proposed method can work well in both the long-term and short-term tracking datasets.

UAV123 [44], NFS [45] and TNL2K [46]. UAV123 and NFS are two small benchmarks with 123 videos captured by drones and 100 videos, respectively. TNL2K is a recently launched massive tracking benchmark containing 700 linguistically labeled sequences. For TNL2K, we provide the evaluation results of the Python toolkit. The experimental results on these challenging benchmarks show that the proposed method achieves a competitive performance compared to many state-of-the-art algorithms (shown in Table 3).

Table 2. Comparison with the state of the art on the LaSOT [39], LaSOT${}_{\mathrm{ext}}$ [40] and TrackingNet [43] benchmarks. The best performance is in bold.

Tracker	LaSOT			LaSOT_ext			TrackingNet
	AUC	P_norm	P	AUC	P_norm	P	AUC	P_norm	P
AdaDe-L256	71.2 (+3.1)	80.9	78.6	50.3 (+1.3)	60.6	57.3	84.3	88.8	83.8
Baseline-L256	68.1	76.5	73.8	49.0	58.6	56.1	83.8	88.2	83.3
AdaDe-B256	68.7 (−0.2)	78.5	74.6	49.5 (+0.8)	59.9	56.0	82.0	86.6	80.1
Baseline-B256	68.9	78.1	74.5	48.7	58.7	55.3	82.8	87.2	81.2
LoRAT-B-224 [47]	72.4	81.6	77.9	48.5	61.7	55.3	83.7	88.2	82.2
GRM-L320 [34]	71.4	81.2	77.9	-	-	-	84.4	88.9	84.0
GRM-B256 [34]	69.9	79.3	75.8	-	-	-	84.0	88.7	83.3
OSTrack-B384 [10]	71.1	81.1	77.6	50.5	61.3	57.6	83.9	88.5	83.2
OSTrack-B256 [10]	69.1	78.7	75.2	47.4	57.3	53.3	83.1	87.8	82.0
SimTrack-L [9]	70.5	79.7	-	-	-	-	83.4	87.4	-
SimTrack-B [9]	69.3	78.5	-	-	-	-	82.3	86.5	-
SwinTrack-B [22]	69.6	78.6	74.1	47.6	58.2	54.1	82.5	87.0	80.4
Mixformer-L [48]	70.1	79.9	76.3	-	-	-	83.9	88.9
MixFormer-22k [48]	69.2	78.7	74.7	-	-	-	83.1	88.1	81.6
AiATrack [49]	69.0	79.4	73.8	47.7	55.6	55.4	82.7	87.8	80.4
ToMP-101 [50]	68.5	-	-	45.9	-	-	81.5	86.4	78.9
GTELT [51]	67.7	-	73.2	45.0	54.2	52.4	82.5	86.7	81.6
KeepTrack [37]	67.1	77.2	70.2	48.2	58.1	56.4	-	-	-
STARK-101 [19]	67.1	77.0	-	-	-	-	82.0	86.9	-
TransT [20]	64.9	73.8	69.0	-	-	-	81.4	86.7	80.3
SiamR-CNN [52]	64.8	72.2	-	-	-	-	81.2	85.4	80.0
TrDiMP [21]	63.9	-	61.4	-	-	-	78.4	83.3	73.1
LTMU [53]	57.2	-	57.2	41.4	49.9	47.3	-	-	-
DiMP [54]	56.9	65.0	56.7	39.2	47.6	45.1	74.0	80.1	68.7
SiamPRN++ [55]	49.6	56.9	49.1	34.0	41.6	39.6	73.3	80.0	69.4
SiamFC [56]	33.6	42.0	33.9	23.0	31.1	26.9	57.1	66.3	53.3

Table 2. Comparison with the state of the art on the LaSOT [39], LaSOT${}_{\mathrm{ext}}$ [40] and TrackingNet [43] benchmarks. The best performance is in bold.

Tracker	LaSOT			LaSOT_ext			TrackingNet
	AUC	P_norm	P	AUC	P_norm	P	AUC	P_norm	P
AdaDe-L256	71.2 (+3.1)	80.9	78.6	50.3 (+1.3)	60.6	57.3	84.3	88.8	83.8
Baseline-L256	68.1	76.5	73.8	49.0	58.6	56.1	83.8	88.2	83.3
AdaDe-B256	68.7 (−0.2)	78.5	74.6	49.5 (+0.8)	59.9	56.0	82.0	86.6	80.1
Baseline-B256	68.9	78.1	74.5	48.7	58.7	55.3	82.8	87.2	81.2
LoRAT-B-224 [47]	72.4	81.6	77.9	48.5	61.7	55.3	83.7	88.2	82.2
GRM-L320 [34]	71.4	81.2	77.9	-	-	-	84.4	88.9	84.0
GRM-B256 [34]	69.9	79.3	75.8	-	-	-	84.0	88.7	83.3
OSTrack-B384 [10]	71.1	81.1	77.6	50.5	61.3	57.6	83.9	88.5	83.2
OSTrack-B256 [10]	69.1	78.7	75.2	47.4	57.3	53.3	83.1	87.8	82.0
SimTrack-L [9]	70.5	79.7	-	-	-	-	83.4	87.4	-
SimTrack-B [9]	69.3	78.5	-	-	-	-	82.3	86.5	-
SwinTrack-B [22]	69.6	78.6	74.1	47.6	58.2	54.1	82.5	87.0	80.4
Mixformer-L [48]	70.1	79.9	76.3	-	-	-	83.9	88.9
MixFormer-22k [48]	69.2	78.7	74.7	-	-	-	83.1	88.1	81.6
AiATrack [49]	69.0	79.4	73.8	47.7	55.6	55.4	82.7	87.8	80.4
ToMP-101 [50]	68.5	-	-	45.9	-	-	81.5	86.4	78.9
GTELT [51]	67.7	-	73.2	45.0	54.2	52.4	82.5	86.7	81.6
KeepTrack [37]	67.1	77.2	70.2	48.2	58.1	56.4	-	-	-
STARK-101 [19]	67.1	77.0	-	-	-	-	82.0	86.9	-
TransT [20]	64.9	73.8	69.0	-	-	-	81.4	86.7	80.3
SiamR-CNN [52]	64.8	72.2	-	-	-	-	81.2	85.4	80.0
TrDiMP [21]	63.9	-	61.4	-	-	-	78.4	83.3	73.1
LTMU [53]	57.2	-	57.2	41.4	49.9	47.3	-	-	-
DiMP [54]	56.9	65.0	56.7	39.2	47.6	45.1	74.0	80.1	68.7
SiamPRN++ [55]	49.6	56.9	49.1	34.0	41.6	39.6	73.3	80.0	69.4
SiamFC [56]	33.6	42.0	33.9	23.0	31.1	26.9	57.1	66.3	53.3

Table 3. Comparison with the state-of-the-art trackers on the UAV123, NFS and TNL2K benchmarks in AUC scores.

Tracker	UAV123	NFS	TNL2K
AdaDe-L256	69.9	67.5	59.1
AdaDe-B256	68.5	67.3	56.2
GRM-L320 [34]	72.2	66.0	-
GRM-B256 [34]	70.2	65.6	-
OSTrack-384 [10]	70.7	66.5	55.9
OSTrack-256 [10]	68.3	64.7	54.3
MixFormer-22k [48]	70.4	-	-
KeepTrack [37]	69.7	66.4	-
STARK-101 [19]	68.2	66.2	-
TransT [20]	68.1	65.3	50.7
TrDiMP [21]	66.4	66.2	-
SiamR-CNN [52]	64.9	63.9	-
SiamPRN++ [55]	59.3	57.1	-

Table 3. Comparison with the state-of-the-art trackers on the UAV123, NFS and TNL2K benchmarks in AUC scores.

Tracker	UAV123	NFS	TNL2K
AdaDe-L256	69.9	67.5	59.1
AdaDe-B256	68.5	67.3	56.2
GRM-L320 [34]	72.2	66.0	-
GRM-B256 [34]	70.2	65.6	-
OSTrack-384 [10]	70.7	66.5	55.9
OSTrack-256 [10]	68.3	64.7	54.3
MixFormer-22k [48]	70.4	-	-
KeepTrack [37]	69.7	66.4	-
STARK-101 [19]	68.2	66.2	-
TransT [20]	68.1	65.3	50.7
TrDiMP [21]	66.4	66.2	-
SiamR-CNN [52]	64.9	63.9	-
SiamPRN++ [55]	59.3	57.1	-

4.3. Other Benchmarks

To further evaluate the robustness and generalization of the proposed tracker, we select four challenging benchmarks focusing on special and challenging scenarios or targets: AVisT [57] with adverse visibility scenarios, UAVDark135 [58] and DarkTrack2021 [59] with night-time scenarios, and TOTB [60] with transparent objects. These scenarios or targets rarely emerge in most tracking datasets, leading to domain gaps and difficulty in tracking.

AVisT [57] is a recently released tracking benchmark with diverse scenarios and adverse visibility, covering severe weather conditions, obstruction effects such as water splashing and unfavorable imaging effects such as low light, distractor objects or camouflage. That benchmark consist of 120 sequences and presents great tracking challenges under complicated conditions.

UAVDark135 [58] and DarkTrack2021 [59] are two night-time tracking benchmarks, containing 135 and 115 videos, respectively. The results of all comparisons are reported in Table 4 and Table 5.

For the adverse visibility tracking dataset AVisT, our AdaDe-B256’s performance only outperforms GRM-B256 [34] with a 0.1% increase, while AdaDe-L256 outperforms AdaDe-B256 with a 5.3% performance increase. Compared to other state-of-the-art trackers, AdaDe-L256 achieves a 2.3% gain over OSTrack-B384 [10] and 4.8% gain over GRM-L320 [34]. This shows the excellent performance of our trackers.

For two night-time datasets, our AdaDe-B256’s performance is basically on par with OSTrack-B256 [10]. Similar to previous conclusions on the AVisT benchmark, performance on these night-time datasets benefits a lot from larger pre-trained weights, indicating that a larger model can indeed bring stronger generalization ability. The AdaDe-L256 tracker exceeds the AdaDe-B256 tracker with a 4.8% increase on the UAVDark135 dataset and a 4.1% increase on the DarkTrack2021 dataset. Although no specific night-time data are utilized for training additionally in our method, we also compare our tracker with DCPT [61] and DiMP-SCT [59] which employ night-time datasets for training. DCPT [61] is a recent night tracker which is designed with prompt tuning with night-time data of BDD100K [62] and SHIFT [63]. DiMP-SCT [59] learns a lowlight enhancer module with paired low/normal light images from the LOL [64] dataset. The performance of the AdaDe-L256 tracker also exceeds DCPT [61] by 2.0% on the UAVDark135 dataset and is only 0.7% behind on the DarkTrack2021 dataset. These comparative results show our better generalization ability in tracking. More detailed analysis on the generalization ability will be unfolded in ablation studies.

Table 4. Comparison with the state-of-the-art trackers on AVisT.

Tracker	AVisT
	AUC	OP50	OP75
AdaDe-L256	59.9	69.6	52.1
AdaDe-B256	54.6	63.3	44.5
OSTrack-B384 [10]	58.1	67.9	48.6
OSTrack-B256 [10]	56.2	65.3	46.5
MixFormerL-22k [48]	56.0	65.9	46.3
MixFormer-22k [48]	53.7	63.0	43.0
GRM-L320 [34]	55.1	63.8	46.9
GRM-B256 [34]	54.5	63.1	45.2
STARK-101 [19]	50.5	58.23	39.0
KeepTrack [37]	49.4	56.3	37.2
TransT [20]	49.0	56.4	37.2
TrDiMP [21]	48.1	55.3	33.8
SiamPRN++ [55]	39.0	43.5	21.2
DiMP [54]	38.6	41.5	22.2

Table 4. Comparison with the state-of-the-art trackers on AVisT.

Tracker	AVisT
	AUC	OP50	OP75
AdaDe-L256	59.9	69.6	52.1
AdaDe-B256	54.6	63.3	44.5
OSTrack-B384 [10]	58.1	67.9	48.6
OSTrack-B256 [10]	56.2	65.3	46.5
MixFormerL-22k [48]	56.0	65.9	46.3
MixFormer-22k [48]	53.7	63.0	43.0
GRM-L320 [34]	55.1	63.8	46.9
GRM-B256 [34]	54.5	63.1	45.2
STARK-101 [19]	50.5	58.23	39.0
KeepTrack [37]	49.4	56.3	37.2
TransT [20]	49.0	56.4	37.2
TrDiMP [21]	48.1	55.3	33.8
SiamPRN++ [55]	39.0	43.5	21.2
DiMP [54]	38.6	41.5	22.2

Table 5. Comparison with the state-of-the-art trackers on the UAVDark135 and DarkTrack2021 benchmarks. The symbol * means that training of the tracker involves additional night-time data. We re-evaluate the performance of DCPT [61] with a Python toolkit.

Tracker	UAVDark135		DarkTrack2021
	AUC	P_norm	AUC	P_norm
AdaDe-L256	59.7	72.0	53.3	63.9
AdaDe-B256	54.9	67.3	49.2	58.5
DCPT * [61]	57.7	70.1	54.0	64.6
DiMP-SCT * [59]	56.2	71.7	52.1	67.7
OSTrack-B256 [10]	55.1	66.2	49.1	60.1
PrDiMP [65]	50.7	63.8	46.4	58.0
DiMP18 [54]	49.4	63.9	47.1	62.0

Table 5. Comparison with the state-of-the-art trackers on the UAVDark135 and DarkTrack2021 benchmarks. The symbol * means that training of the tracker involves additional night-time data. We re-evaluate the performance of DCPT [61] with a Python toolkit.

Tracker	UAVDark135		DarkTrack2021
	AUC	P_norm	AUC	P_norm
AdaDe-L256	59.7	72.0	53.3	63.9
AdaDe-B256	54.9	67.3	49.2	58.5
DCPT * [61]	57.7	70.1	54.0	64.6
DiMP-SCT * [59]	56.2	71.7	52.1	67.7
OSTrack-B256 [10]	55.1	66.2	49.1	60.1
PrDiMP [65]	50.7	63.8	46.4	58.0
DiMP18 [54]	49.4	63.9	47.1	62.0

4.4. Visualizations

We visualize some cases of our tracker and the state-of-art trackers for comparison, as shown in Figure 3. Our tracker performs better than OSTrack-B256 [10] in some cases.

4.5. Effect of Decoder

We evaluate the effects of the decoder part in

Table 6. Ablation studies of generalization analysis. AUC scores are reported. The uparrow represents the improvement and the downarrow represents the decrease.

Variant	Method	LaSOT	LaSOT${}_{\mathbf{ext}}$	AVisT	UAVDark135	DarkTrack2021
L256	+ Adapter Module	71.2	50.6	58.4	58.1	53.8
	+ Adapter Module + Decoder	71.2 (0.0 ↑)	50.3 (0.3 ↓)	59.9 (1.5 ↑)	59.7 (1.6 ↑)	53.3 (0.5 ↓)
B256	+ Adapter Module	68.3	48.7	54.9	53.2	47.3
	+ Adapter Module + Decoder	68.7 (0.4 ↑)	49.5 (0.8 ↑)	54.6 (0.3 ↓)	54.9 (1.7 ↑)	49.2 (1.9 ↑)

. For the B256 variant, adding the decoder brings a performance improvement on almost all tracking benchmarks. The performance improves by 0.4% on LaSOT and 0.8% on LaSOT${}_{\mathrm{ext}}$. For the night-time tracking benchmarks, a 1.7% performance increase is achieved on UAVDark135 and a 1.9% performance increase is achieved on DarkTrack2021. For the L256 variant, there is no significant fluctuation in performance on LaSOT and LaSOT${}_{\mathrm{ext}}$ with the addition of the decoder, while a 1.5% performance increase is achieved on AVisT and a 1.6% performance increase is achieved on UAVDark135. These results are consistent with the B256 variant. The significant performance increase on these challenging benchmarks indicates that the decoder promotes the robustness and generalization of tracking.

4.6. Limitations

Although the proposed method achieves a comparable performance on most tracking benchmarks with the state-of-the-art trackers, it still has some limitations. First, although there may be different impacts due to the distribution of different datasets, the introduction of the pre-trained decoder does not bring a completely stable improvement on all datasets. Secondly, the introduction of the adapter tuning brings more computation costs during inference. An inference-friendly parameter-efficient tuning may be considered and explored.

5. Conclusions

In this work, we present a novel approach to parameter efficiency tuning by migrating pre-trained decoders, to design an effective tracking method. First, we freeze the pre-trained weights of the backbone and then add a dynamic adapter structure for every transformer block for tuning, which makes online fine-tuning effective. Second, we migrate the pre-trained decoder blocks to the tracking head, which is very suitable for object localization. The experimental results demonstrate that the presented tracking algorithm achieves better properties than other competing methods on many challenging tracking benchmarks.