TLPG-Tracker: Joint Learning of Target Localization and Proposal Generation for Visual Tracking

1 Introduction

Visual object tracking is a fundamental problem in computer vision which is widely applied in automatic driving and video surveillance, etc. Given an initial bounding box of an arbitrary target in the first frame, it aims to track the target automatically in every frame that follows. The tracking problem is usually decomposed into a classification task for the target localization and a regression task for the target state estimation. Commonly, the target location and state are represented as the target center and a bounding box.

Mainstream trackers perform tracking via a templatematching strategy which is well-performed in both speed and accuracy. Following the modern trend in object detection, we integrate a general pipeline for template-matching trackers in Figure 1, which uses two siamesed branches and two separate heads for the classification and the box estimation in a tracking problem. The template-matching methods store the target appearance information in a template formulate the tracking problem by learning to generate a score map by correlation operations between features representations of the template and the search region. However, most trackers overlook the inherent relation between target localization with discriminative features and target shape estimation, and try to optimize two subtasks separately. It results in some inevitable shortfalls, e.g., the lack of discriminative abilities in Siamese-like trackers [Li et al., 2018a] and the lack of box estimation capacities in Discriminative Correlation Filter (DCF) methods [Danelljan et al., 2017].

∗Zhi Zhang is the corresponding author.

To overcome aforementioned limitations, we propose a novel architecture to learn target localization and proposal generation jointly in an online learning manner. Inspired by the recently proposed RepPoints [Yang et al., 2019], we introduce a deformable central correlation operation, referred as DCC, to exploit the relation between discriminative features for target-background classification and spatial detailed features for target state estimation. And we propose a two-stage process in which we further refine the proposal and improve robustness. To meet the requirements of the model for both fine-grained target representations and sematic features, we design a balanced neck and an effective feature enhancement module to aggregate multi-level features. Our final tracking architecture is a unified multi-task network with end-to-end learning methods. The main contributions of this paper are three-fold:

• Online learning models with novel deformable central correlations are developed to perform robust target localization and accurate proposal generation jointly;
• An efficient two-stage architecture is designed for more precise box estimation;
• A balanced neck and a feature enhancement module are proposed to extract aggregated features for two-stage tracking.

With the above contributions, our proposed tracker, referred as TLPG-Tracker, establishes a new state-of-the-art in comprehensive experiments while possessing in real-time at over 30 FPS on five challenging benchmarks: LaSOT [Fan et al., 2019], GOT10k [Huang et al., 2018], VOT2018 [Kristan et al., 2018], TrackingNet [Muller et al., 2018], OTB-100 [Wu et al., 2015]. We further provide an extensive experiment to analyze the impact of proposed component with other existing methods.

2 RelatedWork

Most of the modern trackers are performed in a templatematching strategy including DCF approaches [Danelljan et al., 2017; Bhat et al., 2018] and Siamese networks [Bertinetto et al., 2016; Li et al., 2018a]. An effective correlation operation is introduced into visual tracking by seminal work MOOSE [Bolme et al., 2010] to match an explicit template with a search patch with learnable weights in the frequency domain. After the rise of deep learning, DCF-based methods with deep sematic features can achieve impressive robustness in target localization. However, they are hard to make full use of the fine-grained features to estimate target shapes.

In contrast to DCF, the recent Siamese networks [Bertinetto et al., 2016; Li et al., 2018a], which computes cross-correlation similarities between a template and a search region in an embedding space, achieve high accuracy and fast tracking speed. Many Siamese-based trackers can balance target classification and state estimation tasks with their extensive architectures in Figure 1. SiamRPN [Li et al., 2018a] imports an anchor-based region proposal networks from object detection tasks, and its extensions [Zhu et al., 2018; Li et al., 2019] show efficiency of bounding box regression with better discriminative abilities. To gain more accurate box estimation, ATOM [Danelljan et al., 2019] employs an IoU-Net [Jiang et al., 2018] based module to predict Intersection-over-Union (IoU) overlap and select best proposal boxes. More recently, an online learning discriminant models with Siamese-based structure are proposed [Bhat et al., 2019] to make up for the deficiency of discriminative abilities in Siamese-based methods. Overall, they still optimize the classification and estimation separately.

3 Method

In this work, we develop a novel joint learning architecture to exploit the relation between target localization and proposal generation with two main principles: (1) the most discriminative features of a target usually locate in the vicinity of the target center while the target bounding box is supported by fine-granted features near target borders to fit its shapes; (2) the target-background classification task and the proposal generation and regression task are complementary in tracking. In terms of the general pipeline, we split our Siameselike network into a feature exactor with a neck module, online learning models which perform target localization and proposal generation by deformable central correlation and a refinement stage with a feature enhancement module, shown in Figure 2.

In our two branches framework, feature maps for the test and the reference branches are extracted from backbone network with a balanced neck module which fuses the sematic and spatial information from multi-layer features. The target models are constituted by deformable convolution kernels which are learned online on the target template Z (reference) during inference time and perform two-stage tracking. In stage 1, the target in a test frame is tracked by deformable central correlation between the model f and a feature map x of the search region (test) which finds coordinates where suitable target proposal boxes are formed. Followed by a feature enhance module which aggregates multi-level features, the proposal boxes are refined by another deformable central correlation on the enhanced feature x′ in stage 2. We get final output bounding boxes with optional post-process, e.g., non-maximum suppression as NMS and IoU-Net. We update weights of the target models by an online-training optimizer module P .

3.1 Deformable Convolution in Visual Tracking

In order to facilitate the introduction of our target models, we briefly revisit deformable convolution [Dai et al., 2017] in the context of visual tracking. For notation clarity, we describe

Predictor Test the convolution modules in 2D spatial domain, and the operation remains the same in each channel in 3D case.

Deformable convolution can be regarded as a self-attention instantiations of spatial attention mechanisms [Zhu et al., 2019]. From the aspect of query-key pairs in spatial attention, an element on each coordinate of a feature map is regarded as a query element while the elements on sampled coordinates are regarded as key elements. Based on regular convolution, which has fixed sampling grid region R and convolutional weight w, the deformable convolution models the relation between a query with its corresponding keys by augmenting learnable 2D offsets {∆pk}Ns k=1 to R on integral spatial locations in a feature map x ∈ RW×H which are stored in offset field o ∈ RW×H×2NS . Ns refers to the number of sample points in a deformable convolution kernel. For each coordinate pi on output feature map y ∈ RW×H we have output as

In practice, a bilinear interpolation kernel A(., .) is used to map the fractional p = pi+pr+∆pk to an integral coordinate.

In generic tracking, it is a common sense that coarse localization of the target center is much easier than finding an accurate target bounding box. The main reason is that features near the border of the target which determine the shape of the bounding box are unstable and usually mixed with background, while features near the target center are more constant and fixed to some extent. Intuitively, we can learn the relation between discriminative features near target centers and features for target shapes by deformable convolution to generate proper bounding boxes. Inspired by RepPoints, we collect the sampled points of deformable convolution at the coordinate p0 in a feature map as

where pu ∈ R2 refers to a position on feature map x. We define a converting function Γ : Pkey → P, to transform this point-based representation to a proposal bounding box. Specially, the proposal P is formed by the distances from the coordinate p0 to the left-top and bottom-right points as 4D real vector (l, t, r, b). We adopt a min-max function, which performs min-max operation over two axes of Pkey to determine left-top and bottom-right points of P, as the function Γ.

3.2 Deformable Central Correlation

In this section, we detail how to learn target models and perform an online learning deformable central correlation for tracking. Generally, given a set of training sample Strain = {(xi,Bi)}ni=1 , where xi is a feature map from the feature extractor and Bi = (li, ti, ri, bi) is the 4D corresponding bounding box, the target model f is learned online or offline. In recent template-matching trackers, there are a classification task and an estimation task. As for the classification, it usually performs correlation between the template and the input feature map xi by the model f to discriminate the explicit target from background distractors, and outputs a score map s = xi ∗ f where s ∈ RW×H and ∗ denotes correlation operation. The online learning classification loss is formulated as

where the function d(s, yi) computes the residual at every spatial location on the predicted score map s and a corresponding label yi, and λ is a regularization factor. As for the estimation, the traditional methods generate Np proposal bounding boxes {Pit}Np t=1 by anchor strategies SiamRPN. The regression loss is formulated as

where the function r (Pit, Bi) computes the residual between the proposal and a bounding box. As mentioned in 3.1, we adopt a deformable convolution layer in the online learning model f which is defined as f (x; w, o) and aim to find a coordinate p∗ which generates the most precise proposal bounding box P ∗ . The deformable central correlation shown in Figure 3 is carefully designed with an observation that the most representative features are always near the center of mass of the target rather than the bounding box center.

Proposal Generation. Given a location pj on the feature map xi, our model will generate a proposal bounding box Pj with sampled points. We expect to generate accurate proposal bounding boxes of the target on discriminative features around the vicinity of the target center. So, we use two IoU thresholds θhi and θlo on the classification label yi to separate positive and negative training samples. For a positive sample pj = (px, py) which has IoU(Pj,Bj) ≥ θhi, the regression target P ∗ j is formulated as

We adopt the smooth l1 distance to the function r(Pj, P ∗ j ) = Smoothl1(Pj , P ∗ j ) which is easy to optimize online. For a negative sample pj which has yi(pj) < θlo, we simply minimize the IoU score which is non-negative between Pj and Bj using the IoU loss as the function r(Pj,Bj) = IoU(Pj,Bj) in offline training.

Target Localization. To ensure the discriminative capacity, the model f is commonly defined as a discriminative filter to locate the target center (cx, cy). However, this definition is usually faces two problems: (1) data imbalance which makes the model mainly focus on easy negatives; (2) inherent ambiguity of the classification label yi. Benefitted from our proposal generation, the center of mass of the target can be roughly clustered. Thus, we define CRi(cx, cy, σcwb, σchb) as the center region of the bounding box Bi, which roughly covers potential target centers, with a shrink scale σc ∈ (0, 1). Note that the label of each spatial coordinate inside CRi is set to 1 while outside the bounding box Bi is set 0. We ignore the region between CRi and Bi. The intermediate coordinates are smoothed with Gaussian distribution centered at the box center. We formulate the function d(s, yi) in a hinge-like form [Bhat et al., 2019],

Both the loss (3) and (4) are applied in online and offline training. The predicted scores of the proposal which reflect both localization and estimation are more suitable for the NMS procedure.

3.3 Center-bias Refinement with Feature Enhancement

To get more accurate results, we introduce a two-stage refinement method with a multi-layer feature enhancement module. Note that the ResNet architecture is employed as backbone and the feature map x has a spatial stride of 16.

Center-bias Refinement. Our ablation experiments show that multi-stage bounding box regression, which learns the bias between the proposals and the ground truth, yields poor performance (see section 4.1). Unlike the regression methods with fixed centers in anchor-based strategies, the proposed method gains refinement with the bias of the target center locations. As shown in Figure 3, we pay more attention to search representative target features and the target center in the region of interest in stage 1, and focus on target aligning in stage 2 with fewer proposals. We select top Nc centers of the proposal bounding boxes in stage 1 by their IoU scores. After removing the duplicate locations, they are used as new centers to generate new proposals in stage 2.

Feature Enhancement. As we discussed in section3.2, the deformable central correlation actually requires fine-grained and sematic features. Although the feature map xS1 gains more target-related features after stage 1, it losses some finegrained sematic features around the target because of the discriminant ability of the model. Thus, we use a multiresolution aggregation method to fuse fine-grained features from multi-level feature maps including C3, C4, C5 with spatial stride of 8, 16, 32, shown in Figure 4. We import refined features form C ′3 and C ′5 directly from the neck (see section3.4) like residual blocks to form three parallel branches. Then we fuse three parallel branches in a fully-connection manner and combine them to get the enhanced feature map xE. Thus, we use a multi-resolution aggregation method to fuse fine-grained features from multi-level feature maps, as shown in Figure 4. Since more spatial details of target appearance are in lower convolution layers, we import highresolution feature map xF1 directly from backbone like a residual block to keep spatial features. We form three parallel branches to get fully aggregated features. To keep the rich sematic features of the target in current feature map xS1, we perform regular convolution on xS1 to get xS1 ′ with low resolutions. We use 1 × 1 convolution to unified the number of channels in xF1 and xS1 ′ . Then we fuse three parallel branches in a fully-connection manner and combine them to get the enhanced feature map xE.

3.4 Balanced Neck

The proposed balanced neck module serves as a connection between a backbone and head networks, and outputs a single feature map which balances spatial details and robust sematic features for the head network.

We observe that different feature levels at different resolutions of a backbone usually contribute unequally to the fused output. Thus, we design a simple neck structure with bounded weight ni (per-feature), where Σini = 1. As for the ResNet architecture, we first unify the multi-level features C3, C4, C5 to the same channel number as C4 with 1 × 1 convolution as C ′3 , C ′4 , C ′5 . Unlike the bottom-top integrating strategy used in FPN [Lin et al., 2017], we balance semantic and fine-grained features by rescaling and averaging them with a scalar weight as

Note that C3 and C5 are resized to C4 as C ′3 and C ′5 by average-pooling and linear interpolation after 1 × 1 convolution respectively.

3.5 Implementation Details

As for online learning, we set the kernel size of DCC to 5×5 in our target models for two stages. To speed up convergence of online training, we use Steepest Descent (SD) in DiMP. We train 10 iterations at the first frame as initialization. The models are updated in an optimizer module P every 25 frames during inference, and store 50 most recent tracked frames as the template. As for offline training, we train our architecture effectively with a mixed training set created from TrackingNet, LaSOT and GOT10k. We sample various classes of images to form a mini-batch Mtrain and Mtest to train the entire networks for 60 epochs with ADAM [Kingma and Ba, 2015] optimizer with learning rate decay of 0.2 every 15 epochs. We set σc = 0.3 for classification labels and set θhi = 0.6 and θlo = 0.5. In stage 2, we set the maximum of Nc to 25.

4 Experiments

Our new TLPG-tracker is implemented in Python with PyTorch and evaluated on five challenging tracking benchmarks: LaSOT [Fan et al., 2019], VOT2018 [Kristan et al., 2018], GOT10k [Huang et al., 2018], TrackingNet [Muller et al., 2018], OTB2015 [Wu et al., 2015]. Employing with ResNet50 as backbone, we achieve a tracking speed of 36 FPS with IoU-Net and 41 FPS with NMS on a single GeForce RTX 2080 Ti GPU.

4.1 Ablation Study

In this section, we analyze the effectiveness of three proposed components in TLPG-tracker with a combined dataset which contains 200 various video sequences randomly sampled from LaSOT, VOT2018 and OTB2015 datasets. The AUC metric is used in the evaluation.

Deformable Central Correlation. We investigate the impact of key aspects of the proposed deformable central correlation by incrementally adding each part of the module at a time which shows in Table 1. The baseline [Bhat et al., 2019] contains an online learning regular convolution version filter (RC) with IoU-Net (IoU) as post-process. By replacing the filter with an online learning raw deformable convolution (DC), it gains a 0.3% AUC score on the combined dataset. Adding a proposal boxes regression loss (Reg) to the deformable convolution filter improves the AUC score by 0.8%. The deformable central correlation (DCC), which is performed only once during inference for fair comparisons, leads to a major improvement of 1.2% in AUC score with NMS and 1.5% in AUC score with IoU-Net. It shows that the combination of the localization, the representative target features and target shapes with the online learning loss fully exploit the correlation between target localization and shape generation. The deformable central correlation suits the post process well with a total improvement of 2.6% AUC score.

Center-bias Refinement with Feature Enhancement. We compare the proposed refinement method with classical bounding box regression. Here, NMS can indicate whether predicted scores of the proposal reflect the accuracy of both target localization and estimation while IoU-Net (IoU) showing the best accuracy of the proposal. We form a basic regression version (BReg) by replacing our center-bias refinement (CReg) with bounding box regression which refines the residual between proposal boxes and corresponding ground truth with extra convolution layers. Refinements are formed by using the feature enhancement (EN) or not, as shown in Table 2. Without enhancement, the center-bias refinement outperforms bounding box regression by 0.3% AUC score with NMS. The feature enhancement module gains up to 0.4% with center-bias refinement and NMS, but improves little with bounding box regression. It means that the center-bias refinement is compatible with NMS procedure.

Balanced Neck. It is crucial to choose proper feature maps from backbone in tracking tasks. We compare our balanced neck module with FPN [Lin et al., 2017] on two similar backbones, ResNet-50 and DetNet-59 [Li et al., 2018b], used in image classification and object detection, to analyze the impacts of resolution and receptive fields. Backbone feature layers conv3, conv4, conv5 are involved. Using features of same spatial stride from neck modules or backbone as output feature maps, the performance of the Balanced Neck and FPN is shown in Table 3. The proposed balanced neck with ResNet-50 gains 1.3% AUC score improvement on the combined dataset and is 0.7% higher than FPN, which indicates features with balanced spatial details and sematic information are more suitable for object tracking. Note that DetNet-59 preforms better than ResNet-50 by 0.3% without a neck because it maintains high resolution in conv4 and conv5 while FPN aggregates the adjacent feature levels in a bottom-up manner. But they are less likely to provide balanced features.

4.2 State-of-the-art Comparison

The proposed TLPG-Tracker, which employs the backbone ResNet-50 and the post-process IoU-Net, is compared with the state-of-the-art methods including DiMP [Bhat et al., 2019], SiamRPN++ [Li et al., 2019], SiamMask [Wang et al., 2019b], ATOM [Danelljan et al., 2019], SPM [Wang et al., 2019a], LADCF [Xu et al., 2019], SiamGragh [Tu et al., 2019], SiamRPN R18 [Li et al., 2018a], MFT[Kristan et al., 2018], UPDT [Bhat et al., 2018], StructSiam [Zhang et al., 2018], MHIT [Bai et al., 2018], VITAL [Song et al., 2018], DRT [Sun et al., 2018], DaSiamRPN [Dai et al., 2017], BACF [Kiani Galoogahi et al., 2017], ECO [Danelljan et al., 2017], DSiam[Guo et al., 2017], SiamFCv2 [Valmadre et al., 2017], GOTURN [Held et al., 2016], SiamFC [Bertinetto et al., 2016], CCOT[Danelljan et al., 2016], MDNet [Nam and Han, 2016] on following five challenging tracking benchmarks.

ble 5. Our method outperforms the most progressive performance approaches DiMP with average overlap (AO) scores of 62.9% and success rates (SR) at thresholds 0.5 of 73.5%.

TrackingNet. TrackingNet is a huge dataset for target tracking, containing more than 30, 000 videos. As the results shown in Table 6, our TLPG-Tracker outperforms most advanced algorithms on TrackingNet test set, with the best precision rate of 70.6% and success score of 75.8%.

OTB2015. The standardizedOTB2015 benchmark containing 100 videos with various challenging factors provides a fair testbed on robustness. Among all compared methods in Table 7, our TLPG-Tracker achieves a competitive result with an AUC score of 69.9% while UPDT obtaining the best performance with an AUC score of 70.2%.

5 Conclusion

We propose a novel two-stage tracking architecture that learn both target localization and proposal generation simultaneously by deformable central correlation in online training manner. The proposals are generated on discriminative features near target centers in stage 1 and refined by centerbias refinement to provide more accurate results. The model benefits from multi-level aggregation of fine-grained and sematic features in a neck and a feature enhancement module. With robust localization and accurate estimation, our TLPGTracker sets a new state-of-the-art on 5 datasets while operating at over 30 FPS.