---
title: "TLPG-Tracker: Joint Learning of Target Localization and Proposal Generation for Visual Tracking"
authors:
  - "Siyuan Li"
  - "Zhi Zhang"
  - "Ziyu Liu"
  - "Anna Wang"
  - "Linglong Qiu"
  - "Feng Du"
doi: "10.24963/ijcai.2020/99"
url: "https://ijcai.org/proceedings/2020/99"
pdf: "https://ijcai.org/proceedings/2020/0099.pdf"
venue: "Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence"
year: 2020
pages: "708-715"
track: "Main track"
keywords:
  - "Machine Learning: Deep Learning"
  - "Computer Vision: Motion and Tracking"
  - "Machine Learning: Deep Learning: Convolutional networks"
  - "Computer Vision: Video: Events, Activities and Surveillance"
---

# TLPG-Tracker: Joint Learning of Target Localization and Proposal Generation for Visual Tracking

Siyuan Li, Zhi Zhang, Ziyu Liu, Anna Wang, Linglong Qiu, Feng Du

*Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence*, pages 708-715. DOI: [10.24963/ijcai.2020/99](https://doi.org/10.24963/ijcai.2020/99)

## Abstract

Target localization and proposal generation are two essential subtasks in generic visual tracking, and it is a challenge to address both the two efficiently. In this paper, we propose an efficient two-stage architecture which makes full use of the complementarity of two subtasks to achieve robust localization and high-quality proposals generation of the target jointly. Specifically, our model performs a novel deformable central correlation operation by an online learning model in both two stages to locate new target centers while generating target proposals in the vicinity of these centers. The proposals are refined in the refinement stage to further improve accuracy and robustness. Moreover, the model benefits from multi-level features aggregation in a neck module and a feature enhancement module. We conduct extensive ablation studies to demonstrate the effectiveness of our proposed methods. Our tracker runs at over 30 FPS and sets a new state-of-the-art on five tracking benchmarks, including LaSOT, VOT2018, TrackingNet, GOT10k, OTB2015.

## Full text

## 1 Introduction

Visual object tracking is a fundamental problem in computer vision which is widely applied in automatic driving and video surveillance, etc. Given an initial bounding box of an arbitrary target in the first frame, it aims to track the target automatically in every frame that follows. The tracking problem is usually decomposed into a classification task for the target localization and a regression task for the target state estimation. Commonly, the target location and state are represented as the target center and a bounding box.

Mainstream trackers perform tracking via a templatematching strategy which is well-performed in both speed and accuracy. Following the modern trend in object detection, we integrate a general pipeline for template-matching trackers in Figure 1, which uses two siamesed branches and two separate heads for the classification and the box estimation in a tracking problem. The template-matching methods store the target appearance information in a template formulate the tracking problem by learning to generate a score map by correlation operations between features representations of the template and the search region. However, most trackers overlook the inherent relation between target localization with discriminative features and target shape estimation, and try to optimize two subtasks separately. It results in some inevitable shortfalls, e.g., the lack of discriminative abilities in Siamese-like trackers [Li et al., 2018a] and the lack of box estimation capacities in Discriminative Correlation Filter (DCF) methods [Danelljan et al., 2017].

∗Zhi Zhang is the corresponding author.

Figure 1: An overview of a general template-matching pipeline in a Siamese-like way. Sharing the same backbone and neck as the feature extractor, a test branch is used for tracking the test frame while a reference branch stores template from reference frames. Some update methods are employed in online training trackers. The classification (localization) and the proposal boxes regression (estimation) are performed separately in subnet heads. The correlation operation between the feature map x and the target model is the core operation in these trackers.

![Image](https://ijcai.org/proceedings/2020/fulltext/99/image_000000_028e1e0ae7b68c171214aa3204c39d08f24f8602c32a5139f5a09f8000bff0d3.png)

To overcome aforementioned limitations, we propose a novel architecture to learn target localization and proposal generation jointly in an online learning manner. Inspired by the recently proposed RepPoints [Yang et al., 2019], we introduce a deformable central correlation operation, referred as DCC, to exploit the relation between discriminative features for target-background classification and spatial detailed features for target state estimation. And we propose a two-stage process in which we further refine the proposal and improve robustness. To meet the requirements of the model for both fine-grained target representations and sematic features, we design a balanced neck and an effective feature enhancement module to aggregate multi-level features. Our final tracking architecture is a unified multi-task network with end-to-end learning methods. The main contributions of this paper are three-fold:

nt Figure 2: An overview of our tracking architecture. During inference, a test frame is input to the test branch (bottom left), and proposal bounding boxes are the outputs of a two-stage generation and refinement process with DCC1 and DCC2 which refer to deformable central correlation in stage 1 and 2. The final output is the proposal with the highest IoU score or predicted score.

![Image](https://ijcai.org/proceedings/2020/fulltext/99/image_000001_dfbdf6730b2265820c6c2057c194a3161a55a4de5b5ee097c95469e9af107012.png)

- Online learning models with novel deformable central correlations are developed to perform robust target localization and accurate proposal generation jointly;
- An efficient two-stage architecture is designed for more precise box estimation;
- A balanced neck and a feature enhancement module are proposed to extract aggregated features for two-stage tracking.

With the above contributions, our proposed tracker, referred as TLPG-Tracker, establishes a new state-of-the-art in comprehensive experiments while possessing in real-time at over 30 FPS on five challenging benchmarks: LaSOT [Fan et al., 2019], GOT10k [Huang et al., 2018], VOT2018 [Kristan et al., 2018], TrackingNet [Muller et al., 2018], OTB-100 [Wu et al., 2015]. We further provide an extensive experiment to analyze the impact of proposed component with other existing methods.

## 2 RelatedWork

Most of the modern trackers are performed in a templatematching strategy including DCF approaches [Danelljan et al., 2017; Bhat et al., 2018] and Siamese networks [Bertinetto et al., 2016; Li et al., 2018a]. An effective correlation operation is introduced into visual tracking by seminal work MOOSE [Bolme et al., 2010] to match an explicit template with a search patch with learnable weights in the frequency domain. After the rise of deep learning, DCF-based methods with deep sematic features can achieve impressive robustness in target localization. However, they are hard to make full use of the fine-grained features to estimate target shapes.

In contrast to DCF, the recent Siamese networks [Bertinetto et al., 2016; Li et al., 2018a], which computes cross-correlation similarities between a template and a search region in an embedding space, achieve high accuracy and fast tracking speed. Many Siamese-based trackers can balance target classification and state estimation tasks with their extensive architectures in Figure 1. SiamRPN [Li et al., 2018a] imports an anchor-based region proposal networks from object detection tasks, and its extensions [Zhu et al., 2018; Li et al., 2019] show efficiency of bounding box regression with better discriminative abilities. To gain more accurate box estimation, ATOM [Danelljan et al., 2019] employs an IoU-Net [Jiang et al., 2018] based module to predict Intersection-over-Union (IoU) overlap and select best proposal boxes. More recently, an online learning discriminant models with Siamese-based structure are proposed [Bhat et al., 2019] to make up for the deficiency of discriminative abilities in Siamese-based methods. Overall, they still optimize the classification and estimation separately.

## 3 Method

In this work, we develop a novel joint learning architecture to exploit the relation between target localization and proposal generation with two main principles: (1) the most discriminative features of a target usually locate in the vicinity of the target center while the target bounding box is supported by fine-granted features near target borders to fit its shapes; (2) the target-background classification task and the proposal generation and regression task are complementary in tracking. In terms of the general pipeline, we split our Siameselike network into a feature exactor with a neck module, online learning models which perform target localization and proposal generation by deformable central correlation and a refinement stage with a feature enhancement module, shown in Figure 2.

In our two branches framework, feature maps for the test and the reference branches are extracted from backbone network with a balanced neck module which fuses the sematic and spatial information from multi-layer features. The target models are constituted by deformable convolution kernels which are learned online on the target template Z (reference) during inference time and perform two-stage tracking. In stage 1, the target in a test frame is tracked by deformable central correlation between the model f and a feature map x of the search region (test) which finds coordinates where suitable target proposal boxes are formed. Followed by a feature enhance module which aggregates multi-level features, the proposal boxes are refined by another deformable central correlation on the enhanced feature x′ in stage 2. We get final output bounding boxes with optional post-process, e.g., non-maximum suppression as NMS and IoU-Net. We update weights of the target models by an online-training optimizer module P .

## 3.1 Deformable Convolution in Visual Tracking

In order to facilitate the introduction of our target models, we briefly revisit deformable convolution [Dai et al., 2017] in the context of visual tracking. For notation clarity, we describe

Predictor Test the convolution modules in 2D spatial domain, and the operation remains the same in each channel in 3D case.

Figure 3: An overview of the proposal generation and refinement module. With a feature map x, discriminative features localization and proposal generation are performed by deformable central correlation near the previous target center in stage 1. Refinement are performed on top Nc proposals center points in stage 1 after the feature enhancement to get refined bounding boxes in stage 2.

![Image](https://ijcai.org/proceedings/2020/fulltext/99/image_000002_1dc9e1626330942092b6261c30079174c462c2056a9c8545a82046f5835ffa47.png)

Deformable convolution can be regarded as a self-attention instantiations of spatial attention mechanisms [Zhu et al., 2019]. From the aspect of query-key pairs in spatial attention, an element on each coordinate of a feature map is regarded as a query element while the elements on sampled coordinates are regarded as key elements. Based on regular convolution, which has fixed sampling grid region R and convolutional weight w, the deformable convolution models the relation between a query with its corresponding keys by augmenting learnable 2D offsets {∆pk}Ns k=1 to R on integral spatial locations in a feature map x ∈ RW×H which are stored in offset field o ∈ RW×H×2NS . Ns refers to the number of sample points in a deformable convolution kernel. For each coordinate pi on output feature map y ∈ RW×H we have output as

<!-- formula-not-decoded -->

In practice, a bilinear interpolation kernel A(., .) is used to map the fractional p = pi+pr+∆pk to an integral coordinate.

In generic tracking, it is a common sense that coarse localization of the target center is much easier than finding an accurate target bounding box. The main reason is that features near the border of the target which determine the shape of the bounding box are unstable and usually mixed with background, while features near the target center are more constant and fixed to some extent. Intuitively, we can learn the relation between discriminative features near target centers and features for target shapes by deformable convolution to generate proper bounding boxes. Inspired by RepPoints, we collect the sampled points of deformable convolution at the coordinate p0 in a feature map as

<!-- formula-not-decoded -->

where pu ∈ R2 refers to a position on feature map x. We define a converting function Γ : Pkey → P, to transform this point-based representation to a proposal bounding box. Specially, the proposal P is formed by the distances from the coordinate p0 to the left-top and bottom-right points as 4D real vector (l, t, r, b). We adopt a min-max function, which performs min-max operation over two axes of Pkey to determine left-top and bottom-right points of P, as the function Γ.

## 3.2 Deformable Central Correlation

In this section, we detail how to learn target models and perform an online learning deformable central correlation for tracking. Generally, given a set of training sample Strain = {(xi,Bi)}ni=1 , where xi is a feature map from the feature extractor and Bi = (li, ti, ri, bi) is the 4D corresponding bounding box, the target model f is learned online or offline. In recent template-matching trackers, there are a classification task and an estimation task. As for the classification, it usually performs correlation between the template and the input feature map xi by the model f to discriminate the explicit target from background distractors, and outputs a score map s = xi ∗ f where s ∈ RW×H and ∗ denotes correlation operation. The online learning classification loss is formulated as

<!-- formula-not-decoded -->

where the function d(s, yi) computes the residual at every spatial location on the predicted score map s and a corresponding label yi, and λ is a regularization factor. As for the estimation, the traditional methods generate Np proposal bounding boxes {Pit}Np t=1 by anchor strategies SiamRPN. The regression loss is formulated as

<!-- formula-not-decoded -->

where the function r (Pit, Bi) computes the residual between the proposal and a bounding box. As mentioned in 3.1, we adopt a deformable convolution layer in the online learning model f which is defined as f (x; w, o) and aim to find a coordinate p∗ which generates the most precise proposal bounding box P ∗ . The deformable central correlation shown in Figure 3 is carefully designed with an observation that the most representative features are always near the center of mass of the target rather than the bounding box center.

Proposal Generation. Given a location pj on the feature map xi, our model will generate a proposal bounding box Pj with sampled points. We expect to generate accurate proposal bounding boxes of the target on discriminative features around the vicinity of the target center. So, we use two IoU thresholds θhi and θlo on the classification label yi to separate positive and negative training samples. For a positive sample pj = (px, py) which has IoU(Pj,Bj) ≥ θhi, the regression target P ∗ j is formulated as

<!-- formula-not-decoded -->

We adopt the smooth l1 distance to the function r(Pj, P ∗ j ) = Smoothl1(Pj , P ∗ j ) which is easy to optimize online. For a negative sample pj which has yi(pj) &lt; θlo, we simply minimize the IoU score which is non-negative between Pj and Bj using the IoU loss as the function r(Pj,Bj) = IoU(Pj,Bj) in offline training.

Target Localization. To ensure the discriminative capacity, the model f is commonly defined as a discriminative filter to locate the target center (cx, cy). However, this definition is usually faces two problems: (1) data imbalance which makes the model mainly focus on easy negatives; (2) inherent ambiguity of the classification label yi. Benefitted from our proposal generation, the center of mass of the target can be roughly clustered. Thus, we define CRi(cx, cy, σcwb, σchb) as the center region of the bounding box Bi, which roughly covers potential target centers, with a shrink scale σc ∈ (0, 1). Note that the label of each spatial coordinate inside CRi is set to 1 while outside the bounding box Bi is set 0. We ignore the region between CRi and Bi. The intermediate coordinates are smoothed with Gaussian distribution centered at the box center. We formulate the function d(s, yi) in a hinge-like form [Bhat et al., 2019],

<!-- formula-not-decoded -->

Both the loss (3) and (4) are applied in online and offline training. The predicted scores of the proposal which reflect both localization and estimation are more suitable for the NMS procedure.

## 3.3 Center-bias Refinement with Feature Enhancement

To get more accurate results, we introduce a two-stage refinement method with a multi-layer feature enhancement module. Note that the ResNet architecture is employed as backbone and the feature map x has a spatial stride of 16.

Center-bias Refinement. Our ablation experiments show that multi-stage bounding box regression, which learns the bias between the proposals and the ground truth, yields poor performance (see section 4.1). Unlike the regression methods with fixed centers in anchor-based strategies, the proposed method gains refinement with the bias of the target center locations. As shown in Figure 3, we pay more attention to search representative target features and the target center in the region of interest in stage 1, and focus on target aligning in stage 2 with fewer proposals. We select top Nc centers of the proposal bounding boxes in stage 1 by their IoU scores. After removing the duplicate locations, they are used as new centers to generate new proposals in stage 2.

Feature Enhancement. As we discussed in section3.2, the deformable central correlation actually requires fine-grained and sematic features. Although the feature map xS1 gains more target-related features after stage 1, it losses some finegrained sematic features around the target because of the discriminant ability of the model. Thus, we use a multiresolution aggregation method to fuse fine-grained features from multi-level feature maps including C3, C4, C5 with spatial stride of 8, 16, 32, shown in Figure 4. We import refined features form C ′3 and C ′5 directly from the neck (see section3.4) like residual blocks to form three parallel branches. Then we fuse three parallel branches in a fully-connection manner and combine them to get the enhanced feature map xE. Thus, we use a multi-resolution aggregation method to fuse fine-grained features from multi-level feature maps, as shown in Figure 4. Since more spatial details of target appearance are in lower convolution layers, we import highresolution feature map xF1 directly from backbone like a residual block to keep spatial features. We form three parallel branches to get fully aggregated features. To keep the rich sematic features of the target in current feature map xS1, we perform regular convolution on xS1 to get xS1 ′ with low resolutions. We use 1 × 1 convolution to unified the number of channels in xF1 and xS1 ′ . Then we fuse three parallel branches in a fully-connection manner and combine them to get the enhanced feature map xE.

Figure 4: Architecture of the balanced neck and the feature enhancement module. We aggregate multi-level feature maps with the balanced neck by rescaling and explore sematic features and spatial details for the refinement stage in a full-connected manner.

![Image](https://ijcai.org/proceedings/2020/fulltext/99/image_000003_971fdd644f19af2201d3fe800dda80cdf48fe799b3ad5fac73d200fd161b2960.png)

![Image](https://ijcai.org/proceedings/2020/fulltext/99/image_000004_3dd548c45bbf0d30dcac20e61e0de14f3270d3bf54ce9eb7410e2d9b17d054fa.png)

## 3.4 Balanced Neck

The proposed balanced neck module serves as a connection between a backbone and head networks, and outputs a single feature map which balances spatial details and robust sematic features for the head network.

We observe that different feature levels at different resolutions of a backbone usually contribute unequally to the fused output. Thus, we design a simple neck structure with bounded weight ni (per-feature), where Σini = 1. As for the ResNet architecture, we first unify the multi-level features C3, C4, C5 to the same channel number as C4 with 1 × 1 convolution as C ′3 , C ′4 , C ′5 . Unlike the bottom-top integrating strategy used in FPN [Lin et al., 2017], we balance semantic and fine-grained features by rescaling and averaging them with a scalar weight as

<!-- formula-not-decoded -->

Note that C3 and C5 are resized to C4 as C ′3 and C ′5 by average-pooling and linear interpolation after 1 × 1 convolution respectively.

## 3.5 Implementation Details

As for online learning, we set the kernel size of DCC to 5×5 in our target models for two stages. To speed up convergence of online training, we use Steepest Descent (SD) in DiMP. We train 10 iterations at the first frame as initialization. The models are updated in an optimizer module P every 25 frames during inference, and store 50 most recent tracked frames as the template. As for offline training, we train our architecture effectively with a mixed training set created from TrackingNet, LaSOT and GOT10k. We sample various classes of images to form a mini-batch Mtrain and Mtest to train the entire networks for 60 epochs with ADAM [Kingma and Ba, 2015] optimizer with learning rate decay of 0.2 every 15 epochs. We set σc = 0.3 for classification labels and set θhi = 0.6 and θlo = 0.5. In stage 2, we set the maximum of Nc to 25.

Table 1: Analysis of the deformable central correlation on the combined datasets. The baseline and further adding components are trained with pretrained backbone ResNet-50 and the balanced neck.

|        |   RC+IoU |   DC+IoU |   DC+Reg+IoU |   DCC+NMS |   DCC+IoU |
|--------|----------|----------|--------------|-----------|-----------|
| AUC(%) |     57.7 |     58.0 |         58.8 |      60.0 |      60.3 |

Table 2: Comparison of different proposal refinement methods and analysis of the feature enhancement on the combined datasets.

|     |   Single Stage |   BReg |   BReg+EN |   CReg |   CReg+EN |
|-----|----------------|--------|-----------|--------|-----------|
| NMS |           60.0 |   60.0 |      60.1 |   60.3 |      60.7 |
| IoU |           60.3 |   60.4 |      60.5 |   60.6 |      60.8 |

## 4 Experiments

Our new TLPG-tracker is implemented in Python with PyTorch and evaluated on five challenging tracking benchmarks: LaSOT [Fan et al., 2019], VOT2018 [Kristan et al., 2018], GOT10k [Huang et al., 2018], TrackingNet [Muller et al., 2018], OTB2015 [Wu et al., 2015]. Employing with ResNet50 as backbone, we achieve a tracking speed of 36 FPS with IoU-Net and 41 FPS with NMS on a single GeForce RTX 2080 Ti GPU.

## 4.1 Ablation Study

In this section, we analyze the effectiveness of three proposed components in TLPG-tracker with a combined dataset which contains 200 various video sequences randomly sampled from LaSOT, VOT2018 and OTB2015 datasets. The AUC metric is used in the evaluation.

Deformable Central Correlation. We investigate the impact of key aspects of the proposed deformable central correlation by incrementally adding each part of the module at a time which shows in Table 1. The baseline [Bhat et al., 2019] contains an online learning regular convolution version filter (RC) with IoU-Net (IoU) as post-process. By replacing the filter with an online learning raw deformable convolution (DC), it gains a 0.3% AUC score on the combined dataset. Adding a proposal boxes regression loss (Reg) to the deformable convolution filter improves the AUC score by 0.8%. The deformable central correlation (DCC), which is performed only once during inference for fair comparisons, leads to a major improvement of 1.2% in AUC score with NMS and 1.5% in AUC score with IoU-Net. It shows that the combination of the localization, the representative target features and target shapes with the online learning loss fully exploit the correlation between target localization and shape generation. The deformable central correlation suits the post process well with a total improvement of 2.6% AUC score.

Center-bias Refinement with Feature Enhancement. We compare the proposed refinement method with classical bounding box regression. Here, NMS can indicate whether predicted scores of the proposal reflect the accuracy of both target localization and estimation while IoU-Net (IoU) showing the best accuracy of the proposal. We form a basic regression version (BReg) by replacing our center-bias refinement (CReg) with bounding box regression which refines the residual between proposal boxes and corresponding ground truth with extra convolution layers. Refinements are formed by using the feature enhancement (EN) or not, as shown in Table 2. Without enhancement, the center-bias refinement outperforms bounding box regression by 0.3% AUC score with NMS. The feature enhancement module gains up to 0.4% with center-bias refinement and NMS, but improves little with bounding box regression. It means that the center-bias refinement is compatible with NMS procedure.

Table 3: Comparison of different neck modules on the combined datasets.

|           |   NONECK |   FPN |   Balanced Neck |
|-----------|----------|-------|-----------------|
| ResNet-50 |     59.5 |  60.1 |            60.8 |
| DetNet-59 |     59.8 |  60.3 |            60.7 |

Balanced Neck. It is crucial to choose proper feature maps from backbone in tracking tasks. We compare our balanced neck module with FPN [Lin et al., 2017] on two similar backbones, ResNet-50 and DetNet-59 [Li et al., 2018b], used in image classification and object detection, to analyze the impacts of resolution and receptive fields. Backbone feature layers conv3, conv4, conv5 are involved. Using features of same spatial stride from neck modules or backbone as output feature maps, the performance of the Balanced Neck and FPN is shown in Table 3. The proposed balanced neck with ResNet-50 gains 1.3% AUC score improvement on the combined dataset and is 0.7% higher than FPN, which indicates features with balanced spatial details and sematic information are more suitable for object tracking. Note that DetNet-59 preforms better than ResNet-50 by 0.3% without a neck because it maintains high resolution in conv4 and conv5 while FPN aggregates the adjacent feature levels in a bottom-up manner. But they are less likely to provide balanced features.

## 4.2 State-of-the-art Comparison

The proposed TLPG-Tracker, which employs the backbone ResNet-50 and the post-process IoU-Net, is compared with the state-of-the-art methods including DiMP [Bhat et al., 2019], SiamRPN++ [Li et al., 2019], SiamMask [Wang et al., 2019b], ATOM [Danelljan et al., 2019], SPM [Wang et al., 2019a], LADCF [Xu et al., 2019], SiamGragh [Tu et al., 2019], SiamRPN R18 [Li et al., 2018a], MFT[Kristan et al., 2018], UPDT [Bhat et al., 2018], StructSiam [Zhang et al., 2018], MHIT [Bai et al., 2018], VITAL [Song et al., 2018], DRT [Sun et al., 2018], DaSiamRPN [Dai et al., 2017], BACF [Kiani Galoogahi et al., 2017], ECO [Danelljan et al., 2017], DSiam[Guo et al., 2017], SiamFCv2 [Valmadre et al., 2017], GOTURN [Held et al., 2016], SiamFC [Bertinetto et al., 2016], CCOT[Danelljan et al., 2016], MDNet [Nam and Han, 2016] on following five challenging tracking benchmarks.

Table 4: State-of-the-art comparison on the VOT2018 dataset in terms of expected average overlap (EAO), robustness and accuracy.

|                |   ECO |   DRT |   UPDT |   SiamMask |   DaSiamRPN |   MFT |   LADCF |   ATOM |   SiamRPN++ |   DiMP |   TLPG |
|----------------|-------|-------|--------|------------|-------------|-------|---------|--------|-------------|--------|--------|
| EAO (↑)        | 0.281 | 0.356 |  0.378 |      0.380 |       0.384 | 0.385 |   0.389 |  0.401 |       0.414 |  0.440 |  0.459 |
| Accuracy (↑)   | 0.483 | 0.519 |  0.536 |      0.609 |       0.586 | 0.505 |   0.503 |  0.590 |       0.600 |  0.597 |  0.606 |
| Robustness (↓) | 0.276 | 0.201 |  0.184 |      0.276 |       0.276 | 0.140 |   0.159 |  0.234 |       0.204 |  0.153 |  0.149 |

Table 5: State-of-the-art comparison on the GOT10k test set in terms of average overlap (AO) and success rate (SR) at overlap threshold 0.5.

|           |   MDNet |   ECO |   CCOT |   GOTURN |   SiamFCv2 |   SiamRPN R18 |   SPM |   ATOM |   DiMP |   TLPG |
|-----------|---------|-------|--------|----------|------------|---------------|-------|--------|--------|--------|
| AO(%)     |    29.9 |  31.6 |   32.5 |     34.7 |       37.4 |          48.3 |  51.3 |   55.6 |   61.1 |   62.9 |
| SR0.50(%) |    30.3 |  30.9 |   32.8 |     37.5 |       40.4 |          58.1 |  59.3 |   63.4 |   71.7 |   73.5 |

Table 6: State-of-the-art comparison on the TrackingNet test dataset in terms of precision and success.

|              |   UPDT |   SPM |   SiamGraph |   SiamRPN++ |   ATOM |   DiMP |   TLPG |
|--------------|--------|-------|-------------|-------------|--------|--------|--------|
| Precision(%) |   55.7 |  66.1 |        63.8 |        69.4 |   64.8 |   68.7 |   70.6 |
| Success(%)   |   61.1 |  71.2 |        70.9 |        73.3 |   70.3 |   74.0 |   75.8 |

Table 7: State-of-the-art comparison on the OTB2015 dataset in terms of area under the curve (AUC) score.

|        |   ATOM |   DiMP |   SPM |   ECO |   SiamRPN++ |   MHIT |   UPDT |   TLPG |
|--------|--------|--------|-------|-------|-------------|--------|--------|--------|
| AUC(%) |   66.9 |   68.4 |  68.7 |  69.1 |        69.6 |   69.8 |   70.2 |   69.8 |

ble 5. Our method outperforms the most progressive performance approaches DiMP with average overlap (AO) scores of 62.9% and success rates (SR) at thresholds 0.5 of 73.5%.

TrackingNet. TrackingNet is a huge dataset for target tracking, containing more than 30, 000 videos. As the results shown in Table 6, our TLPG-Tracker outperforms most advanced algorithms on TrackingNet test set, with the best precision rate of 70.6% and success score of 75.8%.

OTB2015. The standardizedOTB2015 benchmark containing 100 videos with various challenging factors provides a fair testbed on robustness. Among all compared methods in Table 7, our TLPG-Tracker achieves a competitive result with an AUC score of 69.9% while UPDT obtaining the best performance with an AUC score of 70.2%.

## 5 Conclusion

We propose a novel two-stage tracking architecture that learn both target localization and proposal generation simultaneously by deformable central correlation in online training manner. The proposals are generated on discriminative features near target centers in stage 1 and refined by centerbias refinement to provide more accurate results. The model benefits from multi-level aggregation of fine-grained and sematic features in a neck and a feature enhancement module. With robust localization and accurate estimation, our TLPGTracker sets a new state-of-the-art on 5 datasets while operating at over 30 FPS.

## References

1. [Bai et al., 2018] Shuai Bai, Zhiqun He, Ting-Bing Xu, Zheng Zhu, Yuan Dong, and Hongliang Bai. Multi- hierarchical independent correlation filters for visual tracking. arXiv preprint arXiv:1811.10302, 2018.
2. [Bertinetto et al., 2016] Luca Bertinetto, Jack Valmadre, João F. Henriques, Andrea Vedaldi, and Philip H.S. Torr. Fully-convolutional siamese networks for object tracking. In ECCV, pages 850-865, 2016.
3. [Bhat et al., 2018] Goutam Bhat, Joakim Johnander, Martin Danelljan, Fahad Shahbaz Khan, and Michael Felsberg. Unveiling the Power of Deep Tracking. In ECCV, pages 493-509, 2018.
4. [Bhat et al., 2019] Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte. Learning Discrimina- tive Model Prediction for Tracking. arXiv preprint arXiv:1904.07220, 2019.
5. [Bolme et al., 2010] David S Bolme, J Ross Beveridge, Bruce A Draper, and Yui Man Lui. Visual object tracking using adaptive correlation filters. In CVPR, pages 2544- 2550, 2010.
6. [Dai et al., 2017] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. De- formable convolutional networks. In ICCV, pages 764- 773, 2017.
7. [Danelljan et al., 2016] Martin Danelljan, Andreas Robin- son, Fahad Shahbaz Khan, and Michael Felsberg. Beyond correlation filters: Learning continuous convolution oper- ators for visual tracking. In ECCV, pages 472-488, 2016.
8. [Danelljan et al., 2017] Martin Danelljan, Goutam Bhat, Fa- had Shahbaz Khan, and Michael Felsberg. Eco: Efficient convolution operators for tracking. In CVPR, pages 6931- 6939, 2017.
9. [Danelljan et al., 2019] Martin Danelljan, Goutam Bhat, Fa- had Shahbaz Khan, and Michael Felsberg. ATOM: Accu- rate Tracking by Overlap Maximization. In CVPR, 2019.
10. [Fan et al., 2019] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. Lasot: A high-quality benchmark for large-scale single object tracking. In CVPR, pages 5374-5383, 2019.
11. [Guo et al., 2017] Qing Guo, Wei Feng, Ce Zhou, Rui Huang, Liang Wan, and Song Wang. Learning dynamic siamese network for visual object tracking. In ICCV, pages 1763-1771, 2017.
12. [Held et al., 2016] David Held, Sebastian Thrun, and Silvio Savarese. Learning to track at 100 fps with deep regression networks. In ECCV, pages 749-765, 2016.
13. [Huang et al., 2018] Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-10k: A large high-diversity benchmark for generic object tracking in the wild. arXiv preprint arXiv:1810.11981, 2018.
14. [Jiang et al., 2018] Borui Jiang, Ruixuan Luo, Jiayuan Mao, Tete Xiao, and Yuning Jiang. Acquisition of localization confidence for accurate object detection. In ECCV, pages 784-799, 2018.
15. [Kiani Galoogahi et al., 2017] Hamed Kiani Galoogahi, Ashton Fagg, and Simon Lucey. Learning background- aware correlation filters for visual tracking. In ICCV, pages 1135-1143, 2017.
16. [Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
17. [Kristan et al., 2018] Matej Kristan, Ales Leonardis, Jiri Matas, Michael Felsberg, Roman Pflugfelder, Luka Čehovin, Tomas Vojir, Goutam Bhat, Alan Lukežič, Ab- delrahman Eldesokey, Gustavo Fernandez Dominguez, Alvaro Garcia-Martin, Álvaro Iglesias-Arias, A. Alatan, Abel Gonzalez-Garcia, Alfredo Petrosino, Alireza Memar- moghadam, Andrea Vedaldi, and Andrej Muhič. The sixth visual object tracking vot2018 challenge results. In ECCV workshop, pages 3-53, 2018.
18. [Li et al., 2018a] Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu. High performance visual tracking with siamese region proposal network. In CVPR, pages 8971- 8980, 2018.
19. [Li et al., 2018b] Zeming Li, Chao Peng, Gang Yu, Xiangyu Zhang, Yangdong Deng, and Jian Sun. DetNet: Design backbone for object detection. In ECCV, pages 334-350, 2018.
20. [Li et al., 2019] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan. Siamrpn++: Evolution of siamese visual tracking with very deep networks. In CVPR, pages 4282-4291, 2019.
21. [Lin et al., 2017] Tsung Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Fea- ture pyramid networks for object detection. In CVPR, pages 936-944, 2017.
22. [Muller et al., 2018] Matthias Muller, Adel Bibi, Silvio Gi- ancola, Salman Alsubaihi, and Bernard Ghanem. Track- ingnet: A large-scale dataset and benchmark for object tracking in the wild. In ECCV, pages 300-317, 2018.
23. [Nam and Han, 2016] Hyeonseob Nam and Bohyung Han. Learning multi-domain convolutional neural networks for visual tracking. In CVPR, pages 4293-4302, 2016.
24. [Song et al., 2018] Yibing Song, Chao Ma, Xiaohe Wu, Li- jun Gong, Linchao Bao, Wangmeng Zuo, Chunhua Shen, Rynson WH Lau, and Ming-Hsuan Yang. Vital: Visual tracking via adversarial learning. In CVPR, pages 8990- 8999, 2018.
25. [Sun et al., 2018] Chong Sun, Dong Wang, Huchuan Lu, and Ming-Hsuan Yang. Correlation tracking via joint discrim- ination and reliability learning. In CVPR, pages 489-497, 2018.
26. [Tu et al., 2019] Zhengzheng Tu, Ajian Zhou, Bo Jiang, and Bin Luo. Visual object tracking via graph convolutional representation. In ICMEW, pages 234-239, 2019.
27. [Valmadre et al., 2017] Jack Valmadre, Luca Bertinetto, Joao Henriques, Andrea Vedaldi, and Philip HS Torr. End- to-end representation learning for correlation filter based tracking. In CVPR, pages 2805-2813, 2017.
28. [Wang et al., 2019a] Guangting Wang, Chong Luo, Zhiwei Xiong, and Wenjun Zeng. SPM-Tracker: Series-Parallel Matching for Real-Time Visual Object Tracking. arXiv preprint arXiv:1904.04452, 2019.
29. [Wang et al., 2019b] Qiang Wang, Li Zhang, Luca Bertinetto, Weiming Hu, and Philip HS Torr. Fast online object tracking and segmentation: A unifying approach. In CVPR, pages 1328-1338, 2019.
30. [Wu et al., 2015] Yi Wu, Jongwoo Lim, and Ming-Hsuan Yang. Object tracking benchmark. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(9):1834- 1848, 2015.
31. [Xu et al., 2019] Tianyang Xu, Zhenhua Feng, Xiaojun Wu, and Josef Kittler. Learning adaptive discriminative cor- relation filters via temporal consistency preserving spa- tial feature selection for robust visual object tracking. IEEE Transactions on Image Processing, 28(11):5596- 5609, 2019.
32. [Yang et al., 2019] Ze Yang, Shaohui Liu, Han Hu, Li- wei Wang, and Stephen Lin. RepPoints: Point Set Representation for Object Detection. arXiv preprint arXiv:1904.11490, 2019.
33. [Zhang et al., 2018] Yunhua Zhang, Lijun Wang, Jinqing Qi, Dong Wang, Mengyang Feng, and Huchuan Lu. Struc- tured siamese network for real-time visual tracking. In ECCV, pages 351-366, 2018.
34. [Zhu et al., 2018] Zheng Zhu, Qiang Wang, Bo Li, Wei Wu, Junjie Yan, and Weiming Hu. Distractor-aware siamese networks for visual object tracking. In ECCV, pages 103- 119, 2018.
35. [Zhu et al., 2019] Xizhou Zhu, Dazhi Cheng, Zheng Zhang, Stephen Lin, and Jifeng Dai. An empirical study of spatial attention mechanisms in deep networks. arXiv preprint arXiv:1904.05873, 2019.