Do not Lose the Details: Reinforced Representation Learning for High Performance Visual Tracking

1 Introduction

Visual tracking aims to estimate the trajectory of a target in a video sequence. It is widely applied, ranging from human motion analysis, human computer interaction, to autonomous driving. Although much progress [Ross et al., 2008; Kalal et al., 2010; Henriques et al., 2015] has been made in the past decade, it remains very challenging for a tracker to work at a high speed and to be adaptive and robust to complex tracking scenarios including significant object appearance changes, pose variations, severe occlusions, and background clutters.

Recent CNN based trackers [Tao et al., 2016; Held et al., 2016; Bertinetto et al., 2016; Wang et al., 2018] have shown great potential for fast and robust visual tracking. In the off-line network pre-training stage, they learn a semantic embedding space for classification [Bertinetto et al., 2016; Valmadre et al., 2017] or regression [Held et al., 2016] on the external massive video dataset ILSVRC2015 [Russakovsky et al., 2015] using a backbone CNN architecture such as AlexNet [Krizhevsky et al., 2012] and VGGNet [Simonyan and Zisserman, 2015]. Different from hand-crafted features, the representations projected in the learned semantic embedding space contain rich high-level semantic information and are effective for distinguishing objects of different categories. They also have certain generalization capabilities across datasets, which ensure robust tracking. In the online tracking stage, these trackers estimate the target position at a high speed just through a single feed forward network pass without any network fine tuning.

∗Equal contribution.

†Contact author.

Despite the convincing design of the above CNN based trackers, they still have some limitations. First, the representations in the semantic embedding space usually have low resolution and lose some instance specific details and fine-grained localization information. These representations usually serve the discriminative learning of the categories in training data. Thus, on the one hand, they may be less sensitive to the details and be confused when comparing two objects with the same attributes or semantics as shown in Fig. 1; on the other hand, the domain shift problem [Nam and Han, 2016] may occur especially when trackers encounter targets of unseen categories or undergoing abrupt deformations. Second, these models usually do not perform online network updating to improve tracking speed, which inevitably affects the model adaptability, and thus hurts the tracking accuracy.

To tackle the above limitations, we develop a novel encoder-decoder paradigm for fast, robust, and adaptive visual tracking. Specifically, the encoder carries out correlation analysis on multi-resolution representations to benefit from both the fine-grained details and high-level semantics. On one hand, we show that the correlation filter (CF) based on the high-level representations from the semantic embedding space has good generalization capabilities for robust tracking, because the semantic embedding is additionally regularized by the reconstruction constraint from the decoder. The decoder imposes a constraint that the representations in the semantic space must be sufficient for the reconstruction of the original image. This domain-independent reconstruction constraint relieves the domain shift problem and ensures that the learned semantic embedding preserves all the geometric and structural information contained in the original fine-grained visual features. This yields a more accurate and robust correlation evaluation. On the other hand, another CF working on the low-level high-resolution representations contributes to fine-grained localization. A global context constraint is incorporated into the appearance modeling process of this filter to further boost its discrimination power. This filter serves as a differentiable CF layer and is updated efficiently on-line for adaptive tracking without network fine-tuning. The main contributions of this work are three-fold:

• A novel convolutional encoder-decoder network is developed for visual tracking. The decoder incorporates a reconstruction constraint to enhance the generalization capability and discriminative power of the tracker.
• A differentiable correlation filter layer regularized by the global context constraint is designed to allow efficient on-line updates for continuous fine grained localization.
• A multi-task learning strategy is proposed to optimize the correlation analysis and the image reconstruction in a mutual reinforced way. This guarantees tracking robustness and the model adaptability.

Based on the above contributions, an end-to-end deep encoder-decoder network for high performance visual tracking is presented. Extensive experimental evaluations on four benchmarks, OTB2013 [Wu et al., 2013], OTB2015 [Wu et al., 2015], VOT2015 [Kristan et al., 2015], and VOT2017 [Kristan et al., 2017], demonstrate its state-ofthe-art tracking accuracy and real-time tracking speed.

2 RelatedWork

Correlation filter based tracking. Recent advances of CF have achieved great success by using multi-feature channels [Danelljan et al., 2014b; Ma et al., 2015a], scale estimation [Li and Zhu, 2014; Zhang et al., 2015; Danelljan et al., 2014a], and boundary effect alleviation [Danelljan et al.,

2015b; Kiani Galoogahi et al., 2017; Lukezic et al., 2017; Mueller et al., 2017]. However, with increasing accuracy comes a dramatic decrease in speed. Thus, CFNet [Valmadre et al., 2017] and DCFNet [Wang et al., 2017] propose to learn tracking specific deep features from end to end, which improve the tracking accuracy without losing the high speed. Inspired by the above two trackers, we incorporate a global context constraint into the correlation filter learning process while still obtaining a closed-form solution, which ensures a more reliable end-to-end network training process. Instead of using deep features with wide feature channels and low resolution as in [Valmadre et al., 2017], we focus on learning fine-grained features with fewer channels. This approach is more suitable for efficient tracking and accurate localization.

Deep learning based tracking. The excellent performance of deep convolutional networks on several challenging vision tasks [Girshick, 2015; Long et al., 2015] encourages recent works to either exploit existing deep CNN features within CFs [Ma et al., 2015a; Danelljan et al., 2015a] and SVMs [Hong et al., 2015a], or design deep architectures [Wang and Yeung, 2013; Wang et al., 2015; Nam and Han, 2016; Tao et al., 2016] for discriminative visual tracking. Although CNN features have shown high discrimination, extracting CNN features from each frame and training or updating trackers over high dimensional CNN features are computationally expensive. Online fine-tuning a CNN to account for the target-specific appearance also severely hampers a tracker's speed as discussed in [Wang et al., 2015; Nam and Han, 2016]. Siamese networks are exploited in [Tao et al., 2016; Held et al., 2016; Bertinetto et al., 2016] to formulate visual tracking as a verification problem without on-line updates. We enhance a Siamese network based tracker by exploiting an encoder-decoder architecture for multi-task learning. The domain independent reconstruction constraint imposed by the decoder makes the semantic embedding learned in the encoder more robust to avoid domain shifts.

Hybrid multi-tracker methods. Some tracking methods maintain a tracker ensemble [Zhang et al., 2014; Wang et al., 2015], so the failure of a single tracker can be compensated by other trackers. TLD [Kalal et al., 2010] decomposes the tracking task into tracking, learning and detection where tracking and detection facilitates each other. MUSTer [Hong et al., 2015b], LCT [Ma et al., 2015b] and PTAV [Fan and Ling, 2017] equip short-term correlation filter based tracking with long-term conservative re-detections or verifications. Our online adaptive correlation filter, working on the finegrained representations, complements with the lone-term correlation filter based on the high-level generic semantic embedding. They share network architectures and are learned simultaneously in an end-to-end manner.

3 Encoder-Decoder Correlation Filter based Tracking

The proposed framework named EDCF is illustrated in Fig. 2. It is an encoder-decoder architecture to fully exploit multiresolution representations for adaptive and robust tracking. In particular, a generic semantic embedding is learnt for robust spatial correlation analysis. The embedding benefits from the domain independent reconstruction constraint imposed by the decoder. Fine-grained target localization is achieved using the correlation filter working on the low-level fine-grained representations. This correlation filter is regularized by a global context constraint and implemented as a differentiable layer. Finally, the whole network is trained from end to end based on a multi-task learning strategy to reinforce both the discriminative and generative parts.

3.1 Generic Semantic Embedding Learning for Robust Tracking

Different from recent deep trackers [Tao et al., 2016; Bertinetto et al., 2016], whose semantic embedding spaces only serve discriminative learning, we propose to learn a more generic semantic embedding space by equipping traditional discriminative learning with an extra image reconstruction constraint. Since the image reconstruction is an unsupervised task and is less sensitive to the characteristics of a training dataset, our learned semantic embedding space has a larger generalization capability, leading to more robust visual tracking. Moreover, the reconstruction constraint ensures that the semantic embedding space preserves all the geometric or structural information contained in the original fine-grained visual features. This increases the accuracy of the tracking.

The generic semantic embedding learning is based on an encoder-decoder architecture. The encoder φ : RM×N×3 → RP×Q×D consists of 5 convolution layers with two max pooling layers and outputs a latent representation projected from the semantic embedding space. The decoder ψ : RP×Q×D → RM×N×3 maps this high-level low-resolution representation back to the image space with the input resolution, achieved by stacking 7 deconvolutional layers. Then, the semantic embedding learning is optimized by minimizing the combination loss of the reconstruction loss Lrecon and the tracking loss Lhigh:Lsel = Lrecon + Lhigh, (1)

where parameters of the encoder and the decoder are denoted as θe and θd, z′ is the target image, and x′ is the search image. The tracking loss is discussed as follows.

The spatial correlation operation in the semantic embedding space is used to measure the similarities between the target image and the search image:

where φi,j(z′ ; θe) is a multi-channel entry for position (i, j) ⊂ Z 2 in the latent representation of the target image z′ , m×n corresponds to the spatial size for correlation analysis, and fu,v denotes the similarity between the target image and the search image whose center is of (u, v) ⊂ Z 2 pixels in height and width away from the target center. Each search image has a label y(u, v) ∈ {+1,-1} indicating whether it is a positive sample or a negative sample. Thus, the tracking problem can be formulated as the minimization of the following logistic loss:

where D ⊂ Z 2 is a finite grid corresponding to the search space and |D| denotes the number of search patches.

3.2 Context-Aware Correlation Filter based Adaptive Tracking

Although the reconstruction constraint reinforces the semantic embedding learning to preserve some useful structural details, it is still necessary to carry out correlation analysis on the low-level fine-grained representations for accurate localization. A global context constraint is incorporated into the correlation analysis to suppress the negative effects of distractors. This is achieved by a differentiable correlation filter layer. This layer permits end-to-end training and online updates for adaptive tracking. Note that this correlation analysis is implemented in the frequency domain for efficiency.

We begin with an overview of the general correlation filter (CF). A CF is learned efficiently using samples densely extracted around the target. This is achieved by modeling all possible translations of the target within a search window as circulant shifts and concatenating their features to form the feature matrix Z0. Note that both the hand-crafted features and the CNN features can be exploited as long as they preserve the structural or localization information of the image. The circulant structure of this matrix facilitates a very efficient solution to the following ridge regression problem in the Fourier domain:

where the learned correlation filter is denoted by the vector w, each row of the square matrix Z0 contains the features extracted from a certain circulant shift of the vectorized image patch z′0 and the regression objective y is a vectorized image of a 2D Gaussian.

Inspired by the CACF method [Mueller et al., 2017], our correlation filter is regularized by the global context for larger discrimination power. In each frame, we sample k context image patches z′i around the target image patch z′0 . Their corresponding circulant feature matrices are Zi and Z0 based on the low-level fine-grained CNN features. The context patches can be viewed as hard negative samples which contain various distractors and diverse background. Then, a CF is learned that has a high response for the target patch and close to zero response for context patches:

The closed-form solution in the Fourier domain for our CF is:

where z0 denotes the feature patch of the image patch z′0 , i.e., z0 = ϕ(z′0 ), ϕ(·) is a feature mapping based on the low-level convolutional layers in our decoder, ˆz0 denotes the discrete Fourier transform of z0, z∗0 represents the complex conjugate of z0, and ⊙ denotes the Hadamard product.

Different from CACF that directly adopts hand-crafted features for correlation analysis, we propose to actively learn low-level fine-grained representations fitting to a CF by transforming the above correlation filter into a differentiable CF layer which is cascaded behind a low-level convolutional layer of the encoder. This design permits end-to-end training of the whole encoder-decoder based network. In particular, the representations provided by a low-level convolutional layer of the encoder are designed to be fine-grained (without maxpooling) and with thin feature maps which are quite sufficient for accurate localization. The representations are denoted as x = ϕ(x′ ; θel), where x′ is a search image and θel denotes the parameters of these low-level convolutional layers. Then, representations are learned via the following tracking loss:

where X is the circulant matrix of the representations x for the search image patch, and w is the learned CF based on the representations z0 = ϕ(z′0 ; θel) for the target image patch and the representations zi = ϕ(z′i ; θel) for the global context as in Eqn. (7). The derivatives of Llow in Eqn. (8) are then obtained:

where ˆD := ˆz∗0⊙ˆz0+λ1+λ2 ∑ki=1 ˆz∗i ⊙ˆzi is the denominator of ˆw and Re(·) is the real part of a complex-valued matrix.

3.3 Multi-task Learning and Efficient Tracking

Considering above two differentiable functional components which complement with each other in fine-grained localization and discriminative tracking based on the multi-resolution representations, we propose to utilize the multi-task learning strategy to end-to-end train our network to simultaneously reinforce two components. Our multi-task loss function is:

where R(θ) is introduced as ℓ2-norm of the network weights in order to regularize the network for better generalization.

In the tracking stage, given an input video frame at time t, we crop some large search patches centered at the previous target position with multiple scales, denoted as x′s . These search patches are fed into the encoder to get two representations. The fine-grained representation is fed into the context aware correlation filter layer given in Eqn. (9). The semantic representation is evaluated based on the spatial correlation operation given in Eqn. (3). Then, the target state is estimated by finding the maximum of the fused correlation response:

Note that the high-level spatial correlation response map f(·) is up-sampled using the bilinear interpolation method to have a consistent resolution with the low-level response map g(·). Because the tracking process only involves a network feedforward pass and correlation analysis in the frequency domain, it is quite efficient and works in real-time.

We propose to use a fusion of long/short-term update strategy. The semantic representation of the target image φ(z′ ; θe) in Eqn. (3) is only calculated in the first frame for generic long-term tracking. The context aware correlation filter w in Eqn. (7) is updated online to adapt to target appearance changes via the linear interpolation:

where the dynamic learning rate αt is determined by the basic learning rate α, and the maxima of the spatial correlation response maps in the first and current frames, i.e. f ∗ (x′1 ), and f ∗ (x′t ). Since the semantic representation of the target template is fixed, variations of the correlation response indicate target appearance changes and background disturbances. Benefited from this update strategy, the two functional components complement with each other in the temporal domain.

4 Experiments

4.1 Implementation Details

Network architecture. Our tracker exploits an encoderdecoder architecture. The encoder has the same architecture as the baseline tracker SiamFC [Bertinetto et al., 2016], using an AlexNet by removing the fully connected layers. Its input has a size of 255×255×3. Its output is of size 22×22×256 provided by the Conv5 layer, which is then fed to the spatial cross correlation layer for correlation analysis and also fed to the decoder for image reconstruction. The decoder contains 7 deconvolutional layers and is removed in the tracking process. The fine-grained representations from the Conv2 layer of size 125×125×8 are fed into a context-aware CF layer for accurate localization.

Training Data and method. Our network is end-to-end trained on the video dataset from the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) [Russakovsky et al., 2015]. The data set contains more than 4000 sequences and nearly two million annotated object image patches. It can safely be used to train a deep model for tracking without overfitting to the domain of videos used in the tracking benchmarks. Two frames containing the same object are randomly picked. The target patch and the search patch are cropped with a padding size of 2, and then resized to the input size of 255 × 255 × 3. We use the SGD solver with a learning rate exponentially decaying from 1e - 2 to 1e - 5.

Online tracking parameters. Given the pre-trained models, the online tracking is only affected by the dynamic learning rate in Eqn. (18). The basic learning rate is set as α = 0.017. The scale interval is set as S = 1.02 and 3 scale layers are exploited. The regularization parameters in Eqn. (6) are set as λ1 = 1e - 4 and λ2 = 0.1.

Tracking benchmarks. We provide ablation studies and the overall evaluations of our tracker on the OTB2013 [Wu et al., 2013], OTB2015 [Wu et al., 2015], VOT2015 [Kristan et al., 2015], and VOT2017 [Kristan et al., 2017] datasets. Two evaluation metrics are exploited on OTB datasets including distance precision (DP) and overlap precision (OP). On VOT datasets, the expected average overlap (EAO) is exploited to quantitatively analyze the tracking performance.

4.2 Ablation Study

This section shows the effectiveness of the generic semantic embedding learning and the fine-grained localization achieved by the context-aware correlation filter. Table 1 gives comparisons between the baseline trackers and the variations of our tracker.

Generic Semantic Embedding. By introducing an image reconstruction constraint into a SiamFC [Bertinetto et al., 2016] tracker based on the encoder-decoder network architecture, denoted EDSiam, large DP gains of 3.7% are obtained on OTB2015 datasets. This domain independent constraint improves the generalization capability of the learned semantic embedding and ensures robust tracking.

Context Aware Correlation Filter. Our second functional part discussed in Section 3.2 is denoted as CACFNet, which learns fine-grained Conv2 representations fitted to a context aware CF for accurate tracking. Compared to the CACF [Mueller et al., 2017] tracker, large OP gains of more than 8% are obtained on OTB datasets, which proves that our learned representations are much more discriminative than traditional HOG features. Compared to the CFNet [Valmadre et al., 2017] tracker that learns Conv2 representations for a general CF, significant OP gains of more than 10% are obtained by our tracker. Our tracker exploits fine-grained Conv2 representations with thinner channels and incorporates a global context constraint into a general CF, leading to a stable appearance modeling.

Multi-task Learning. CACFNet+ enhances CACFNet by introducing the correlation analysis in the semantic embedding space and the reconstruction constraint exploited in EDSiam to the training process of CACFNet+. In the tracking stage, CACFNet+ estimates the target state only based on the fine-grained correlation response from the context aware

.

CF. The performance gains of CACFNet+ over CACFNet prove that the high-level constraint reinforces the learning process of the fine-grained correlation appearance modeling for discriminative tracking. Finally, EDCF outperforms CACFNet+, which shows the effectiveness of the fusion of the fine-grained and the semantic correlation responses as discussed in Eqn. (16). The tracking speed of EDCF is also given in Table 1. The tracker achieves real-time tracking with significant performances gains on both datasets.

4.3 Evaluations on OTB2013 and OTB2015

The EDCF tracker is compared with recent state-of-the-art trackers including BACF [Kiani Galoogahi et al., 2017], SAMF CA [Mueller et al., 2017], CFNet [Valmadre et al., 2017], SiamFC [Bertinetto et al., 2016], SINT [Tao et al., 2016], LCT [Ma et al., 2015b], MEEM [Zhang et al., 2014], CF2 [Ma et al., 2015a], SRDCF [Danelljan et al., 2015b], KCF [Henriques et al., 2015], and DSST [Danelljan et al., 2014a] on OTB2013 and OTB2015 datasets. Fig. 3 shows the success plots on the two datasets.

Among the compared trackers using deep features, EDCF provides the best results with AUC scores of 66.5% and 63.5%, respectively. Compared to the Siamese network based trackers [Valmadre et al., 2017; Bertinetto et al., 2016; Tao et al., 2016], our tracker obtains significant AUC gains especially on OTB2015 of more than 4.3%. Among the CF trackers using pre-trained features, CF2 achieves an AUC score of 57.7% at 15 FPS on OTB2015. Our tracker obtains a relative gain of 10.4% in AUC with more than 3 times faster tracking speed. Among the real-time trackers, BACF, LCT, MEEM, DSST and KCF, are more likely to track the target with lower accuracy and robustness or may lose the targets in case of background clutters. The results prove that robust and accurate target localization can be achieved by the cooperation of our two complementary parts, namely the fine-grained context-aware correlation filter based tracking and the generic semantic embedding learning based tracking.

4.4 Evaluations on VOT2015 and VOT2017

Our tracker is evaluated on VOT2015 [Kristan et al., 2015] and VOT2017 datasets [Kristan et al., 2017] as shown in Fig. 4. The horizontal grey lines are the state-of-the-art bounds. EDCF ranks 3rd and 8th respectively in the overall performance evaluations based on the EAO measure. In particular, EDCF ranks first in the VOT2017 real-time experiment as shown by the red polygonal line in Fig. 4b.

(b) EAO on VOT2017

Figure 4: Expected average overlap plot for VOT2015 (top) and VOT2017 (bottom) benchmarks with the proposed EDCF tracker. Legends are shown only for top performing trackers.

Among the top 10 compared trackers on VOT2015, only NSAMF runs at real-time with an EAO score of 0.254. EDCF also operates in real-time (65 FPS) with an EAO score of 0.315. EDCF achieves comparable accuracy scores to DeepSRDCF and MDNet, while running at orders of magnitude faster. Compared to the baseline tracker SiamFC [Bertinetto et al., 2016] which has an EAO score of 0.188 on VOT2017, EDCF substantially outperforms SiamFC by an absolute gain of 7.0% in EAO and demonstrates its superiority in tracking robustness and accuracy. To further evaluate the efficiency in EDCF, we conduct the real-time experiment on VOT2017. CSRDCF++ [Lukezic et al., 2017] achieves top performance on real-time performance with an optimized C++ implementation. Our EDCF achieves state-of-the-art performance with real-time EAO of 0.241, which obtains a 14% relative improvement over the VOT2017 winner.

5 Conclusions

We have proposed an end-to-end encoder-decoder network for the CF based tracking. A domain independent image reconstruction constraint is incorporated to the semantic embedding learning to generate high-level representations with strong generalization capabilities while maintaining the structural information. A fine-grained context aware CF is learned for accurate localization and online updated for adaptive tracking. Experiments show that the proposed tracker significantly boosts the accuracy of Siamese network based trackers while maintains high speed. In future work, we plan to incorporate middle level feature representation learning in the EDCF model to further improve its effectiveness.

Acknowledgements

This work is supported by the Natural Science Foundation of China (Grant No. 61672519, 61751212, 61472421, 61602478), the NSFC-general technology collaborative Fund for basic research (Grant No. U1636218), the Key Research Program of Frontier Sciences, CAS, Grant No. QYZDJSSW-JSC040, and the CAS External cooperation key project.