Feature Integration with Adaptive Importance Maps for Visual Tracking

1 Introduction

Visual tracking is one of the fundamental problems in computer vision and has numerous applications such as surveillance, robotic services and so on. Given the bounding box in the first frame, its task is to estimate the trajectory of a target in an image sequence. In the absence of prior information, it needs to handle various challenging problems, i.e., deformations, occlusions, while maintaining tracking speed. Therefore, although significant progress has been made in recent years, visual tracking remains a challenging problem.

Here, we focus on the most general scenario of shortterm, model-free tracking. Short-term means that tracker does not perform re-detection no matter the model drifts or not. Model-free is the main challenge where no prior information regarding the category and the appearance of a target but the bounding box in the first frame is available.

Most existing trackers focus on either the model for detection or feature representation. Discriminative correlation filters (DCF) for visual tracking [Bolme et al., 2010] have been very popular due to robustness to the change in appearance and the superior computation. Dense sampling is the key to success of DCF, which is performed by circular shifts. However, these circular shifts are not equal to the actual translations. This results in boundary effects, which severely degrade the DCF's performance. There are two main solutions of boundary effects at present. One of them is to introduce a spatial regularization component (SRDCF) [Danelljan et al., 2015b], which suffers from low frames per second. Another is to learn from the cropped patch [Kiani Galoogahi et al., 2015; 2017], which can make the filter discriminative to the background.

∗Corresponding author.

Feature representation is critical to the performance of visual tracking [Wang et al., 2015]. DCF uses the grayscale intensity at the early stage because it cannot be applied to multi-channel. To improve the performance, the DCF formulation has been extended to multi-channel [Kiani Galoogahi et al., 2013; Henriques et al., 2015], which makes it possible to use more effective representation such as HOG, color features and deep features. At present, most of handcrafted features based trackers use color names and HOG as feature representation. For the better accuracy, DCF begins to use hierarchical convolutional features, sacrificing the tracking speed.

However, most trackers don't take into consideration importance and complementary information of features. Specifically, several recent trackers [Wang et al., 2017; Lukezic et al., 2017] are based on HOG features and color features, and simply concatenate them. As observed, HOG features focus on gradients of the target, which can describe the target's shape and are robust to illumination, while color features can extract features from homogeneous regions and show robustness to deformation. Therefore, these trackers don't exploit the advantages of every feature, just viewing all features as a multichannel feature.

In order to address this limitation, we propose feature integration with adaptive importance maps for DCF (CFWFI). The method is based on BACF [Kiani Galoogahi et al., 2017] which learns from background to deal with boundary effects. The main contributions of our work are as follows:

• In every frame, we jointly learn adaptive filters and importance maps which are used to achieve complementary traits. All channels of the same feature share an importance map, instead of learning an importance map for every channel, which maybe leads to overfitting.
• The adaptive importances of features are controlled by the penalty factors of importance maps. Specifically, compared with the grayscale intensity, HOG has a lower penalty factor for the importance map.
• We evaluate our method on OTB13 and OTB15 [Wu et al., 2013; 2015] datasets. The results demonstrate that our method achieves competitive accuracy compared to the state-of-the-art DCF (Figure 1). Moreover, the handcrafted feature based tracker maintains a speed of 22 frames per second on an i5 CPU.

2 RelatedWork

2.1 The Recent Improvements of DCF

Due to the impressive speed of MOOSE [Bolme et al., 2010], DCF has attracted wide attention. To improve the DCF's performance, researchers take into consideration multichannel features [Kiani Galoogahi et al., 2013; Henriques et al., 2015], adaptive scale [Danelljan et al., 2014a; Li and Zhu, 2014], nonlinear kernel [Henriques et al., 2015; Tang and Feng, 2015], boundary effects [Danelljan et al., 2015b; Kiani Galoogahi et al., 2015; 2017], ensemble methods [Zhang et al., 2017a; Zhang and Suganthan, 2017] respectively.

Inherent boundary effects of DCF severely degrade the tracking performance. To address this problem, SRDCF [Danelljan et al., 2015b] introduces a spatial regularization depending on the spatial location to penalize the filter, which suffers from the slow speed. Context-aware CF [Mueller et al., 2017], in addition to using the predicted outcomes as the training samples, imposes explicitly that filter's responses to the surrounding background equal zero. CFLB [Kiani Galoogahi et al., 2013] crops the shifted patch to learn from the background. Afterwards, Hamed et al. [2017] extend it to the multi-channel, maintaining the real-time tracking and excellent accuracy.

2.2 Feature Representation of DCF

As is shown in [Wang et al., 2015], feature representation plays the most critical role in tracking. Since the DCF formulation is extended to the multichannel, researchers begin to adopt more powerful handcrafted features to improve the tracking performance. KCF [Henriques et al., 2015] works on HOG descriptors and achieves more superior accuracy compared with CSK [Henriques et al., 2012]. However, when extending the linear kernel to the Gaussian kernel, KCF solely achieves a slight improvement. And Multi-kernel CF [Tang and Feng, 2015] attempts to take advantage of the invariancediscriminative power spectrums of various features to further improve the performance.

Besides HOG, CNs [Danelljan et al., 2014b] and DAT [Possegger et al., 2015] make color features attract more attention due to the favorable robustness to appearance's variance. Since observing the phenomenon that models based on color features can cope well with variation in shape but suffer when illumination, Staple [Bertinetto et al., 2016a] combines HOG based DCF with color histogram based regression in order to make full use of complementary traits. However, to maintain real-time speed, Staple solves two independent ridge regression problems.

Several recent research [Lukezic et al., 2017; He et al., 2017] pays attention to the spatial and channel reliability. CSR-DCF [Lukezic et al., 2017] based on HOG and color names learns separately the spatial reliability map, filters and channel reliabilities. The spatial reliability map is estimated using the output of a graph labeling problem. Filters are computed under the assumption of independent feature channel. And channel reliabilities depend on the maximum responses of channels. CF with weighted convolution [He et al., 2017] extracts features from different layers of the CNN in order to learn filters. Weights in convolution responses represent the significance of layers. And then, the weighted convolution responses from all filters are summed to produce the final confidence. As other deep features based trackers, it also has a slow speed.

3 DCF tracking

We first review the conventional DCF. It extracts features x ∈ RDMN from the predicted location of the target in the current frame to learn a correlation filter h ∈ RDMN used to locate the target in the next frame, where MN represents the size of vectorized features, D denotes the number of channels or dimensions. x and h are represented by concatenating their channels xl ∈ RMN , where l is the lth channel, x = [xT1 , xT2 , ..., xTD ]T and h = [hT1 , hT2 , ..., hTD ]T . The key to DCF's high performance is dense sampling where circular shifts approximate the actual translations in an image. The objective function is to minimize the sum of squared error between the actual output and the desired output, which is ridge regression with a closed form solution.

where y ∈ RMN is the vectorized 2D Gaussian response centered on the target in training image. The symbol ⋆ denotes the circular correlation. Due to Parseval's theorem and convolution theorem, ridge regression can be optimized quickly in the frequency domain.

where ˆhl, ˆxl and ˆ y represents discrete Fourier transforms (DFT), i.e., ˆhl = Fhl, ˆxl = Fxl and ˆ y = Fy, F denotes a DFT matrix. ˆh∗l indicates complex conjugate. The symbol ⊙ denotes element-wise multiplication. Equation (2) can be solved efficiently in the Fourier domain.

And detection is also computed in the frequency domain.

where r denotes the response, F -1 represents the inverse of DFT.

4 Feature Integration with Adaptive Importance Maps

In this section, we first present the problem formulation of feature integration with adaptive importance maps. Then, we deduce an optimization algorithm with ADMM.

4.1 ProblemFormulation

Feature representation is crucial to a tracker. Proper features can dramatically improve the tracking performance. Therefore, we use adaptive importance maps to improve the tracking performance. They take into consideration both the complementary traits and the importance of features. Specifically, m ∈ RMN×K denotes the importance maps and K indicates the number of features.

As observed, HOG features focus on gradients of the target, which can describe the target's shape and are robust to illumination, while color features can extract features from homogeneous regions and are robust to deformation. Also, the observation can be found in Figure 2(a) (the cyan map seems sparse and the magenta map seems smooth). In addition, in order to avoid overfitting, an importance map is shared by all channels of the same feature, instead every channel has its own importance map. Therefore, importance maps can achieve the complementary knowledge by laying stress on the focuses of every feature (see Figure 2(b)). However, for the sake of simplicity, importance maps are used for filters in our formulation.

Specifically, given a set of features x ∈ RDMN extracted from an image, x euqals [x1 ; x2 ; . . . ; xK ], xk ∈ RDkMN represents the kth features. Dk denotes the dimension of the kth feature and D equals the sum of Dk. xk,l ∈ RMN denotes the lth channel of the kth feature and hk,l ∈ RMN represents the filter of xk,l. To be simple, we convert the vector mk to a diagonal matrix diag(mk) ∈ RMN×MN , where mk is the importance map of the kth feature. We learn a filter h ∈ RDMN used to locate the target in next frame by the following objective function:

As we all know, since having the different discriminative ability, features are not equally important. Instead of weighted response sum [He et al., 2017], we use the importance map to indicate the significance of a feature. The greater an importance map's magnitude is, the more important corresponding feature is. Hence, we introduce the L2 regularization for importance maps, where α ∈ RK is the penalty coefficient of regularization. α controls the significance of every feature. Specifically speaking, when the value of αk is small, the magnitude of mk will be large which means the corresponding feature is more important. The complete formulation is as follows:

where m ⪰ 0 represents that every element in m is greater than or equal to zero.

4.2 Optimization Algorithm

To improve the computation efficiency, Equation (6) is transformed to the frequency domain according to convolution theorem and Parseval's theorem. Moreover, for the sake of simplicity, we introduce a variable g ∈ RDMN to remove filters with important maps, where gk,l ∈ RMN represents the lth filter with the importance map in the kth feature. x ∈ RDMN is transformed to Z ∈ RMN×DMN by diagonalization. And the primal objective function Equation (6) can be rewritten by vectorization as follow:

Employing augmented Lagrangian, we get Equation (8) where ξk,l ∈ RMN is the multiplier of gk,l, ξ ∈ RDMN denotes the Lagrange multiplier and µ denotes the penalty factor.

Equation (8) can be solved with alternating direction method of multipliers (ADMM). Therefore, it can be divided into three subproblems. We first solve the subproblem ˆ g, which is applied to detect the target in next frame.

Subproblem ˆ g∗ = argminˆ g L(ˆ g, h,m): Similar to ridge regression, this problem has a closed solution ˆ g∗ = ( ˆ Z T ˆ Z + µI)-1 ( ˆ Z T ˆ y - ˆ ξ + µˆh′ ), where ˆh′ ∈ RDMN , ˆh′k,l = Fdiag(mk)hk,l and I is the identity matrix. Observing that ˆx is sparse banded [Kiani Galoogahi et al., 2013], we can use Sherman-Morrison formula to speed up avoiding the inverse operation. Just as BACF's optimization [Kiani Galoogahi et al., 2017],

where t = [1, 2, ...,MN] and t represents a position in the filters or features. And ˆ g(t) = conj([ˆ g1(t), ..., ˆ gD(t)]T ), ˆ Z(t) = [ˆx1(t), ..., ˆxD(t)]T , ˆ ξ(t) = [ ˆ ξ1(t), ..., ˆ ξD(t)]T , ˆh′ (t) = [ˆh′1 (t), ..., ˆh′D (t)]T . conj represents conjuate.

Subproblem m: Observing that the objective function w.r.t. m is composed of K channels, mk ∈ RMN is independent to each other, which can be solved separately.

If an element of mk is less than zero, it will be set to zero. Subproblem h: The objective function of h is a quadratic programming problem. And every element hk,l in h is independent, which can be solved easily.

After estimating ˆh′k,l = Fdiag(mk)hk,l, the Lagrange multiplier is updated as

For updating the penalty factor µ, the standard scheme is applied as following:

Algorithm 1 CFWFI tracking algorithm

Require:

Image It, location pt-1, filter gt-1, model Z t-1 model Ensure:

• 1: Extract features Z t centered on pt-1;

Position pt, updated model Z tmodel ;

• 2: Prediction: pt = argmaxp F -1 ( ˆ Z t ˆ gt-1 );
• 3: Extract features Z t centered on pt;
• 4: Online update: ˆ Z tmodel = (1 - η) ˆ Z t-1 model + η ˆ Z t ;
• 5: Initial: h ⇐the cropped filter based on KCF, m⇐ 0;
• 6: repeat
• 7: Sequentially estimate ˆ gt,m, h with Equations (9), (11), (13);
• 8: Sequentially estimate ˆ ξ, µ with Equations (14), (15);
• 9: until Maximum iteration;
• 10: return ˆ gt, Z tmodel .

It is noteworthy that initial values of h come from KCF's solutions. And Similar to BACF [Kiani Galoogahi et al., 2017], we crop the center patch of h, other's values are assigned to zero. What's more, compared with the unitary DFT transform, the results of fft2 implemented by Matlab are √MN times.

Instead of updating filters, we use linear interpolation ˆ Z tmodel = (1 - η) ˆ Z t-1 model + η ˆ Z t to update target's model, where η is the online adaptation rate. A predicted location is the position with maximum value, pt = arg maxp F -1 ( ˆ Z t ˆ gt-1 ). The whole algorithm is illustrated in Algorithm 1.

5 Experiments

To validate the effectiveness of proposed approach, we implement CFWFI based on handcrafted features and DeepCFWFI which is achieved by adding CNN features to CFWFI. Then we evaluate our trackers on the OTB13 and OTB15 [Wu et al., 2013; 2015] benchmark datasets and compare them with some state-of-the-art methods.

5.1 Implementation Details

For real-time tracking, CFWFI's features are the grayscale intensity and HOG features with 31 channels. HOG features use 4*4 cell size to extract from an image patch. The area of image patch used to extract features is proportional to the area of the target bounding box. We set the region area to 5*5 times the target area and make it square. Samples extracted from the region are multiplied by a Hann window due to the DFT of samples. And this action ensures that tracker puts more emphasis near the center of the target. To learn and detect quickly, the maximum sample size is set to 50*50. Similar to SAMF's way [Li and Zhu, 2014] to deal with the scale variations, the number of scales is set to 5 with a scale-step of 1.01. The target output of correlation filter is a 2D Gaussian shaped response with the standard deviation of √wh/16, where w and h are the size of the target. The regularization factor λ of the filter is set to 0.01. In addition, the regularization factor α for importance maps is [0.5, 0.01]. The former is used for the grayscale intensity and the latter is used for HOG features. The learning rate η of updating the model is 0.013. Parameters of the alternating direction multiplier method are set to µ = 1 and β = 10 where the former is the penalty factor and the latter is the update rate. We empirically find that ADMM with 2 iterations can achieve the solution close to the optimal, thus set the number of iteration to 2. Our Matlab R2014a implementation runs on an Intel i5 CPU PC with 4 GB memory.

Moreover, following DeepSRDCF [Danelljan et al., 2015a], we add 96 channel features from the initial convolutional layer to CFWFI to implement DeepCFWFI. Learning rate η, regularization factor λ and α for CNN features are set to 0.009, 0.0001 and 0.05, respectively. Our Matlab R2015b implementation runs on an Intel i7 CPU PC and TITAN X.

5.2 Analyses of CFWFI

Here, we compare CFWFI with its variants, in order to demonstrate the effect of the proposed feature integration with adaptive importance map for visual tracking. We first investigate the consequence of removing adaptive importance maps (CFWFInoims), which is equivalent to the condition that the importance map is a constant matrix composed of zero and one. In addition, we compare the proposed CFWFI with BACFgray based on HOG and the grayscale intensity. What's more, to validate the previous suppose that overfitting occurs when all channels have their own importance map, we change CFWFI to CFWFIims. At the same time, we investigate the effect of integration of different features. We compare CFWFI with CFWFIcn. The former's features are composed of HOG and the grayscale intensity, the latter's features consist of HOG and color names. Table ?? shows characteristics of compared trackers and tracking performance of on OTB13 and OTB15. Figure 3 shows the success plots on OTB13 and OTB15, where numbers in legend represent the AUC.

As shown in Table ??, frames per second of these handcrafted features based trackers are close to each other. Trackers with 20fps can be qualified for the real-time application. And, the success rate indicates that CFWFI achieves the best accuracy and robustness on OTB13 and OTB15, which benefits from the proposed feature integration with adaptive importance maps. CFWFInoims achieves a poorer performance than CFWFI, without adaptive importance maps. It validates the suppose that adaptive importance maps are helpful for feature integration to get complementary information. CFWFIims is the poorest tracker, because it learns an importance map for every channel, which leads to overfitting. Compared with CFWFIcn, CFWFI has a superior performance, which shows that feature integration with HOG and the grayscale intensity is more effective. The reason may be that color names have more channels and show the variance of an instance. Thus the importance map can not find the complementary traits effectively.

Success plots of OPE

Success plots of OPE

5.3 Comparison with Handcrafted Feature based Trackers

Here, we compare our proposed CFWFI with handcrafted feature based trackers, including MCPF [Zhang et al., 2017b], SRDCFdecon [Danelljan et al., 2016a], ECO-HC [Danelljan et al., 2017], BACF [Kiani Galoogahi et al., 2017], SRDCF [Danelljan et al., 2015b], Staple [Bertinetto et al., 2016a], DSST [Danelljan et al., 2014a] on the OTB13 and OTB15.

Figure 4 illustrates the success plots on OTB13 and OTB15, where CFWFI achieves the best AUC 0.695 on OTB13 and AUC 0.641 on OTB15, while maintaining real-time speed. Our method outperforms BACF's success rate of 0.648 on OTB13 and 0.617 on OTB15. These results demonstrate the effectiveness of feature integration with adaptive importance maps. In addition, Staple has poorer AUC scores compared with CFWFI, which learns filters from two independent ridge regressions in order to get complementary traits from HOG and color histogram. This indicates that our method is more effective to jointly learn adaptive importance maps and filters. MCPF provides AUC scores of 0.677 and 0.628, which gets a great performance at the cost of speed. ECO-HC obtains competitive AUC scores of 0.652 and 0.643. Different from SRDFC, our method learns from the background to address boundary effects, which has greater AUC scores and a faster speed than SRDCF and SRDCFdecon.

Attribute based Evaluation

We perform an attribute based evaluation of our approach on OTB13. Figure 5 shows success plots of six different attributes, including deformation, occlusion, illumination variation, background clutter, in-plane rotation, out-of-plane rotation. Our approach outperforms other trackers on these attributes. This indicates the effectiveness of feature integration with adaptive importance maps on different attributes and how adaptive importance maps improve the performance of a tracker.

Specifically, in the case of deformation, Staple achieves an AUC score of 0.607, which uses the color histogram to address deformation. Our approach achieves a best AUC score of 0.712, followed by MCPF with an AUC score of 0.651. In the case of illumination variant, CFWFI achieves the best AUC score of 0.646 followed by the AUC score of 0.624 over SRDCFdecon. These results demonstrate that importance maps can effectively address deformation and illumination (see Figure 6), because they can help tracker achieve complementary information to improve robustness. In addition, because HOG focuses on gradients, the setting that HOG is more important in CFWFI improves the robustness to illumination variant. MCPF and ECO-HC provide the great results in the case of occlusion. Our method achieves an improvement of 0.022 over MCPF. As for background clutter, CFWFI provides a gain of 0.018 compared with MPCF and 0.035 compared with BACF. This benefits from the cropping operator coming from BACF, which can learn from the background. Eventually, CFWFI also provides the greater AUC scores on in-plane rotation and out-of-plane rotation, which achieves a gain of 0.02 and 0.031 compared with the second tracker. In addition to the online update, the fact that CFWFI can learn from the background contributes to the results.

5.4 Comparison with Deep Feature based Trackers

To validate the effectiveness of the proposed method, we follow DeepSRDCF [Danelljan et al., 2015a] and implement DeepCFWFI by adding CNN based features to CFWFI. As shown in Figure 7, compared with C-COT [Danelljan et al., 2016b], CFNet [Valmadre et al., 2017], SiamFC [Bertinetto et al., 2016b] and DeepSRDCF [Danelljan et al., 2015a], DeepCFWFI achieves a competitive performance at a speed of 9 fps on an i7 CPU and TITAN X.

5.5 Analyses of Regularization Parameter α

We select sequentially α for HOG, gray and CN and analyze the effect of regularization parameter based on handcrafted features. Combining HOG with color features, the proposed tracker is more robust when HOG's α is less than CN's or gray's. It validates our assumption that HOG is more important than CN or gray. Furthermore, compared with using HOG and gray as features, adding CN to features sacrifices the speed of tracker and cannot significantly improve performance. Because with the help of the importance maps, both gray and CN extract features from homogeneous regions to cooperate with high-frequency features extracted by HOG.

6 Conclusion

In this paper, we propose a novel feature integration with adaptive importance maps for visual tracking. To avoid overfitting, every feature has its own importance map which is shared by all channels of the same feature. Since importance maps can put emphasis on the focuses of every feature, tracker achieves the complementary traits. In addition, avoiding weighted response sum, our approach controls the significance of feature by the regularization factor of its importance map. We jointly learn importance maps and filters using ADMM and perform experiments on OTB13 and OTB15. Comparison with state-of-the-art trackers demonstrates the effectiveness of importance maps. Especially when deformation or illumination variant occurs, our method can use complementary information and address them effectively. And our approach achieves the competitive performance while maintaining the tracking speed.

Acknowledgments

This work is supported from the National Natural Science Foundation of China (Nos. 61603193, 61432008, 61272222), the Natural Science Foundation of Jiangsu Province (Nos. BK20171479, BK20161020, BK20161560) and Jiangsu Postdoctoral Science Foundation (No. 1701157B).