Robust Joint Discriminative Feature Learning for Visual Tracking

1 Introduction

As an important research topic in the field of computer vision, visual tracking has been extensively studied in last two decades with the development of numerous tracking algorithms. However, it still remains challenging due to large appearance variations of the tracked object caused by illumination, occlusion, etc. To account for these variations as well as their complicated interaction, different kinds of visual cues (features) that describe different characteristics of the object, e.g. color, texture are fused and jointly exploited for more robust appearance modeling [Grabner and Bischof, 2006; Hong et al., 2013; Lan et al., 2014]. Given multiple visual cues (features) of the tracked object, a key problem is how to fuse them properly to obtain both accurate target representation and effective target/background discrimination.

A variety of feature fusion-based tracking methods have been proposed, which can be roughly divided into two categories: discriminative methods and generative methods according to their feature fusion strategies. Discriminative models combine the discriminabilities of different features to facilitate the discrimination between the target and its background. Typical approaches belonging to this category such as [Grabner and Bischof, 2006; Grabner et al., 2008; Babenko et al., 2011] are grounded on online boosting. For example, [Grabner and Bischof, 2006] proposed an online boosting-based tracker in which a large weak classifier pool is learned and updated from various kinds of raw features, and an online boosting algorithm is employed to select and fuse weak classifiers for target/background discrimination. Along this line, more variants of boosting-based approaches such as [Grabner et al., 2008; Babenko et al., 2011] are developed to deal with the drifting problem. Discriminative models can alleviate the background distraction problem to some extent because different features are jointly exploited for foreground/background separation. However, the classifiers which discriminative methods exploit for feature fusion are directly learned and updated with the raw features extracted from the target samples. If the target samples are corrupted/contaminated by external variations such as occlusion, illumination, etc, the extracted features may not well reflect the intrinsic properties of the object appearance. Therefore, learning and updating classifiers using such corrupted features may degrade the fusion performance, which urges the need to remove the corrupted features or learn some uncontaminated features for robust appearance modeling.

Unlike discriminative methods, generative methods directly fuse multiple features to represent the object to increase the representation ability of a tracker. To enhance the tracking robustness to large appearance variations, some strategies are adopted, e.g. using trivial templates [Mei et al., 2015; Hu et al., 2015b] to model the outliers existing in the target's appearance, removing unreliable features [Lan et al., 2015] for robust feature-level fusion, or integrating responses from various Gabor kernels to capture the local appearance changes [Zhang et al., 2016]. Since generative methods directly fuse features for model learning without mapping them to classification scores, they preserve more information of multiple features, and are more capable of accounting for appearance changes than discriminative methods. However, generative methods do not take advantage of background information for appearance modeling with multiple features, which may make them easily to be distracted by cluttered background and lead to tracking failure.

Generally speaking, discriminative and generative methods have complementary advantages in feature fusion-based appearance modeling, and the success of a visual tracker depends on both its representation ability against appearance variations and its discriminability between the target and its background. As such, the advantages of these two approaches should be exploited jointly for more robust feature fusion, so that multiple features can be employed simultaneously to describe the object appearance accurately and separate the object from background discriminatively. In addition, different features extracted from the same object share some consistency while each feature should also have some specific knowledge in their discriminability. As is pointed out in [Liu et al., 2014], consistency is closely related to agreement while feature specific knowledge provides complementarity and is related to disagreement. While exploring featurespecific discriminative information for feature fusion has been shown to be effective in some methods such as online boosting [Grabner and Bischof, 2006; Grabner et al., 2008; Babenko et al., 2011], the benefits of exploiting shared information among multiple features/modalities/views have also been well demonstrated in some learning and classification tasks recently [Yang et al., 2012; Hu et al., 2015a; Wang et al., 2015]. This motivates us to explore an effective strategy to jointly consider the shared and feature-specific discriminative information for feature fusion.

Based on aforementioned motivations, we propose a novel robust joint discriminative feature learning framework for object tracking using multiple visual cues. Different from other feature fusion-based trackers which directly employ potentially contaminated raw features and utilize the representation ability or discriminability of different visual cues alone, the proposed method aims to learn uncontaminated and discriminative features to jointly exploit the representation and discriminative power of multiple visual cues for visual tracking. Within this unified framework, feature learning is performed by simultaneously and optimally removing corrupted features and learning reliable classifiers. As such, feature learning from multiple visual cues with corrupted feature removal offers uncontaminated features for reliable classifier learning while discriminative classifier learning with multiple visual cues imposes the discriminability to the learned features. Therefore, the limitations of the generative and discriminative approaches can be compensated and the benefits of these approaches can be combined. In addition, we incorporate a novel feature fusion scheme into the feature learning framework to further exploit the shared and feature-specific discriminative information for feature fusion, and the importance of different features in target/background discrimination is also dynamically weighted in this optimal learning framework. By jointly exploiting the learned features and classifiers from multiple visual cues for target representation and target/background classification, the learning framework enhances the tracking performance in term of representation accuracy and discrimination reliability.

It should be noted that some hybrid approaches which attempt to combine the benefits of both the generative and discriminative approaches have been developed, e.g. [Yu et al., 2008]. Their models are developed in the context of using a single feature, while the proposed model is developed for multi-feature appearance model and can be more effectively used for features learning and fusion with multiple visual cues. Although existing single feature-based hybrid approaches may be applied to multiple features by feature concatenation, such an approach ignores different statistical properties of different features and may result in a long feature vector that may degrade the learning efficiency. The proposed method is also different from the recent developed fusion-based tracker [Zhang et al., 2015b] since the proposed model focuses on feature fusion while [Zhang et al., 2015b] focuses on tracker fusion.

The contributions of this paper are listed as follows:

• ÷ We propose a novel feature learning model which is able to simultaneously and optimally learn discriminative features and reliable classifiers from potentially contaminated samples to exploit the representation and discriminative power of multiple visual cues for visual tracking.
• ÷ We propose a novel feature fusion scheme which simultaneously considers the shared and feature-specific discriminative information from multiple visual cues. Therefore, both the consistency and complementarity of the discriminability of multiple features are jointly exploited for more robust feature fusion.
• ÷ We derive a four-step iterative optimization algorithm to effectively solve the proposed robust joint discriminative feature learning model.

2 RelatedWork

2.1 Feature Learning in Visual Tracking

Recent works on feature learning-based tracking include dictionary learning-based approaches and deep learning-based approaches. Various dictionary learning based-trackers are proposed to update tracking model effectively [Zhang et al., 2015a], enhance the reconstructive and discriminative power [Fan et al., 2014] of the appearance model, etc. Most dictionary learning-based trackers use a single feature, i.e. intensity only, which may not be sufficient to account for large appearance variations. Deep learning-based trackers such as [Li et al., 2014] tune an off-line pre-trained deep neural network online to adapt the appearance variations, which may not be efficient. Different from the aforementioned approaches, this paper aims to exploit multiple visual cues for feature learning without off-line large-scale training samples.

2.2 Shared and Feature-Specific Information among Multiple Features/Modalities/Views for Pattern Classification

Exploiting the shared and feature-specific information among multiple features/modalities/views jointly has been shown to be beneficial for pattern classification. [Yang et al., 2012] proposed a multi-feature collaborative model which simultaneously models the similar and distinctive information among multiple features for image classification. [Hu et al., 2015a] proposed to learn the shared and feature-specific structure of heterogeneous channels for RGB-D activity recognition. A multi-modal sharable and specific feature learning approach is proposed for learning the shared and model-specific properties for RGB-D object recognition [Wang et al., 2015]. These research findings motivate us to exploit the shared and feature-specific discriminative information for feature fusionbased visual tracking.

3 ProposedModel

3.1 Robust Joint Discriminative Feature Learning

In the t-th frame, let Y k F = [yk 1 ↪ ▷ ▷ ▷ ↪ ykn1 ] be the recently obtained target samples of the k-th visual cues, and n1 denote the number of target samples in the sample set. Since large appearance variations, e.g. occlusion, illumination may occur during tracking, the captured samples may be contaminated/corrupted. To ensure the robustness of the learned features, explicitly seperating out the corrupted samples is essential. Therefore, one objective of the learning model is to learn the uncontaminated features while separating out the corrupted features as follows:

where XkF and EkF are the learned uncontaminated features and the separated corrupted features, respectively, and K is the number of visual cues. The background samples near the target position, also known as local context information in the current frame may share some similarity with the recently obtained target samples, such as illumination conditions. Exploiting such context information has been shown to be beneficial for tracking [Grabner et al., 2010]. Besides, the target samples in recent frames are temporally correlated, and mining the latent structure embedded in the target samples can facilitate revealing the intrinsic characteristic of the target's features [Ross et al., 2008]. To further exploit the spatial and temporal correlation of the background and the target samples for feature learning while separating out the corrupted features from multiple visual cues, we cast the objective discussed above into the following rank and sparsity minimization problem with the sample set of multiple visual cues:

where Y k B = [ykn1+1 ↪ ▷ ▷ ▷ ↪ ykN ] is the nearby background samples in current frame, N is the total number of samples. Y k = [Y k F ↪ Y k B ], Xk = [XkF ↪XkB ] and Ek = [EkF ↪EkB ] is the original feature set, the learned feature set and the separated feature set of the target samples and the background samples in the k-th visual cue, respectively. With the same merit of RPCA [Cand`es et al., 2011], the rank minimization term is able to uncover the shared latent space embedded in the samples of different visual cues which characterizes intrinsic properties of uncontaminated features of the target and the background, while the sparsity regularization is employed to model the outliers that exist in the corrupted features.

Although the feature learning scheme in (2) is able to learn uncontaminated informative features from multiple visual cues for target representation via joint low-rank and sparse matrix decomposition, it cannot guarantee that the target and background samples can be well discriminated with the learned features. As such, appearance modeling with the learned features may suffer the loss of discriminability, which may lead to the background distraction problem. To strengthen the discriminability of the learned features while modeling such different discriminabilities of different visual cues for robust feature fusion, we impose the discriminability regularization which measures the prediction loss using the learned classifiers to the feature learning process as follows:

where Lk = [Lk1 ↪ ▷▷▷↪ LkN ]T is the label vector, Lki = +1(-1) means that the i-th sample of the k-the visual cue xki belongs to the target (background) class, βk is the importance weight of the prediction loss corresponding to the k-th visual cue, wk 2 Rdk , 1 2 RN whose elements are all 1s, bk 2 R and dk is the dimension of the k-th visual cue. From (3), we can see that imposed discriminability regularization aims to minimize the weighted sum of the prediction loss of the learned features in different visual cues based on multiple linear classifiers ¶wk ↪ bk ♦Kk=1 which are learned jointly. Therefore, it ensures the learned features in multiple visual cues for the target and the background samples can be linearly separated as well as possible, which is able to enhance the discriminability of the tracking model and hence alleviates the background distraction problem. Moreover, dynamically learning and updating the importance weights during tracking allows the discriminative powers of different visual cues to be adaptively evaluated, which guarantees that more discriminative features play more important roles in target/background discrimination. Here we use (βk )2 instead of βk for feature fusion because we want to ensure all the weights are positive which avoids the trivial solution that the weight corresponding to the lowest prediction loss is 1, and 0 otherwise.

To further exploit the shared and feature-specific discriminative information of multiple visual cues, we introduce the following objective function into the proposed feature learning framework:

where the L⇤ = [L⇤1 ↪ ▷ ▷ ▷ ↪L⇤N ]T , and L⇤i is the learned consensus classification score of different visual cues in the ith training sample, which reflects the consistent discriminative information from different visual cues. Different from other discriminative feature fusion models which enforce different visual cues to share the same classification score [Yu et al., 2013] or to be with diverse discriminative information [Liu and Yao, 1999], the objective function softly regularizes the classification scores towards the consensus while enabling them to have some disagreement with the consensus. Therefore, both the consistent and feature-specific information among multiple visual cues are explicitly and jointly employed for learning informative features and reliable classifiers. Moreover, the disagreement with the consensus can be controlled by ✓k , and larger (less) ✓k will promote less (larger) disagreement.

Unifying them all together. Based on the aforementioned analysis, we formulate the objectives as mentioned above into an unified joint discriminative feature learning framework in which uncontaminated and corrupted features, classier parameters of multiple visual cues, denoted as ⌦ = ¶L⇤ ↪ Xk ↪ Ek ↪ wk ↪ bk ↪ βk ♣k = 1↪ ▷ ▷ ▷ ↪ K♦ are jointly estimated as follows:

where ↵1 and ↵2 are the nonnegative parameters associating with different objective functions, and the constant 12 is used for simplifying deductions. Since the rank minimization in (2) is a NP-hard problem, we employ the standard approach [Cand`es et al., 2011] and relax this problem by using nuclear norm k · k⇤ instead. The optimization procedure for (5) is derived in Section 3.2.

3.2 Optimization

The objective function in (5) is convex with respect to one of these four blocks ¶Xk ↪Ek ♦Kk=1 , ¶wk ↪ bk ♦Kk=1 , L⇤ and ¶βk ♦Kk=1 when the other three blocks are fixed, and it's difficult to derive the analytical solution to (5). Therefore, we derive an iterative optimization algorithm to solve the problem. To make the problem separable, we introduce the auxiliary variables ¶Z k ♦Kk=1 to replace ¶Xk ♦Kk=1 in the nuclear norm k · k⇤ of (5). Accordingly, ¶8k↪Xk = Z k ♦ act as additional constraints. Let C be the constraint set of (5) on ¶βk ♦Kk=1 , and ai = ↵i N for i = 1 or 2 . Then the augmented Lagrange function of (5) L⌦2C is

Algorithm 1: Optimization Algorithm for (5)

Input: Sample matrix ¶Y k ♦Kk=1, label vector ¶Lk ♦Kk=1, sample number N and feature number K

Output: ¶Xk↪i ↪ Ek↪i ↪ wk↪i ↪ bk↪i ↪ βk↪i ♦Kk=1↪ L⇤ Initialization: i 1↪Xk↪i Y k ↪Ek↪i 0↪ βk↪i 1K ↪wk↪i 0↪ bk↪i 0↪ k = 1↪ ▷▷▷K↪ at ↵t N ↪ t = 1 or 2

while stopping conditions are not satisfied do

end

with the definition Φ(A↪ B) =< A↪ B > + θ2 kBk2F , where θ is a positive penalty scalar, < A↪ B >= trace(AT B) and, ¶⇤k ↪ Γk ♦Kk=1 are the Lagrangian multipliers. Based on (6), the solutions to (5) can be obtained by iteratively solving the subproblems of (5) in which an inner loop procedure is employed to solve ¶Xk ↪ Ek ♦.

¶Xk ↪ Ek ♦-subproblem: Keeping other variables fixed, we obtain Xk ↪Ek ↪ k = 1↪ ▷ ▷ ▷ ↪K using Alternating Direction Method of Multipliers (ADMM) [Boyd et al., 2011]. In the (j + 1)-th step of ADMM, by some algebraic manipulations, Ek↪j+1 and Z k↪j+1 are obtained as

where Qk↪j = Xk↪j + Γk↪j θ , and P k↪j = Y k↪j -Xk↪j + ⇤k↪j θ . S(·)(·) is a soft-thresholding operator such that Sp(A)m↪n = sign(Am↪n) · max(♣Am↪n♣ - p↪ 0), and J(·)(·) is the singular value soft-thresholding operator such that Jp(A) = UASp(⌃A)V T A where UA⌃AV T A is the singular value decomposition of A. Then by taking partial derivatives of (6) with respect to Xk , we obtain

where Gk↪j 1 = (a1(βk )2 + a2)wk (wk )T + 2θI, and Gk↪j 2 = a1(βk )2wk ((Lk )T - bk 1T ) + a2✓kwk ((L⇤ )T - bk 1T ) + θ(Y k↪j+1 - Ek↪j+1 + Z k↪j+1 ) + ⇤k↪j - Γk↪j .

After ¶Xk↪j+1 ↪Z k↪j+1 ↪Ek↪j+1 ♦Kk=1 are obtained, the Lagrangian multipliers are updated as follows:

The ADMM algorithm iteratively updates the optimal variables and the Lagrangian multipliers in the inner loop of the whole optimization algorithm until kY k - Xk - Ek k2F < ✏kY k k2F or the maximum iteration number is reached. Then with ¶Xk↪i ↪Ek↪i ♦Kk=1 in the i-th step of the outer loop, we can solve the subproblem with respect to other variables.

Algorithm 2: Solver for ¶Xk ↪ Ek ♦-subproblem

Input: Sample matrix of k-th visual cue Y k , label vector Lk , sample number N, other optimal variable with fixed values wk ↪ bk ↪ βk ↪ L⇤ , initial values Xk↪i-1 ↪ Ek↪i-1 from (i - 1)-th iteration in Algorithm 1

Output: Xk↪i ↪ Ek↪i

while stopping conditions are not satisfied do

Check stopping conditions

end

¶wk ↪ bk ♦-subproblem: With other variables fixed, it is equivalent to solve the following problem:

which is an unconstrained quadratic programming problem and can be solved by some standard optimization techniques, e.g. conjugate gradient descent method.

L⇤ -subproblem: Similar to the ¶wk ↪ bk ♦-subproblem, some standard optimization techniques can be used to solve the following unconstrained quadratic programming problem:

βk -subproblem: Let Rk = ak 2 k(Xk )T wk +1bk -Lk k22 . Then this subproblem can be rewritten as

which is a quadratic programming problem with linear constraints. Based on its Lagrange function, the solution can be derived as βk0 = (Rk0 )-1 PKk=1 (Rk )-1 . We iteratively solve these four subproblems until the relative change of the valuables in the adjacent iterations are less than a predefined threshold or the maximum iteration number is reached. The optimization algorithm for (5) and the solver for ¶Xk ↪Ek ♦-subproblem are summarized in Algorithm 1 and 2, respectively.

4 Implementation Details

4.1 Target Representation

For the sake of robustness and effectiveness [Zhang et al., 2013a], we adopt the sparse representation scheme [Mei and Ling, 2011] for target representation. To further enhance the adaptivity of the proposed tracker, we augmented the learned features for each visual cue Ak with the recent obtained important target samples RkF to construct a template set Dk = [XkF ↪RkF ] where RkF is updated adaptively similar to [Mei and Ling, 2011]. Then we can obtain the sparse representations of the target candidates of each visual cue uki ↪ i = 1↪ ▷ ▷ ▷ ↪m↪ k = 1↪ ▷ ▷ ▷ ↪K as follows:

where λ is the tradeoff between the reconstruction error and the sparseness, tki is the k-th visual cue of the i-th target candidate sampled by particle filter in every frame.

4.2 Observation Likelihood for Particle Filtering

The proposed tracking algorithm is developed in the particle filtering framework. After obtaining the optimal solution in (13), we derive the observation likelihood based on the learned features and classifiers as follows:

Here we use the joint decisions based on the reconstruction error and classification reliability of multiple visual cues to find out the true state of the target. This is because a good candidate should have high confidence to be the label of the target (+1) while it should also have low reconstruction error with multiple features. Therefore, both the representation abilities and discriminabilities of multiple visual cues are jointly exploited for target state estimation. It should be noted that the classification reliability is based on the reconstructed samples from the learned features instead of using the original samples. This is because the classifiers are estimated using the learned features, and it is more reliable to perform classification in the learned feature space.

5 Experiments

In this section, we report the experimental results of the proposed tracker quantitatively and qualitatively.

5.1 Experimental Setting

In this section, we evaluate the proposed tracker using fifteen sequences which covers different kinds of challenging factors including cluttered background, illumination variations, partial occlusion, pose variation, etc. We compared the proposed tracking algorithm with other ten state-of-the-art trackers which include discriminative multiple feature trackers: OAB [Grabner and Bischof, 2006], SemiT [Grabner et al., 2008], MIL [Babenko et al., 2011], generative trackers which explicitly model the noise/outliers: L1T [Mei and Ling, 2011], MTT [Zhang et al., 2013b], feature learningbased trackers: IVT [Ross et al., 2008], and other stateof-the-art methods: CT [Zhang et al., 2012], DFT [SevillaLara and Learned-Miller, 2012], LOT [Oron et al., 2012], STRUCK [Hare et al., 2011]. We use the source codes provided by the authors of these papers and set them to be with the same initialization parameters for fair comparison.

We empirically set ↵1, ↵2, λ1, and λ2 to be 0▷25, 0▷25, 0▷1, 0▷01, respectively. All ✓k(k = 1↪ ▷▷▷↪K) are set to be 1. The training samples Y k (k = 1↪ ▷▷▷↪K) consists of the tracking results of the initial 5 and recent 10 frames, and 10 background samples in the current frame. We implement the `1 tracker [Mei and Ling, 2011] with multiple features to track the target in the initial 15 frames to get target samples for feature learning. We use HOG [Dalal and Triggs, 2005] as global features and covariance descriptors [Tuzel et al., 2006] with log-Euclidean metric [Arsigny et al., 2006] as local features which describe the 2-by-2 non-overlapping parts. So totally 5 kinds of raw visual features are used.

5.2 Experimental results

We adopt two metrics: center location error and success rate for quantitative comparison. The center location error is the Euclidean distance between the center of bounding box and the ground truth. The VOC overlapping rate is defined as area(ROIT T ROIG) area(ROIT S ROIG) , where ROIT and ROIG are the bounding boxes of the tracker and ground-truth. The tracking result of each frame is considered as a success if the overlapping rate is larger than 0.5.

Tables 1 and 2 record the center location error and the success rate on the 15 videos, respectively. The results show that the proposed tracker outperforms other compared trackers on most videos in terms of both two evaluation metrics such that the center location error of the proposed method rank in top three on 12 videos while the success rate ranks in top three on 14 videos. Particularly, the proposed method demonstrates its superior performance in videos which cover cluttered background (e.g. Trellis, Crossing, Bolt), occlusion (e.g. DavidOutdoor, Faceocc), and illumination (e.g. Shaking, Skating1). This is because the feature learning is performed by simultaneously removing contaminated features and imposing discriminability which enables the appearance model to be less sensitive to target samples contaminated by occlusion and large illumination variation and more discriminative under cluttered background. Figure 1 shows the quantitative frame-by-frame comparison result on some challenging videos, i.e. Trellis, Skating1, DavidOutdoor and Bolt. We can see that compared with most of other trackers, the proposed tracker can maintain a relatively low tracking error throughout these videos. We can also see that although the proposed tracker is able to track the target throughout the videos FaceOcc2 and Walking2 which encounter in-plane rotation and large scale change, respectively, it does not achieve good quantitative results. This is mainly because the rotation and scale state of the tracked target is not well modeled by the particle filter used in the proposed tracker. Since we adopt the sparse representation scheme for target representation and the iterative algorithm for optimization, the proposed tracker can not run in real-time speed. The running time is about 3 frames per second.

Figure 2 illustrates some qualitative results on some typical frames including cluttered background and appearance variations caused by illumination, pose, and occlusion. By properly fusing the learned features frommultiple visual cues, the proposed tracker is more stable under some large illumination variations (e.g. Car4#196, Trellis#242). And explicitly modeling the discriminability of the learned features facilitates the resistance to cluttered background (e.g. Shaking#60, Mountain-bike#172). Removing contaminated/corrupted feature for appearance modeling enhances the robustness to occlusion (e.g. DavidOutdoor#88, Faceocc2#162).

6 Conclusion

In this paper, we proposed a novel feature learning framework called robust joint discriminative feature learning for visual tracking with multiple features. By removing the corrupted/contaminated features, introducing the discriminabilities and explicitly modeling the consistency and complementarity in the discriminabilities of multiple visual cues for feature learning in an optimal unified framework, the proposed framework is able to jointly exploit the representation abilities and discriminabilities from multiple visual cues for appearance modeling with multiple features. Extensive comparison experiments with other ten state-of-the-art trackers show its effectiveness and superior performance.

Acknowledgements

This work was supported in part by Hong Kong RGC General Research Fund HKBU 212313 and HKBU 12202514, the Natural Science Foundation of China (No. 61300111) and the Hong Kong Scholars Program (No. XJ2013030). The authors would like to thank the anonymous reviewers for their suggestions on improving the quality of this paper.