Learning Robust Gaussian Process Regression for Visual Tracking

1 Introduction

Visual object tracking is one of the fundamental problems in computer vision with many applications. In generic visual object tracking, given the initial state (e.g., position and size) of a target object in the first frame, the task is to estimate the states of the target in the subsequent frames. It is commonly known that despite significant progresses in recent decades, visual object tracking is still a challenging problem due to some extremely challenging factors (e.g. large appearance changes, occlusions, background clutters and fast motion) and very limited sample datas [Wu et al., 2015]. Therefore, it is crucial to research how to construct a robust appearance model from very limited samples to distinguish target from background in visual tracking.

Recently, Correlation Filter based trackers (CF trackers) have shown high accuracy and robustness when tracking different types of targets. In CF trackers, they mainly learn the filter from circular samples to regress a gaussian response map in fourier domain, and the process of learning and target detection can be accelerated by using the Convolution Theorem and Fast Fourier Transform (FFT). [Bolme et al., 2010] proposed to learn a Minimum Output Sum of Squared Error (MOSSE) filter for visual tracking on gray-scale images. They used a base image patch and the circulant virtual ones to train the filter directly in the Fourier domain. [Henriques et al., 2015] proposed Kernelized Correlation Filter (KCF) which not only expended the MOSSE tracker to learn multi-channel filters on multi-dimensional features, but also took advantage of the kernel trick. Compared to MOSSE, the tracking performance had been greatly improved by KCF when using the HOG [Dalal and Triggs, 2005] feature and gaussian kernel on OTB-2013 [Wu et al., 2013]. However, the basic periodic assumption leads to an inaccurate representation of the real samples, and produces unwanted boundary effects in CF trackers. This problem obviously reduces the discriminative power of the learned model and severely degrades the performance of standard CF trackers. In order to relax the boundary effect, [Kiani Galoogahi et al., 2015] and [Danelljan et al., 2015] respectively proposed Correlation Filters with Limited Boundaries (CFLB) and Spatially Regularized Discriminatively Correlation Filters (SRDCF). In CFLB, Galoogahi et al. added a rectangular window to the circular samples in CF trackers, and then used the Alternating Direction Method of Multipliers (ADMM) algorithm to solve a objective optimization problem with equality constraints. In SRDCF, Danelljan et al. added a spatial regularization component in the standard CF formulation to penalize the correlation filter coefficients depending on their spatial location in the learning process. Even though the CFLB and SRDCF relaxed the boundary effect in CF trackers, they still can not solve the problem thoroughly. Furthermore, CFLB and SRDCF can not use the kernel trick to improve the accuracy further [Henriques et al., 2015], because there is no way to calculate the value of the kernel after they add a window to sample space or filter space.

∗This work was supported by National Natural Science Foundation of China under Grants 61772527 and 61375035.

In this paper, to solve the problems in CF trackers, we apply the Gaussian Process Regression (GPR) [Rasmussen and Williams, 2006] to visual tracking and propose a novel GPR based tracker (GPRT) which is a conceptually natural tracking approach. In our GPRT, following the basic GPR theory, we construct gaussian covariance matrix using training samples and employ the constructed GPR model to predict the regression values of the test samples in the detection process. Through above, our GPRT does not have the boundary effect and we can also promote the tracking performance by utilizing the kernel trick (see Figure. 1). For the sake of simplicity, in experiments we only use single gaussian kernel with hand-crafted features (i.e. HOG [Dalal and Triggs, 2005] and Color-Names [Danelljan et al., 2014b]). As for the parameters in gaussian kernel, it is time-consuming to solve the maximum likelihood to get the optimal parameters for each frame, and we found that there is no significant difference in tracking performance between only solving them in the first frame and just like the KCF to design them by the rule of thumb. Additionally, considering the importance of the update method in visual tracking, we present two different update methods which are efficient and effective for updating our GPRT in the tracking process. The one is based on the directly apply GPR to the tracking problem which mainly update the appearance model in order to make the detection process faster, and the other, which not only the appearance model but also the GPR model needed to be updated in tracking process, is based on the relationship between GPR and Kernel Ridge Regression (KRR) [Rasmussen andWilliams, 2006]. It is worth mentioning that our GPRT can be expanded to multi-kernel, conveniently. Furthermore, convolutional neural networks (CNN) features can also accessibly be added to our approach, even though they may have different resolutions in different layers.

In summary, we make the following three contributions.

• We propose a novel and conceptually natural tracking approach, called GPRT, which mainly apply the GPR to object tracking.
• We present two different update methods for our GPRT, and these methods are proved to be effective and effi-

cient in experiments.

• We perform our GPRT on two public datasets: OTB2013 and OTB-2015, and our GPRT achieves state-ofthe-art results. Without bells and whistles, on these two datesets, our GPRT obtains 84.1% and 79.2% in mean overlap precision, respectively, and outperforms all the existing trackers with hand-crafted features.

We will release code to facilitate future research.

2 RelatedWork

[Rasmussen and Williams, 2006] briefly introduced what GPR is and how to use it in practice to solve regression problem. Different from general tasks based on the regression problem, image processing, especially video analysis task, has the characteristics of needing to deal with a large number of high-dimensional data. There are many approximation schemes which rely on the use of a series of inducing points to reduce the high computational complexity when applying the GPR to large data [Qui˜nonero-Candela and Rasmussen, 2005]. [Titsias, 2009] suggested a variational approach which provides an objective function to optimize these inducing points. In addition, [Hensman et al., 2013] proposed Stochastic Variational Inference (SVI) for GPR to extend the variational idea. They showed how the variational objective could be reformulated with additional parameters to enable stochastic optimization. Furthermore, they trained the GPR model using mini-batches which allowed them to learn from a large dateset which contains 700,000 samples. However, their method may not suitable for object tracking problem, because we need a simple and efficient online update method for object tracking in order to make the tracking model adapt to the change of the target appearance. Unlike [Hensman et al., 2013], for applying GPR to object tracking, we propose two efficient and effective methods to update the GPR model, and after effective update we can gradually obtain better generalization ability of the GPR model through a large number of data in online tracking process.

In recent years, there are a few researches applying the GPR to the visual tracking problem. [Gao et al., 2014] proposed the TGPR which directly analyzes the probability of target appearance using the GPR. They divided the labeled samples into auxiliary and target samples, and learned the observation model for regression in a semi-supervised fashion. Unlike TGPR, we apply all the known samples as many as possible to construct and update the GPR model, and our online learning and update method are effective and efficient. The experiments confirm that in the mean distance and overlap precision, our GPRT obviously exceeds (about 11% and 14%, respectively) the TGPR on OTB-2013 dataset 1 .

3 ProposedMethod

In this section, we firstly review the gaussian process regression, and secondly introduce our GPRT tracking framework including online learning, update and detection, finally we show how to accelerate calculation in our GPRT.

1 The TGPR does not report its performance on OTB-2015 dataset, so here we only compared with it on OTB-2013 dataset.

3.1 Gaussian Process Regression

For a useful introduction to GPR, we suggest the reader referring to [Rasmussen and Williams, 2006; Ebden, 2015].

Traditionally parametric models can be used for regression and classification problems and they have an advantage in ease of interpretability. However, for complex datasets, simple parametric models may lack expressive power, and even though their more complex counterparts (such as neural networks) may have a stronger expressive power, they may not easily work in practice. In order to solve the problems above, the advent of GPR has opened the possibility of flexible models which are practical to work for complex datasets [Rasmussen and Williams, 2004].

Given a training set D = {(xi, yi) | i = 1, ..., n}, where X = {x1, ..., xn} ∈ RD×n is the matrix of input data, y = {y1, ..., yn} ∈ Rn is the vector of desired regression values with respect to X and D is the dimension of input data, the main assumption of GPR is y = f (x) + ϵ where the output y are considered as the points sequence sampled from a multivariate gaussian distribution, and ϵ ∼ N (0, σ2n ) represents the noice in training outputs. Moreover, it can be assumed that this gaussian distribution has the mean of zeros and a commonly used covariance function is the Radial Basis Function (RBF) with additive white noise and it can be expressed as:

where the hyperparameter σf governs the magnitude of covariance, l modulates the exponential decay of covariance, and σn describes additive white noise.

In GPR method, the reliability of our regression is dependent on how well we select the covariance function, so it is necessary to estimate the values of the hyperparameters in kernel. According to the GPR theory, the optimal hyperparameters of RBF corresponding to maximizing:

where θ = {l, σf , σn}.

According to above, given the training samples and labels, it is necessary to establish three covariance matrices as follows when we would like to estimate the regression value of the test samples.

⎡⎢⎢⎢⎣

k (x1, x1) k (x1, x2) ... k (x1, xn)

k (x2, x1) k (x2, x2) ... k (x2, xn)

...

...

...

...

...

...

...

...

⎤⎥⎥⎥⎦

k (xn, x1) k (xn, x2) ... k (xn, xn)

where x∗ is the sample to be predicted. And then, the probability of y∗ follows a gaussian distribution:

K =

(3a)

Namely, the mean and variance of y∗ can be formulated as:

where ¯ y∗ and var (y∗) are the expectation and variance of the test sample x∗, respectively.

3.2 Our Tracking Framework

Here, we predefine the KXiXj as follows.

where Xi and Xj come from the different frames when i ̸= j or same frame when i = j, and xin represents the n-th sample in Xi. (6) represents the constructed covariance matrices with Xi and Xj, and it is often used in subsequent sections.

Online Learning. For each frame fi, where i = 1, ..., T , assuming that the target position is pi and the area of the target is si = wi × hi. We densely sample the training data around pi with sampling region Si = σ√wi ×hi × σ√wi ×hi and all samples Xi = {xij | j = 1, ...,N} in fi have the same size si. Then, we straightforwardly construct the covariance matrix KXiXi based on training samples Xi, and in common with KCF to construct the gaussian like label yi for calculating the expectation of test samples.

Online Update and Detection (I) Considering that the general object detection method based on linear weighted in tracking can be formulated as:

where Xt represents the test samples in the current frame ft, Xi (i < t) represents the training samples in the previous frames fi (i < t), βi represents the weight of fi, h(Xt,Xi) represents the predicted expectation of Xt based on Xi, and according to (5a) in our GPRT it can be formulated as:

where yi is the label vector which obeys the gaussian distribution, and in our GPRT yi = yj , ∀i, j.

Combining (7) and (8), it is easy to obtain the detection formula in our GPRT as follows.

In (9), we can observe that for getting f (Xt) we need to construct the KXtXi as well as KXiXi and calculate the K-1 XiXi yi as well as KXtXi (K-1 XiXi yi ) for all i < t. Compared to the other three operations, the complexity of calculating KXtXi (K-1 XiXi yi ) can be negligible. Despite constructing KXiXi and calculating K-1 XiXi yi can be done only once then saved, we still need to utilize (6) (t -1) times to construct the KXtXi (i < t) in each prediction process and it is extremely time-consuming when t and N is large.

For applying the GPR to tracking, we propose (9) can be approximately replaced by the following expression:

and after such a simple approximation we have been greatly reduced the amount of calculation in the process of detection.

After we replace the f (Xt) in (7) with (10), it is easy to obtain the ultimate detection and update formula as follows.

where δ represents the learning rate in tracker, moreover the relationship between {βi | i = 1, ..., t - 1} and δ can be described as:

Following (11), we can not only predict the regression value of test samples quickly by our GPRT model but also update our GPRT model in an efficient way. It is worth noting that this scheme allows the model to be updated without storing all the previous information, and only the current model At and Bt need to be saved for predicting the next frame. By using this predict and update method for our GPRT, we call it GPRTE in the following sections.

Online Update and Detection (II) In the Kernel Ridge Regression (KRR) [Vovk, 2013], we are mainly solving empirical risk minimisation in dual space as:

where φ(xi) is mapping the input of a linear problem to a non-linear space.

Considering the relationship between the GPR and KRR [Rasmussen and Williams, 2006], it is easy to prove that if in KRR we use the same kernel as the covariance function in GPR and moreover, the regularisation parameter λ in KRR is the same as the noise variance σ2n in GPR, then the GPR posterior mean coincides with the KRR estimates of the function. For this reason, we wish to take advantage of the relationship between the GPR and KRR to get the update method when applying the GPR to tracking.

For tracking, we can formulate the cost function of KRR in dual space as:

where P represents the number of frames, N represents the number of training samples in each frame, xmj and ym j represents the m-th sample in j-th frame and its label, similar to (7), βj represents the weight of each frame, α = {α1, α2, ..., αN} represents the dual variables to be optimized and the solution is restricted to only contain one set of dual variables for keeping the simplicity

The cost function in (14) is minimized by:

According the relationship between GPR and KRR and compared (5a) and (15), we can naturally bring (15) into (8), and utilize the approximate method in (10) as well, then it is easy to obtain the ultimate detection and update formula as follows.

Compared to GPRTE, this update approach considers the relationship between GPR and KRR, moreover, from the perspective of minimizing empirical risk directly derived how to update the GPR model during tracking and it is slightly more complex than GPRTE. Similar to GPRTE, this scheme also allows the model to be updated without storing all the previous information, and only the current model At, Mt and Nt need to be saved for predicting the next frame. By using this update method for our GPRT, we call it GPRT in the following sections.

3.3 Accelerated Calculation

As we know, in GPR, when the number of training samples and the feature dimension of each sample are large, it is highly time-consuming to calculate the gaussian covariance matrix K. However, it is not difficult to observe that there are a large number of parallel operations in calculating the K. Assuming that the target size is w × h, the sampling region is W ×H = σw × σh, the channels of hand-crafted features is d, so the feature dimension of each sample is d×w×h and the number of samples is (W - w + 1)×(H - h + 1). To establish the gaussian covariance matrix K, the time complexity is about O (dwhWH) = O (dw2h2σ2 ). We propose that using the GPUs allows the K to be computed in the time complexity O (dw2h2σ2/P ) instead of O (dw2h2σ2 ), where P is the maximum degree of parallelism in GPU. In practice, w and h

Algorithm 1 Proposed tracking framework.

Input:

Initial target location and scale x1 = (p1, s1);

Output:

Estimated target location and scale xt = (pt, st);

• 1: Compute µ = √min (4000,max (s1, 1000)) /s1 as the scaling ratio to ensure the speed and accuracy;
• 2: repeat
• 3: Crop out the square searching region St in frame t based on xt-1 and crop ratio σ;
• 4: Resize St according to the s1 and scaling ratio µ;
• 5: Extract the HOG and Color-Names features from St and do L2 regularization to 1 and 0.4, respectively;
• 6: Compute the regression value of test samples using (11a) or (16a) and find the maximum value to locate the new position of the target pt;
• 7: Build the scale pyramid based on pt and st-1, then using DSST to estimate the scale of the target st;
• 8: Crop out the square learning regionLt in frame t based on xt and crop ratio σ;
• 9: Do 4 and 5 operations on Lt;
• 10: Update the GPRT model using (11) or (16);
• 11: Build the scale pyramid based on pt and st, then update the DSST model;
• 12: until End of video sequences;

are about 15 (HOG and Color-Names features with cell size 4), σ is about 4, d is about 40, and P is about 3600. So, we can probably reduce the time complexity from O (dw2h2σ2 ) to O (dwh) by GPU which allows us to perform about 3000 times faster than before.

4 Implementation Details

We present an outline of our method in Algorithm 1 and more implementation details are discussed as follows.

Features. For simplicity, our GPRTE and GPRT are both based on hand-crafted features. We use 31-channels HOG [Dalal and Triggs, 2005] and 10-channels ColorNames [Danelljan et al., 2014b] features since they show high robustness in object tracking [Danelljan et al., 2016a; Tang and Feng, 2015; Danelljan et al., 2016a]. Similar to MKCF tracker [Tang and Feng, 2015], we also set the cell size of 4 × 4 in both HOG and Color-Names features, but unlike it, we normalize the HOG and Color-Names features and make their L2-norm equal to 1 and 0.4, respectively.

Scale. As for scale estimation, we use DSST [Danelljan et al., 2014a] and similar to it we only use HOG features.

Parameters Setup. We set the learning rate δ in section 3.2 to 0.004 and 0.007 for our GPRTE and GPRT, respectively. Meanwhile, we set the learning and search ratio σ in section 3.2 to 4 for both GPRTE and GPRT 2 . For accuracy and speed, we resize the target in the first frame to ensure the minimum and maximum area are 1000 and 4000 pixels, respectively, and in the subsequence, to ensure the number of the training samples in each frame are the same, and the feature dimension of each sample are the same too, we resize the target in each frame to the same as the size in the first frame.

2 In fact, because of our GPRT does not exist the boundary effect, this characteristic allow us to set up a wider range to learning and detection. But considering the speed of the tracker, we set the learning and search ratio similar to SRDCF.

Kernel Selection. We use single gaussian kernel described in (1). In gaussian kernel, we set σf = 1.0 and σn = 0.01, and even though the parameter l can be obtained by maximization (2) in the first frame of every sequence, we observe that this parameter roughly presents a gaussian distribution centered at 1.4 on OTB-2015 dataset, therefore for the sake of simplicity we set it to 1.4 on all sequences.

Platform. Our GPRTE and GPRT are both implemented under MATLAB and C++. The experiments are performed on Linux with IntelE5-2673 2.4GHz CPU and single TITAN X GPU with CUDA-8.0.

5 Experimental Results

We show the performance of our GPRTE and GPRT on two public benchmarks: OTB-2013 [Wu et al., 2013] and OTB2015 [Wu et al., 2015], and compare them with the state-ofthe-art trackers with hand-crafted features. All parameters of our GPRTE and GPRT are kept consistent across all experimental comparisons. The mean fps of our GPRTE over the OTB-2015 sequences is about 7 and GPRT is about 5.

All trackers are quantitatively evaluated by six metrics, (i) Center Error, which is calculated as the average Euclidean distance between the centers of located objects and their ground truths in a sequence; (ii) Distance Precision, which is the percentage of frames where the objects are located within the center errors of 0 to tc pixels, with tc = 20; (iii) Precision Plot, which is simply a curve of the distance precisions with tc changing from 0 to 50 pixels; (iv) Overlap Ratio, which is defined as the average ratio of intersection and union of the estimated bounding box and ground truth in a sequence; (v) Overlap Precision, which is the percentage of frames with the overlap ratio exceeding to in a sequence, with to = 0.5; (vi) Success Plot, which is simply a curve of overlap precisions with the overlap ratio changing from 0 to 1, and AUC is the area under the success plot.

Overall performance. Our tracker, denoted by GPRT and GPRTE, are compared with the other seven state-of-the-art trackers, including ECOHC [Danelljan et al., 2016a], SRDCFdecon [Danelljan et al., 2016b], SRDCF [Danelljan et al., 2015], Staple [Bertinetto et al., 2016], LCT [Ma et al., 2015], KCF [Henriques et al., 2015] and DSST [Danelljan et al., 2014a]. All trackers are based on the hand-crafted features, and all except KCF have the scale estimation part.

Table. 1 shows the mean Overlap Precision (OP) of our GPRT and GPRTE, as well as the other state-of-the-art trackers on OTB-2013 and OTB-2015 datasets. The best results on these two datasets are obtained by our GPRT with a mean OP of 84.1% and 79.2% respectively, leading to a significant gain of 4.2% and 1.5% compared to the second best tracker except for our GPRTE. In addition, our GPRTE also ahead of the other state-of-the-art trackers on both datasets in mean OP.

Figure. 2 shows the average precision and success plots of our GPRT and GPRTE, as well as the other state-of-the-art trackers on OTB-2013 and OTB-2015 datasets, respectively. We also report the distance precisions and AUCs in the legend. Our GPRT and GPRTE have the similar performance on both datasets. Our GPRT obtains the distance precision score of 86.7% and 84.3% as well as the AUC score of 67.7% and 65.5% on OTB-2013 and OTB-2015 datasets, respectively. Furthermore, our GPRT and GPRTE outperform the best existing tracker (ECOHC) with hand-crafted features.

Compared to all the existing CF trackers which achieve the state-of-the-art performance, the boundary effect is eliminated thoroughly and the kernel trick can be employed in our GPRT and GPRTE, therefore our GPRT and GPRTE can achieve better tracking performance than them.

Attribute-based evaluation. The videos in the benchmark dataset OTB-2015 are annotated with 11 attributes which are useful for analyzing the performance of trackers in different aspects. Following [Ma et al., 2015], we also compared our GPRT and GPRTE to the other state-of-the-art trackers on 8 main challenging attributes of OTB-2015 dataset. Figure. 3 shows the success plots and AUCs of our GPRT and GPRTE as well as the other state-of-the-art trackers on 8 main challenging attributes, respectively. It illustrates that our GPRT and GPRTE outperform the other state-of-the-art trackers in most cases.

6 Conclusion

A novel tracking framework, GPRT which applying the Gaussian Regression Processes to visual tracking, has been presented in this paper. Compared to all the existing CF trackers, our GPRT not only does not exist the boundary effect, but also can take advantage of the kernel trick at the same time. In addition, we propose two different efficient and effective update methods for our GPRT. We perform comprehensive experiments on two benchmark datasets: OTB-2013 and OTB2015. Our GPRT outperforms all the existing trackers with hand-crafted features on both two datasets.