Self-paced Convolutional Neural Networks

1 Introduction

In recent years, convolutional neural networks (CNNs) [LeCun et al., 1998] have attracted widespread interests in the field of machine learning and computer vision. CNN is a supervised learning algorithm that aims to learn the mapping between the input data and its corresponding label. In general, CNN consists of three main components: convolution kernels, active function and pooling. Convolution kernels work on small local receptive fields of input data in a sliding-window fashion and then the active function is imposed on the convolution kernel output. Each kernel can be considered as a specific feature detector. The pooling can reduce the data size to make the final prediction be robust to the translation of input data. The parameters of CNN, i.e., the coefficients of convolution kernels and the final fully connected layer, are learnt by using stochastic gradient descent with the standard back-propagation algorithm. The success of CNNs brought about a revolution through the efficient use of graphics processing units (GPUs), rectified linear units, new dropout regularization, and effective data augmentation [LeCun et al., 2015]. Convolutional networks have been successfully applied to various applications, such as image and video classification [Krizhevsky et al., 2012; Karpathy et al., 2014], face recognition [Taigman et al., 2014] and objective detection [Girshick et al., 2014].

∗Corresponding author

The goal of robust learning is to reduce the influence of the noisy and confusing data. The learning robustness relies on a sample selection to distinguish the reliable samples from the confusing ones [Pi et al., 2016]. The recently proposed self-paced learning (SPL) is such a representative method for robust learning. SPL can trace back to curriculum learning (CL) [Bengio et al., 2009] proposed by Bengio et al., in which a curriculum defines a set of training samples organized in ascending order of learning difficulty. However, in CL, the curriculum remains unchanged during the iterations and is determined independently of the subsequent learning [Bengio et al., 2009; Jiang et al., 2015]. Then self-paced learning [Kumar et al., 2010] have been proposed to dynamically generate the curriculum according to what the learner has already learned. SPL is inspired by the learning process of humans/animals, which learns with easier concepts at first and then gradually involves more complex ones into training. Both CL and SPL have been proved to be beneficial in avoiding bad local minima and in achieving a better generalization result [Khan et al., 2011; Basu and Christensen, 2013; Tang et al., 2012b].

Based on the above analysis, we notice that convolutional neural networks and self-paced learning are consistent and complementary. For consistency, both convolutional networks and self-paced learning are biologically-inspired. In fact, CNN is nicely related to the self-paced learning idea that it may be much easier to learn simpler concepts first and then build higher level ones on top of simpler ones [Bengio et al., 2013]. Furthermore, convolutional neural networks and selfpaced learning are complementary because CNNs aim to extract features and self-paced learning tends to explore the data smoothly with more robustness. Therefore the two schemes can benefit from each other.

In this paper, a self-paced convolutional network (SPCN) is proposed to enhance the learning robustness of CNNs. In the proposed SPCN model, each sample is assigned to a weight to reflect the easiness of the sample. Then a convolutional network is established to learn the weighted samples. In SPCN, the weighting and minimization are integrated in a single function by incorporating a novel dynamic self-paced function into the learning objective of CNN. SPCN iteratively updates the model parameters and the weight variable. The learning rates of SPCN are dynamically changed based on the current sample weights. Theoretical studies show that SPCN converges to a stationary solution and is robust to the noisy and confusing data. Experimental results on MNIST and rectangles datasets demonstrate the effectiveness and robustness of the proposed method.

The contribution of this paper are summarized as follows: (1) We improve CNNs with self-paced learning for enhancing the learning robustness of CNNs. (2) A dynamic self-paced function is proposed for the proposed SPCN model. (3) We conduct theoretical studies to show that SPCN converges to a stationary solution and is robust to the noisy and confusing data.

2 Self-paced Learning

Compared to the other machine learning methods, SPL incorporates a self-paced function and a pace parameter into the learning objectives to jointly learn the curriculum and model parameters. Formally, we denote the training dataset as D = {(x1, y1), · · · , (xn, yn)}, where xi ∈ Rm denotes the ith observed sample, and yi represents its label. Let L(yi, g(xi,w)) denote the loss function which calculates the cost between the ground truth label yi and the estimated label g(xi,w). Here w represents the model parameter inside the decision function g. In SPL, variable v is introduced into the learning objective to indicate whether the ith sample is easy or not. The target of SPL is to jointly learn the model parameter w and the latent weight variable v = [v1, · · · , vn] by minimizing:

where v ∈ [0, 1]n and f(v) is the self-paced function. The parameter λ is the "age" of the SPL model to control the learning pace. In the process of the SPL calculation, we gradually increase λ to learn new samples. When λ is small, only "easy" samples with small losses will be considered into training. As λ grows, more samples with larger losses will be gradually appended to training a more "mature" model.

The algorithm of SPL is shown in Algorithm 1. In the inner loop, the model parameter w and the weight variable v are iteratively updated with a fixed pace parameter λ. Then the pace parameter λ increases in the outer loop to progressively approach the rational solution. Kumar et al. proposed a hard weighting scheme [Kumar et al., 2010] to indicate whether the sample is easy or not. Then Jiang et al. proposed three soft weighting methods [Jiang et al., 2014a] to assign real-valued weights to reflect the importance of samples, such as the linear soft weighting, logarithmic soft weighting and mixture weighting schemes. Then several useful sample importance priors have been incorporated into the SPL model, such as spatial smoothness [Zhang et al., 2015], partial order [Jiang et al., 2015] and diversity [Jiang et al.,

Algorithm 1 Algorithm of Self-paced Learning.

Input: The training dataset D, the initial step λ, the step size µ.

Output: Model parameter w.

• 1: Initialize w∗ . 2: while not converged do //Outer Loop 3: while not converged do //Inner Loop 4: Update v∗ = arg minv E(w∗ , v). 5: Update w∗ = arg minw E(w, v∗ ). 6: end while 7: λ ← λ+µ. 8: end while 9: return w =w∗ .

2014b]. In [Meng and Zhao, 2015], Meng et al. have proved that the solving strategy on SPL exactly accords with a majorization minimization algorithm implemented on a latent objective and the loss function contained in the latent objective has a similar configuration with non-convex regularized penalty. SPL has been successfully applied to various applications, such as segmentation [Kumar et al., 2011], domain adaption [Tang et al., 2012a], dictionary leaning [Tang et al., 2012b], long-term tracking [Supanˇciˇc and Ramanan, 2013], reranking [Jiang et al., 2014a], multi-view learning [Xu et al., 2015], action and event detection [Jiang et al., 2014b; 2015; Li et al., 2016], matrix factorization [Zhao et al., 2015; Li et al., 2016], and co-saliency detection [Zhang et al., 2015].

3 Self-paced Convolutional Networks

3.1 SPCNModel

The cost function of CNNs can be expressed as

When the softmax layer is the last layer of a CNN, the cross entropy between the softmax activations α1, · · · , αK of the output neurons and the binary target vector y is - ∑Kk=1 yk ln(αk). In this paper, we propose a self-paced convolutional network (SPCN) to enhance the learning robustness. Figure 1 presents the schematic diagram of the proposed SPCN model. Each sample is assigned to a weight to reflect the easiness of the sample. Then we incorporate a selfpaced function into the learning objective of CNNs to jointly learn the model parameters and the latent weight variable. The proposed SPCN model is shown as follows:

where f(v; t) is a dynamic self-paced function with respect to v and t.

In this paper, we use a majorization minimization algorithm to solve the proposed SPCN model. MM algorithms have been widely used in machine learning and aim to convert a complicated optimization problem into an easy one by alternatively iterating the majorization step and the minimization step. We can get the integrative function of v∗ (λ, L) as

We use Qλ(w|w∗ ) to represent a tractable surrogate for Fλ(L(w)). In [Meng and Zhao, 2015], Meng et al. have proved that

and

In this paper, we denote wt as the model parameters in the tth iteration of the MM algorithm.

Majorization Step

We can calculate v∗ i (λ; Ln(wt )) by solving the following problem to obtain each Q(i) λ (w|wt )

The self-paced function needs to be specified for solving v∗ i (λ; Li(wt )) in Eq. (7). In recent years, several efficient SPL functions have been proposed for various data with different characteristics, such as linear soft weighting, logarithmic soft weighting and mixture weighting [Jiang et al., 2014a]. However, all these methods assign low weights to the easy samples in the early stage of self-paced learning. To address this issue, we propose a dynamic self-paced function for reasonably assigning weights. The proposed dynamic self-paced function f(v; t) is described as follows:

where q(t) > 1, which is a monotonic decreasing function with respect to the time t.

In order to analysis the proposed dynamic self-paced function, we deduce the closed-form solutions of the SPCN model under the proposed dynamic self-paced function. Eq. (8) is a convex function of v in [0,1] and thus the global minimum can be obtained at ∇vE(v) = 0. Therefore we have

The closed-form optimal solution for vi(i = 1, 2, · · · , n) can be written as:

We can also deduce the closed-form solutions of the SPCN model under the hard, linear soft, logarithmic soft and mixture weighting functions. Then we can graph the curves of the sample weight with respect to the sample loss based on the obtained closed-form solutions. Figure 2 shows the comparison of the hard, linear soft, logarithmic soft, mixture weighting and dynamic self-paced functions. Obviously, when q(t) is smaller than 2, the proposed self-paced function is similar to the logarithmic soft weighting method. When q(t) is equal to 2, the proposed self-paced function is to linearly discriminate samples with respect to their losses, which is equivalent to the linear soft weighting method. When q(t) is larger than 2, the proposed self-paced function obtains higher weight values when the loss is low. Therefore both the proposed method and mixture weighting scheme can tolerate small errors.

The proposed SPCN model has some important properties. First, at the tth iteration (or the time t), q(t) can be considered as a constant. The second derivative of Eq. (8) is

Therefore f(v; t) is convex with respect to v ∈ [0, 1]n because the above second derivatives are non-negative and the sum of convex functions is convex. Second, the optimal solution v∗ is shown in Eq. (10). Obviously, vi is decreasing with respect to Li and we have that lim Li→0 v∗ i = 1, lim Li→∞ v∗ i = 0. It indicates that the SPCN model favors easy samples because the easy samples have lower loss values and larger weights. Finally, each individual v∗ i increases with respect to λt in the closed-form solution in Eq. (10). In an extreme case, when λt approaches positive infinity, we have lim λt→∞ v∗ i = 1. Similarly, when λt approaches 0, we have lim λt→0 v∗ i = 0. When the model "age" gets larger, it trends to incorporate more samples into training.

Minimization Step

In this step, we need to calculate

Then the cost function of SPCN is

ECN and ESPCN are the error functions of the standard convolutional networks and the proposed SPCN model, respectively. For a single pattern, the two error functions have the following relationship

where w is the weights in a certain layer of the network, and η is the learning rate. It is obvious that the sample weight vi can control the learning rate and the sample weight vi are constrained in [0,1]. When the samples are easy, the learning rates are high values and the network can update the parameters in a large step. The samples have small learning rates when the samples are difficult. With the small learning rates, the network is able to update the parameters slowly for converging better values.

3.2 SPCN Implementation

It is difficult to decide the increasing pace λt [Jiang et al., 2014a]. In general, we need the range of loss values in advance and give the initial value and the step size. A robust way is to search the solution path with respect to the sample number involved into training instead of extracting the solution path with respect to the pace parameter. Therefore we can predefine a sample number sequence N = {N1,N2, · · · ,Nmaxgen} (Ni < Nj for i < j) representing the number of selected samples in self-paced learning process. Each Nt denotes how many samples will be selected in the tth SPL stage, and Nmaxgen = n means finally all samples are chosen into training. For tth iteration, we can get Lsort by sorting the loss L in ascending order and select the Ntth loss value of Lsort as the values of λt , i.e., λt can be represented as

where Lsort is obtained by sorting the loss L in ascending order.

Algorithm 2 Algorithm of Self-paced Convolutional Networks.

Input: The training dataset D.

Output: Model parameter w.

• 1: Initialize w∗ , and predefine the pace sequence N = {N1,N2, · · · ,Nmaxgen}.
• 2: for t=1 to maxgen do //Outer Loop
• 3: Calculate λt by Eq. (15) and q(t) by Eq. (16).
• 4: while not converged do //Inner Loop
• 5: Update v∗ = arg minv E(w∗ , v).
• 6: Update w∗ = arg minw E(w, v∗ ).
• 7: end while
• 8: end for
• 9: return w =w∗ .

The function q(t) controls the selection of the learning schemes. In the early stage of SPCN, only simple samples are involved into training and their weights should be large. As shown in Figure 2 (b), the self-paced function with large q(t) works better in this stage. During iterations, more complex samples are taken into consideration. Therefore q(t) should be set to a small value. To implement the SPCN model, the solution path is searched with respect to the sample number. Then q(t) can be also defined with respect to the sample number, which is shown as follows

The algorithm of SPCN is shown in Algorithm 2. In the initialization, a sample number sequence N = {N1,N2, · · · ,Nmaxgen} is predefined. In the sequence, we make Ni < Nj for i < j to simulate the increasing process of the pace parameter. Then we obtain λt and q(t) by Eq. (15) and Eq. (16) in the outer loop, respectively. The model parameter w and the weight variable v are iteratively updated with the fixed λt and q(t) in the inner loop.

4 Theoretical Analysis

In this section, we analyze the convergence and properties of the proposed method. Let wt and vt indicate the values of w and v in the tth iteration. As minwE(w, v) is a quadratic programming problem [Jiang et al., 2014a; 2014b], the solution wt is the global optimum, i.e.

f(v, t) is a convex function of v at the time t. Therefore Eq. (3) is a convex funtion of v. Suppose that vt is a solution found by gradient descent, due to the convexity, vt is the global optimum for Eq. (3), i.e.

Substitute Eq. (18) back into Eq. (17), we have

Obviously, the objective values decrease in every iteration in SPCN algorithm. The algorithm is bounded from below because E is the sum of finite elements. Therefore Algorithm 2 can converge to a stationary solution.

The solving strategy on SPL exactly accords with a majorization minimization algorithm implemented on a latent objective and the loss function contained in this latent objective has a similar configuration with non-convex regularized penalty [Meng and Zhao, 2015]. In SPCN, we can obtain the solution v∗ as follows:

We can get the integrative function of v∗ (λ; L) calculated by Eq. (20) as:

where c is a constant.

Now we try to discover more interesting insight under the proposed SPCN method from this latent objective. To this aim, we first calculate the latent losses under the proposed dynamic self-paced function (8) by Eq. (21) as follows:

Note that when λt = ∞, the latent loss Fλt(L) will degenerate to the original loss L. There is an evident suppressing effect of Fλt(L) on large losses as compared with the original loss function L. When L is larger than a certain threshold, Fλt(L) will become a constant thereafter. This provides a rational explanation on why SPCN can perform robust in the presence of noisy and confusing data: The samples with loss values larger than the age threshold will have no influence to the model training due to their 0 gradients. Corresponding to the original SPL model, these large loss samples will be with 0 importance weights vi, and thus have no effect on the optimization of model parameters.

5 Experiments

In order to evaluate the performance of the proposed SPCN, we studied variants of MNIST digits [Larochelle et al., 2007] and the rectangle datasets [Larochelle et al., 2007] in the experiments. The first database consists of five variants of MNIST digits, i.e. {mnist-basic, mnist-rot, mnist-back-rand, mnist-back-image, mnist-rot-back-image}. Each variant includes 10,000 labeled training, 2,000 labeled validation, and 50,000 labeled test images. The second database includes two subsets, i.e. {rectangle, rectangle-image}. The dataset rectangle consists of 1000 labeled training, 200 labeled validation, and 50,000 labeled test images. The dataset rectangleimage includes 10,000 labeled train, 2000 labeled validation and 50,000 labeled test images.

5.1 Analysis of the Pace Sequence

Concerning the influence of the pace sequence, we tested the initial value of the pace sequence in the range N1 ∈ {5500, 6000, 6500, 7000, 7500}. The step size of the pace sequence is same to the batch size. In this experiment, all training samples are exploited to train a CNN and then this trained CNN is used to initialize the proposed SPCN. Figure 3 shows the results of SPCN with different pace sequence on the mnist-basic dataset. As shown in Figure 3, the errors of SPCN with different pace sequences are much lower than these of the initial CNN on the validation and test sets. When the initial value of the pace sequence is small, SPCN needs many iterations to involve all samples into training. We can always obtain satisfactory results compared to the original CNN.

5.2 Analysis of the Dynamic Self-paced Function

In this experiment, we compared the proposed dynamic selfpaced function with the linear soft weighting, logarithmic soft weighting and mixture weighting schemes [Jiang et al., 2014a]. Figure 4 shows the results of SPCN under the four self-paced functions. It can be observed that the errors obtained by SPCN under the proposed dynamic self-paced function are much lower than those of competing methods on seven variants of MNIST and rectangles datasets. In the early stage of self-paced learning, the involved samples tend to be easy and should be given large weights. However, the previous self-paced functions assigned low weights to the easy samples. In the proposed dynamic self-paced function, we dynamically select suitable learning curve shown in Figure 2(b). The proposed dynamic self-paced function gives large weights to the easy samples in the early stage and can reduce the influence of the complex samples during iterations.

5.3 Experimental Results

In this experiment, we compare the results provided by the proposed SPCN model with those obtained by SVMrbf, SVMpoly, NNet, DBN-1, DBN-3, SAA-3 [Larochelle et al., 2007], SDA [Vincent et al., 2008], CAE-1, CAE-2 [Rifai et al., 2011] and GSN [Z¨ohrer and Pernkopf, 2014]. Table 1 shows that the proposed method outperforms baseline methods on six out of seven variants on MNIST and rectangles datasets. The proposed SPCN model ranks first except for the mnist-rot datasets. These statistics indicate that our approach can achieve the higher accuracy and better stability in the statistical sense when compared with the other competing methods.

6 Conclusion

In order to distinguish the reliable data from the noisy and confusing data, we have proposed a self-paced convolutional neural network model. In the proposed SPCN model, the loss of a sample is discounted by a weight. Then a dynamic self-paced function is incorporated the learning objective of CNNs. SPCN uses a majorization minimization algorithm to jointly learn the model parameters of CNN and the laten variable. In the majorization step, SPCN selects the reliable samples based on the acquired model parameters. In the minimization step, SPCN updates the model parameters with the sample weights. The leaning rates are dynamically changed based on the current sample weights during iterations. The complex samples always have small learning rates to get better results.

Our theoretical analysis shows that SPCN can converge to a stationary solution and SPCN can perform robust in the presence of noisy and confusing data. Furthermore, we also achieved state-of-the-art performance on variants of MNIST digits and the rectangle datasets. The proposed method can enhance the learning robustness of convolutional neural networks. In future research, we will explore other deep architectures with self-paced learning.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Grant no. 61422209, 61672409), and the National Program for Support of Top-notch Young Professionals of China.