Adaptive Sharing for Image Classification

1 Introduction

Image classification is a core problem in computer vision and artificial intelligence. Learning with a number of class labels poses a great challenge on traditional multi-class classification models, where training multiple classifiers is typically independent or mutually exclusive. The task of distinguishing one class from hundreds of class labels would be difficult, especially when its training data is insufficient.

Much of the effort in deploying algorithms is devoted to leveraging rich task relationship to transfer information [Caruana, 1997; Thrun and Pratt, 1998; Evgeniou and Pontil, 2004]. The learning paradigms aim to achieve better generalization performance by encouraging common knowledge to be shared across related ones. Accordingly, the crucial aspect is the introduction of hypothesis to model the relatedness among tasks. For example, widely used assumptions that task parameters (i.e., classifier parameters) either lie in a common feature subspace [Obozinski et al., 2006; Argyriou et al., 2008; Liu et al., 2009] or share a common probabilistic prior [Yu et al., 2005; Fei-Fei et al., 2006], are essentially based on the hypothesis that all tasks are related and all relevant information should be shared. When such strong assumptions do not hold, such sharing may incur adverse effect on overall performance. To avoid this, some methods have been proposed to discover specific structures, such as outliers [Gong et al., 2012; Pu et al., 2014] or disjoint groups [Zhou et al., 2011; Kang et al., 2011; Srivastava and Salakhutdinov, 2013].

∗Corresponding author.

Regarding the realistic classification problems, it is complicated to model class relatedness in the target space. Determining how to share is hard to be addressed accordingly. For example, the introduction of disjoint group structure reflects the desire of intra-group sharing, which drives the classifier parameters in a group close to each other. As a matter of fact, the sharing among classes usually forms a continuum in the more realistic setting. Some classes are less related than others even if they are partitioned to a group. On the other hand, it is imperative to highlight the specific features of one class against others even if they have much in common, since the original goal of developing model is to distinguish the set of classes. A robust method is needed to effectively share information and identify individual difference.

In this regard, we propose an Adaptive Sharing method for image classification. A distinct insight from our method is to selectively share information among classes while capturing respective disparities simultaneously. The learning model is expected to leverage feature relevance when it exists, but not require it strictly satisfying certain structure. This goal is achieved by estimating a composite of two parameter sets with different types of regularization. The classifier parameters for all classes are decomposed into two parts: one corresponds to the shared features and the other corresponds to the class-specific features. A nuclear norm penalty is exploited on the first part to capture the underlying relatedness structure among classes and an element-wise sparsity penalty is imposed on the second part to highlight the disparities of each class. The objective is formulated as a non-smooth convex optimization problem when given the feature space. We adopt the accelerated proximal gradient algorithm [Beck and Teboulle, 2009] to solve the problem.

Besides, we propose to learn the classifier parameters in the context of deep architectures (e.g., convolutional neural networks) by incorporating Adaptive Sharing. Convolutional neural networks with discriminative training have obtained state-of-the-art results on many classification problems. Our method augments the network by encoding the class relatedness. We also develop an effective way to solve the optimization by using stochastic gradient descent algorithm with approximation on low-rank constraint. In summary, the main contributions in this paper are as follows:

• We propose a novel Adaptive Sharing method which selectively shares information among classes and captures the class-specific properties simultaneously. Compared with imposing strong assumptions or discovering specific structures, we provide an elegant way to appropriately transfer knowledge among classes.
• Besides applying it for learning classifiers on pre-computed features, we also integrate Adaptive Sharing with deep neural networks, where the performance can be improved by encoding class relatedness structure. Consequently, we develop different optimization strategies in the two scenarios. Experimental results on multiple challenging datasets demonstrate the efficacy of such selectively transfer for improving the overall classification performance.

2 RelatedWork

Image classification in real world scenario has drawn increasing attention. Complex appearance variations and class correlation bring in the difficulties for classifying many classes. Much work is proposed to study and exploit the relatedness among classes to transfer information in a multi-task learning paradigm. A family of methods are developed based on sharing a prior in the hierarchical Bayesian framework [Fei-Fei et al., 2006; Archambeau et al., 2011]. Another direction is formulating the approaches in the regularization framework where the tasks are assumed to lie in a common feature subspace [Argyriou et al., 2007; Liu et al., 2009]. However, these methods typically assume strong shared relationship among tasks, which might degrade the overall performance due to the information transfer among unrelated tasks. Some methods are further proposed to discover relatedness structure for sharing. For example, a mixed penalty is adopted in [Mei et al., 2012], outlier tasks are detected in [Gong et al., 2012; Pu et al., 2014] and task grouping is learnt in [Zhou et al., 2011; Kang et al., 2011]. Different from our method, these methods encourage the relatedness satisfying certain structural bracket. Moreover, the work in [Jalali et al., 2010; Chen et al., 2012] captures the inherent relationship among tasks while allowing the existence of different features. The two work provide theoretical analysis based on the linear feature space, whereas we are concerned with a more realistic problem and further incorporate our method with deep architectures.

Alternatively, much effort is devoted to developing feature representation learning [Krizhevsky et al., 2012; Lin et al., 2014; Lee et al., 2014; Stollenga et al., 2014; He et al., 2014;

Simonyan and Zisserman, 2014], which achieves state-of-theart performance on image classification. However, the development of these models does not take advantage of class relatedness. The work [Deng et al., 2014] exploits semantic prior in the deep model, which is different from implicitly learning the relatedness structure in our method. In [Srivastava and Salakhutdinov, 2013], a group-based structure is estimated with deep model iteratively. As any change on class partitioning would lead to the re-training of overall network, the method suffers from considerable time cost for reaching a plateau and might be intractable for dealing with many classes. Our method adopts a more concise and effective way to combine the class relationship.

3 Adaptive Sharing Approach

Assume we have a set of training images X = {xi}Ni=1 . Y = {yi}Ni=1 is the corresponding label set. yi is a K dimensional vector (whose value can be binary or one-of-K) for indexing target classes. The learning of each classifier is regarded as a single task. Notation ‖ · ‖1, ‖ · ‖F and ‖ · ‖∗ denote the ℓ1 norm, Frobenius norm and nuclear norm of matrix [Lin et al., 2011], respectively. The nuclear norm is the sum of singular values.

In this section, we first describe our method based on precomputed features and the optimization strategy by using accelerated proximal gradient algorithm. Then we incorporate Adaptive Sharing with convolutional neural networks and present the corresponding optimization strategy.

3.1 Learning on pre-computed features

Suppose each image x ∈ X has been represented by a precomputed feature vector x ∈ RD . Let wk ∈ RD denote the corresponding parameter vector of task k (i.e., the classifier parameters of class k) which is a column of the parameter matrix W ∈ RD×K . In the standard learning paradigm where each task is learnt independently, the objective function typically takes the form:

where λ is the regularization factor, and ℓ(wT k xi, yk i ) denotes the loss between the prediction wT k xi and the true value yk i . When considering feature selection [Obozinski et al., 2006], ℓ1 norm regularization is utilized instead.

In multi-task learning paradigm, the tasks are expected to learn jointly and share a common feature subspace, such as imposing a low-rank structure [Argyriou et al., 2008]:

Such a regularizer enforces the strict sharedness, which might drive multiple classifiers too close. However, with respect to classification problems, the essential goal is to distinguish between classes. For example, it is imperative to highlight the features differentiating "Siberian husky" and "Malamute" although they share much common information. Thus, we suppose that each task depends on the shared information and the additional specific properties.

Here we leverage a composite of two parameter vectors, ck and sk, to represent the classifier parameters towards class k, i.e., wk = ck + sk. The two components correspond to the shared features and the specific features, respectively. Let C = (c1, · · · , cK) and S = (s1, · · · , sK). We introduce different structures on the matrics C and S. A low-rank structure is exploited to capture the inherent relatedness among tasks and an element-wise sparsity structure is leveraged to highlight the disparities of each task simultaneously. Formally, the learning problem can be formulated as:

where λc and λs are the penalty factors. The nuclear norm penalty has the effect of encouraging a low rank solution on matrix C for coupling the related tasks. The ℓ1 norm penalty is adopted to characterize the specific features by encouraging the sparsity on matrix S.

To solve the optimization problem (3), we use accelerated proximal gradient algorithm [Beck and Teboulle, 2009]. Let L(X , Y; C, S) symbolically denote the empirical loss:

Z denote the variables to be optimized,

g(·) and h(·) refer to the smooth term and non-smooth convex term, respectively,

The updating at the t-th iteration performs as follows:

where ηt denotes step size. g(Z) is approximated by a quadratic local model around Ut. The variable Ut can be set as a combination of Zt and Zt-1 from previous iterations:

where bt = (1+ √4b2t-1 +1) /2 for t ≥ 1, and b0 = 1. Considering that (7) takes an equivalent form:

ηtλs. We can leverage the decomposability in (9) to optimize variables C and S separately, and the closed-form solutions can be obtained [Lin et al., 2011], respectively.

3.2 Integrating Adaptive Sharing with Deep Neural Networks

Deep neural networks have shown strong power on image classification. We aim to integrate Adaptive Sharing into deep neural network framework to augment the network by encoding the relatedness among classes. Our model can be implemented as a standalone layer in such a framework. We exploit it to replace the last full-connected layer in deep neural networks (such as convolutional neural networks [Krizhevsky et al., 2012]), that is to say, the last layer weight parameters (connected to K nodes corresponding to the K classes) are comprised of two components.

The neural network can be regarded as a feature space projection for each image xi. Different from learning classifiers on pre-computed features (in Section 3.1), the parameters in the projection should be learnt jointly. Let θ denote the parameters in the network except the ones at the last layer. Then the empirical loss in (4) can be reformulated with negative log-likelihood:

θ can be regarded as the parameters to generate the representation for image xi. The matrices C and S compose the last layer weight parameters, i.e., wk = ck + sk, which denote the weight parameters towards class k.

The optimization can be done using stochastic gradient descent method with mini-batches. However, the problem is non-trivial due to the low-rank constraint on the matrix C. Therefore, we adopt an operator Ωε(C) [Mei et al., 2012] to approximate the nuclear norm penalty λc‖C‖∗:

where ε is the approximation factor. The approximation Ωε(C) is convex and smooth with respect to C. Then the gradient can be computed as:

where Q∗ = arg minQ ( 1 2ε ‖Q-C‖2F + λc‖Q‖∗ ). With respect to nuclear norm, Q∗ can be computed with a closedform expression by utilizing the soft-thresholding operator on the singular values of the matrix C [Lin et al., 2011]. Consequently, the gradient of the regularized loss over a batch of data D = {xi, yi} is estimated as:

where 〈·〉D denotes the average operator over the batch D. The update rule for the parameters S and θ follows the standard algorithm [Jia et al., 2014].

4 Experiments

We present extensive empirical studies to evaluate our Adaptive Sharing learning in two scenarios: pre-computed features and deep neural networks. We first compare our method with some representative multi-task learning methods in the finegrained classification problem and assess the performance on a widely used dataset CUB-200-2010 [Welinder et al., 2010]. Moreover, we evaluate it on two challenging datasets, CIFAR-100 [Krizhevsky, 2009] and ImageNet 2012 classification dataset [Russakovsky et al., 2014], in the context of deep architectures.

4.1 Evaluation with Pre-computed Features

The CUB-200-2010 (Birds200) is a widely used dataset for fine-grained classification (FGC). It contains 6,033 images of birds belonging to 200 species, where only 15 images per class are used for training, the rest are used for testing. Some example images from the dataset are shown in Fig. 1. Each image is first cropped against the provided bounding box and resized such that the longer side is no more than 300 pixels. Our method does not specify features, and we use kernel descriptors (KDES) [Bo et al., 2010] as the imagelevel representation. Specifically, four types of the KDES are applied: gradient-based, color-based, normalized colorbased, and local-binary-pattern-based. The patch size is set to 16 × 16, and the stride is set to 8 pixels. We adopt the squared hinge loss [Yang et al., 2009]. We change λc from 0 to 1.0 with step 0.1, and λs from 0.001 to 1.0 with ratio 10.

We compare our Adaptive Sharing with three representative multi-task learning methods (MTL): JFS [Argyriou et al., 2007], CMTL [Zhou et al., 2011] and GMTL [Pu et al.,

2014]. All the methods are based on the same KDES features as ours. The regularization parameters are chosen by cross validation on the training set for all the MTL methods. In order to comprehensively assess the results, we also provide some FGC methods as baselines. Recently, several work exploits the techniques of segmentation and localization on object parts to obtain strong performance on the dataset, such as [Chai et al., 2013; Gavves et al., 2013]. These methods contain extra procedures, and the results highly depend on the quality of segmentation and localization. Thus, we leave them out of the comparison, however our advantage is complementary to their strength of modeling object parts.

The results of different methods on overall 200 classes are summarized in Table 1. Our result is obtained when λc = 0.8 and λs = 0.1. The performance is measured by accuracy. Adaptive Sharing achieves better result than baseline methods. In particular, we can observe that it clearly outperforms baseline MTL methods. Compared with the FGC methods, baseline MTL methods achieve marginal improvement, even worse performance. Complex intra-class variations and interclass correlation incur negative transfer in these methods, which hurts the overall performance. In contrast, our method appropriately exploits shared information by jointly capturing the task relationships and individual disparities.

To further evaluate the performance among the MTL methods, we test them on a frequently used subset of CUB- 2002010 dataset, Birds14. The subset is comprised of 14 species birds, which are from two families: vireos and woodpeckers [Duan et al., 2012]. Following [Duan et al., 2012], the evaluation is measured by Average Precision (AP) on each class. We report the mean AP (mAP) as the overall performance of different methods in Table 1. Our result is obtained when λc = 0.1 and λs = 0.01. Adaptive Sharing achieves an outstanding result compared with these baseline methods. Moreover, we list the per-class results of our method in Fig. 2. As shown in the figure, the performance of our method is consistently better than CMTL and GMTL on almost all of the species. This further illuminates that our method benefits from appropriately sharing information, and hence can effectively improve the generalization performance.

4.2 Evaluation on Deep Neural Networks

The CIFAR-100 dataset [Krizhevsky, 2009] is composed of 32 × 32 color images belonging to 100 classes, with 50,000 images for training and 10,000 images for testing. We choose this dataset because there are a large number of classes but each one has a few samples, making it suitable for demonstrating the efficacy of sharing paradigm. The dataset is preprocessed by subtracting the mean image over the training set. We evaluate without any data augmentation.

Our Adaptive Sharing (AS) is independent of the deep architectures used. We investigate two different convolutional neural network architectures, as shown in Table 2. The model A is the one used in [Hinton et al., 2012]. Inspired by [Simonyan and Zisserman, 2014], we develop a deeper network, model B, by replacing one 5 × 5 convolutional (conv) layer with a stack of two 3 × 3 conv layers. We also incorporate spatial pyramid pooling (spp) [He et al., 2014] into the model B, where the pyramid configuration is 4×4, 2×2 and 1×1. All weights layers (except for the last Fully-Connected (FC) layer) are followed by the Rectified Linear Unit (ReLU). Dropout is applied to all the pooling layers and the first two FC layers, with the dropout ratios 0.25 and 0.5, respectively. We implement the Adaptive Sharing as a standalone layer, which can be integrated by replacing the last FC layer. We denote the models as A-AS and B-AS, respectively. Consequently, there are four models, A and B, as well as A-AS and B-AS for comparison.

Our implementation is based on the publicly available code of Caffe [Jia et al., 2014]. We train the networks by applying stochastic gradient descent with a mini-batch size of 128 and a fixed momentum of 0.9. The training is regularized by weight decay (the ℓ2 penalty factor is set to 0.004). Particularly, the parameter matrix S in Adaptive Sharing is regularized with ℓ1 weight decay (the penalty factor is set to 0.0005). The learning rate is initialized to 0.001, is divided by 10 when the error plateaus.

The performance on the four models is shown in Table 3. With respect to model A, we achieve a coincide result as the one reported in [Hinton et al., 2012]. By integrating our Adaptive Sharing, the model performance can be effectively enhanced (with 1.45% improvement). On the other hand, [Srivastava and Salakhutdinov, 2013] achieves marginal improvement (test error is 36.85%, with 0.35% improvement) by using the same network configuration. The method is also formulated in transfer learning framework. However, information sharing is strictly governed by the determination of group structure, which may hurt the performance due to hard partitioning. In contrast, our method does not strictly require the relatedness to satisfy certain structure. It is flexible to couple related classes, and the information can be appropriately transferred in our model. The clear improvement over [Srivastava and Salakhutdinov, 2013] (1.10%) demonstrates the advantage of Adaptive Sharing. It is worth noting that Adaptive Sharing can enhance the model A and B consistently, and the superiority is more significant in the deeper network B. This implies that such sharing paradigm is beneficial for discovering useful features.

In order to comprehensively confirm the effectiveness of our method, we compare with the previous state-of-the-art results, as shown in Table 4. By virtue of a deeper architecture, the model B (in Table 3) outperforms the published best result. However, the superiority is marginal. Our Adaptive Sharing (B-AS) further improves the result. A test error of 31.20% is achieved, which surpasses dasNet [Stollenga et al., 2014] by 2.58%.

The balance between the shared part C and the specific part S can be regularized by the values of the parameters λs and λc. Due to many parameters in deep neural networks, we apply a simple strategy that specifies λs (and other parameters in the networks) with aforementioned empirical value, and study the effect of low rank parameters (i.e., the regularization factor λc and the approximation factor ε in (11)) on model performance. The test errors of B-AS with different values of λc and ε are summarized in Table 5. The results show that the model is insensitive to the factor ε, that similar performance can be obtained with a fixed λc. In contrast, the regularization factor λc plays a critical role in model performance. The best result is obtained when λc is set to 10-3 .

To investigate the power of our Adaptive Sharing in capturing the class relatedness, we make an analysis on the matrix C (corresponding to the shared features) of the model B-AS. We utilize CT C to represent the similarity matrix of all the classes in CIFAR-100, which is shown in Fig. 3. Darker color describes larger value in the similarity matrix (diagonal line denotes self-similarity). The matrix exhibits block-diagonal structure, indicating that related classes are encouraged to be coupled and share information. While hard partitioning is not applied in Adaptive Sharing, we adopt the Normalized Cut [Shi and Malik, 2000] to visualize the induced class groups of the similarity matrix, as shown in Table 6. For comparison, we also provide the class groups of the model B by adopting a similar operation on the weight matrix of the last FC layer, and the result is shown in Table 7. It is obvious that the relatedness captured in Adaptive Sharing is more reasonable. In fact, the cluster relationship among classes is introduced in our method although there is no explicit partitioning as [Srivastava and Salakhutdinov, 2013] did.

We perform additional experiments on the ImageNet 2012 classification dataset [Russakovsky et al., 2014], which is a challenging dataset with 1000 classes. We use the AlexNet architecture [Krizhevsky et al., 2012] (based on Caffe training protocol [Jia et al., 2014]) as baseline, which achieves 57.1% on top-1 accuracy and 80.2% on top-5 accuracy on the validation set, using the center crop. We integrate our Adaptive Sharing in the network by replacing the last FC layer. We apply the same setting of the parameters λc, λs and ε as in CIFAR-100, and set the other parameters the same as the baseline. Our model is trained on a single Tesla K40 GPU within two weeks. We obtain 57.7% on top-1 accuracy and 81.3% on top-5 accuracy on the validation set, where the improvements over the baseline are 0.6% and 1.1%, respectively. Due to the high baseline on top-5 accuracy, the improvement is more difficult than the one on top-1 accuracy.

Nevertheless, our model achieves more improvement on top5 accuracy by virtue of appropriately transferring knowledge among classes.

5 Conclusion

In this paper, we present a novel adaptive sharing method for image classification. The shared information is selectively extracted and exploited to improve the generalization performance while simultaneously identifying the class-specific properties. We further integrate such adaptive sharing with deep neural networks. The outstanding performance on multiple challenging datasets verifies the effectiveness of such adaptive transfer. As a future direction, we are interested in leveraging such sharing paradigm to model the relation among the filters in deep neural networks.

Acknowledgments

This work was supported in part by National Basic Research Program of China (973 Program): 2012CB316400, in part by National Natural Science Foundation of China: 61332016, 61025011 and 61272326, in part by Grant of University of Macau (MYRG202(Y1-L4)-FST11-WEH). Z. Lin is supported by 973 Program of China (grant no. 2015CB352502), NSF China (grant nos. 61272341 and 61231002), and Microsoft Research Asia Collaborative Research Program. We would like to thank NVIDIA for GPU donation.