JM-Net and Cluster-SVM for Aerial Scene Classification

1 Introduction

Aerial scene classification aims to automatically give a semantic label to each image in order to know which category it belongs to, It is a fundamental problem in aerial image comprehending. Recently, aerial image becomes great important for earth observation [Hu et al., 2013; Cheng et al., 2015; Hu et al., 2015; Cheng et al., 2014], Because of the highly complex geometrical structures and spatial patterns, to efficiently understand the semantic information of them is very important, driven by many real-world applications in remote sensing community.

With the development of deep learning, Convolutional Neural Networks (CNNs) achieved state-of-art result on the task of aerial scene classification. However, most of these methods directly use the pre-trained CNN without considering the aerial images characteristics. The pre-trained CNN is only learned on natural images without fine-tuning. Furthermore, it is not feasible to fully train a new CNN model for aerial scene classification [Nogueira et al., 2017]. Because the traditional networks usually contain millions of parameters, therefore, to fully train a new CNN model will require a considerable amount of labeled data and demand high computational costs. Indeed, aerial datasets only have a few hundreds or thousands images. It is difficult to train a highpowered deep CNN with small datasets in practice [Hu et al., 2015]. This paper focus on the following question: How to fully train a new ConvNet with only few aerial data? It is a huge demand to design a deep network that are able to effectively train on aerial images.

To address this question, we design Justify Mentioned Network (JM-Net), a novel CNN model to extract the feature vector for aerial scene classification. Compared with traditional Networks [Jia et al., 2014; Szegedy et al., 2015], JM-Net has fewer parameters according to the following steps:

Firstly, to decrease the number of parameters, Justify Mentioned Net (JM-Net) is proposed, which ignores the fully convolution layer. In CNN, the most parameters exist in the fullyconnected (FC) layer. In the proposed JM-Net, the FC layers are abandon to decrease the number of parameters.

Secondly, Compress and Expand Convolution module (CEC) is proposed, which has compress layer and expand layer. The CEC module has 3 × 3 and 1 × 1 filters in the same expand convolution layer, besides it has fewer 1 × 1 filters in the same compress convolution filter, so the ECE module helps JM-Net decreases the number of parameters further.

Finally, Cluster-SVM is used to speed up the process of classification. Two images with similar feature vectors belong to the same category, we cluster the feature vectors to some piles, then use SVM to classify the centers of these piles.

In summary, the main contributions of our work include:

1. We have proposed JM-Net, a novel convolutional neural network with one-fortieth parameters of Alex-Net. JM-Net is fully trained on aerial images, achieves good performance on the challenging aerial datasets RSD and AID.
2. We have used Cluster-SVM to speed up the process of classification. Cluster-SVM is a strategy that classify the centers of feature vectors but not the feature vectors themselves.

2 RelatedWork

Currently, with the development of deep learning, many methods based on it achieve impressive results on many computer vision tasks such as image classification, object recognition, image retrieval, etc. Deep learning methods also achieve state-of-the-art performance on aerial scene classification [Hu et al., 2015], Deep learning methods on aerial scene classification can be divided into two kinds, pretrained-CNN method and retrain-CNN method.

The pretrained-CNN method is directly using pre-trained deep neural network architectures on the natural images [Russakovsky et al., 2014], the extracted features showed impressive performance on aerial scene classification [Penatti et al., 2015]. The two freely available pre-trained deep Convolutional Neural Network (CNN) architectures are OverFeat [Sermanet et al., 2013] and CaffeNet [Jia et al., 2014]. In [Castelluccio et al., 2015], another promising architecture, i.e. GoogLeNet [Szegedy et al., 2015], was considered and evaluated and this architecture also showed astounding performance. In [Luus et al., 2015], it demonstrated that a multiscale input strategy for multi-view deep learning can improve the performance of aerial scene classification. Others use the deep-CNN as feature extractor and combine it with feature coding techniques. For instance, Hu et. al [Jgou et al., 2012] extracted multi-scale dense CNN activations from last convolutional layer as features descriptors and further coded them using feature encoding methods like BoVW [Sivic and Zisserman, 2003], Vector of Aggregated Descriptors(VLAD) [Jgou et al., 2012] and Improved Fisher Kernel(IFK) [Perronnin et al., 2010] to generate the final image representation. For all the deep-CNN architectures used above, the features were obtained from the networks pre-trained on natural image datasets and were directly used for classification aerial images.

The retrain-CNN method is training a new deep network with aerial images. However, as reported in [Nogueira et al., 2017], using the exsisting aerial scene datasets to fully train the networks such as CaffeNet [Jia et al., 2014] or GoogleNet [Szegedy et al., 2015] showed a drop in accuracies compared with using the networks as global feature extractors. This because that the large scale networks usually contain a large number of parameters to be trained, therefore, to train them using the aerial datasets with fewer images will easily result in over fitting. Thus, to better fit the dataset, smaller networks for classification were trained [Zhang et al., 2015; Zou et al., 2015]. In [Zhang et al., 2015], a Gradient Boosting Random Convolutional Network(GBRCN) was proposed for classifying aerial images with only two convolutional layers. In [Zou et al., 2015], a deep belief network(DBN) [Hinton et al., 2006] was trained on aerial images, and the feature selection problem was formulated as a feature reconstruction problem in the DBN scheme. By minimizing the reconstruction error over the whole feature set, the features with smaller reconstruction errors can hold more feature intrinsics for image representation. However, the generalization ability of a shallow network is often lower than that of a deep one. It is highly demanded to design a deep network but with fewer parameters.

3 ProposedMethod

This section details the proposed method with JM-Net and Cluster-SVM. The flowchart of the proposed is showed in Figure 1.

Firstly, JM-Net is used to extract aerial images' feature vector. JM-Net has the module CEC (including compress layer and expand layer) and ignores the fully convolution layer, so it has fewer parameters and can be trained well on aerial image datasets.

Secondly, cluster the feature vectors' to fewer clusters, and assume each image in the same cluster has the same category label with their cluster center.

Finally, support vector machine is used to classify the cluster centers, then the cluster center' category is used to represent each image's category label in this cluster, and the task of aerial scene classification is finished.

3.1 JM-Net

The overarching of this part is to identify a model that has very few parameters while preserving accuracy on the task of aerial image classification. To address this problem, a sensible approach is to take an existing CNN model (in this paper, we are based on Alex-Net) and compress it in an lossy fashion, and we use the following 4 strategies to modify the Alex-Net and named it Justify Mentioned Net (JM-Net):

Strategy 1: Inspired by the [Szegedy et al., 2015], which has different size of filters in the same convolution layer, we replace part of the filters from 3 × 3 to 1 × 1 in Alex-Net. Given a bugget of a certain number convolutional filters, we will choose to make the majority of these filters 1×1, because that a 1 × 1 filter has 9X fewer parameters than a 3 × 3 one.

Strategy 2: Remove the fully convolutional layers, and use average-pooling layer to extract the feature vector from the last convolutional layer. As we all know, the deep convolution layers of CNNs are already have the high-level information and the fully convolutional layers have most parameters of the CNNs. So, to improve the efficiency, we remove the fully convolutional layers.

Strategy 3: Decrease the number of input channels to 3×3 filters. Consider a convolution layer that is comprised entirely of 3 × 3 filters, The total quantity of parameters in this layer is (number of input channels)*(number of filters)*(3×3). So, to maintain a small total number of parameters in a CNN, it is important not only to decrease the number of 3 × 3 filters, but also to decrease the number of input channels to the 3×3 filters.

Strategy 4: Postpone the process of downsampling in the network. Downsampling early will led to the information loss early, and decrease the demonstrate capacity of deep layers, and it is proved that delayed downsampling result in higher classification accuracy.

Strategy 1, 2 and 3 are about decreasing the number of parameters in a CNN while attempting to maintain the accuracy, and Strategy 4 is about increasing the accuracy as much as possible.

The CEC-module

Inspired by [Denton et al., 2014], we simply use 1×1 reduction layers followed by the combination of 1 × 1 and 3 × 3

convolution filter layers. The architecture of CEC module is showed in Figure 2.

For the sake of simplicity, we define the Compress and Expand Convolution (CEC) module as follows. A CEC module is comprised of: a compress layer which only has c1×1 (compress layer's 1 × 1 convolution filter), feeding into an expand layer that has both e3×3 (expand layer's 3×3 convolution filter') and e1×1 (expand layer's 1×1 convolution filter) filters. When we use the SF module, we set the number of c1×1 less than the sum of e1×1 and e3×3, so the compress layer helps to limit the number of the input channels to 3 × 3 filter.

JM-Net Architecture

The JM-Net begins with a convolution layer(conv1), followed by 8 CEC modules(CEC2-9) and a convolution layer, ending with a final Softmax layer. We gradually increase the number of filters per CEC module from the beginning to the end of the JM-Net. the net performs max-pooling with a stride of 2 after conv1, CEC4, CEC8, and conv10; We present the full JM-Net architecture in Table 1.

3.2 Cluster-SVM

Inspired by the super-pixel, we see every image's feature vector as an pixel's RGB feature, through cluster method, we think that the images with similar features belong to the same category. Among many cluster method, K-means is used widely. However, K-means method gives each element of feature vector the same importance, but there are some problems. For example, if we need to classify a cow and a sheep, the size is more important than the length of fur. In other words, each element of feature has different importance for the classification. In this paper, to improve the classify accuracy, we replace the Euclidean distance with Sim loss, which include both distance and angle information about the feature vector, and the proposed method overcomes the shortcomings described above. The specific correlation function is as follows:

The first part of the function is based on consine angle and the second part is based on Euclidean distance, λ is a balanced parameter between 0 and 1, and we make it 0.7 in our experiment, and we replace the Euclidean distance with Sim loss.

Where

In our work, we use JM-Net to extract features of images, use traditional SVM as classifier when training, and at the testing stage, we use the proposed cluster strategy to speed up the classification process.

4 Experiments

In our experiments, we use two challenging datasets, RSD and AID. RSD consists of 1005 pictures within 19 classes, and AID consists of 10000 pictures within 30 classes (some samples of AID dataset are showed in Figure 3). As we all know, aerial image has the characteristics of scale and direction changeable, so we can not use the images from the dataset only to train a robust Convolutional Neural Network effectively. To address this problem, the datasets are expanded according to the following steps:

1. Randomly extract nine different square regions with different size and direction on each original images, resize all the extracted images to the size of the original

images, so we obtain a new dataset which is ten times larger than the original one.

1. Turn all the images obtained in step 1 up and down.
2. Turn all the images obtained in step 2 left and right.

Through above steps, a new dataset with forty times images compare to the original one can be obtained. The schematic is showed as Figure 5.

4.1 Choose the Parameters of JM-Net

According to the proposed method, there are two important parameters needing to choose: The ratio between 3 × 3 and 1 × 1 expand convolution filters.

If fewer parameters can be used to demonstrate the feature of an image accurate enough, we can compress the CNN model to some extent. We have used KL method to decrease the dimension of standard VGG-16's and Alex-Net' 4096-D feature vector. In order to simplify the experiment, we decrease the number of feature element by half during each step. Then these features are used to train the classifier. Finally, we calculate classification accuracy as Table 2.

From Table II, it is obvious that 2048 dimension vectors are enough to represent the aerial scene images. when design the architecture of JM-Net, we make the output of JM-Net's last layer to be 2048 dimensions.

As we all know, a 1 × 1 filter has 9X fewer parameters than a 3 × 3 one. In other words, if we want to reduce the parameters of the Convolutional Neural Network, we should use more 1×1 filters and fewer 3×3 filters. When the number of 3 × 3 filters are reduced, the model perhaps cannot extract typical features of the aerial images and achieve competitive performance on the task. In this section, we will choose the best ratio between c1×1 and c3×3 , to use least parameters and remain the classification accuracy. From TABLE 3, It can be seen, when the ratio of 3 × 3 and 1 × 1 convolution filters are both 0.5, the classification accuracy reaches to the top.

4.2 Choose the Parameters of Cluster-SVM

For the Cluster-SVM, there are also two important parameters influencing the experiment result: The number of cluster centers and the balance parameter λ of Sim loss.

Firstly, the number of cluster centers is a very important factor to the final result. Specific to the task of aerial scene classification, if the number of clustering centers is too big, we can not speed up the testing process efficiently. Oppositely, if it is too small, perhaps we will lose the powerful classification capacity of the support vector machine and led to the debasement of classification accuracy. in this section, specific to different datasets, we choose different number of clustering centers to reach the equilibrium. From the TABLE 4 and TABLE 5, we can see when the cluster number is about 1/8 of the amount of the overall images in the datasets, that the classification accuracy reaches the peak.

Additionally, the parameter λ of Sim loss reflects the attention of the cluster strategy. From the TABLE 6 we can see that, when the λ is about 0.7, the experiment result reaches to the peak.

4.3 The Performance of ProposedMethod

Because JM-Net is modified from the Alex-Net, so we need to compare the performance between these two nets and demonstrate the advantages of JM-Net.

Compare Table 2 and Table 7, Figure ?? and Figure 6. we can see JM-Net gets a better result than Alex-Net, this is because that JM-Net is trained on aerial images and can describe the aerial images better. Besides, From the TABLE VI, we can see from the cluster strategy, the classification accuracy improves generally, especially when using fewer feature information. The reasons are as follows:

1. Two images with similar feature that belongs to an category, some centers of the feature vector are got through cluster strategy, and these centers are much easier to classify than the raw feature vectors;
2. The cluster center's category is used to represent each image's category label in this cluster. in other words, the cluster strategy decreases the number of testing samples in some terms, and so avoid the likelihood of confusion.

In summary, the proposed method method achieves higher accuracy while using smaller storage space and shorter time.

5 Conclusion

In this paper, we analyzed the difference between natural and aerial images, the dataset of aerial images is much smaller than traditional images, we can not train a standard Alex-Net effectively with aerial images only. For these points, we propose to use a compressed Convolutional Neural Network, JMNet, which occupies 1/40 space of the Alex-Net and can be stored in portable storage device conveniently, to extract the features of remote sensing images. To speed up the process of the task further, we propose a strategies of clustering to suitable for the real-time applications. Experiment results on datasets RSD and AID demonstrate that the proposed method achieves higher accuracy while used shorter time and smaller storage space, and the proposed JM-Net can be generalized to other aerial image tasks conveniently.