DRLnet: Deep Difference Representation Learning Network and An Unsupervised Optimization Framework

1 Introduction

As remote sensing technology develops, there are more and more on-orbit satellites, which bring a large size of remote sensing data with high time-, spatial- and spectral resolutions [Chi et al., 2016; Marchetti et al., 2016]. It is demanding to process these increasing remote sensing data, and CDA is one of the most important applications in joint interpretation of remote sensing data, which aims not only to detect changes but also distinguish different types of changes. To better achieve this, it is necessary to learn more powerful and discriminative DRs from spatial-temporal images.

Recently, deep learning has achieved tremendous success in many vision and speech tasks such as image classification [Hinton and Salakhutdinov, 2006] and recognition [He et al., 2016; Krizhevsky et al., 2012], video understanding [Shao et al., 2016], image captioning [You et al., 2016] and natural language processing [Conneau et al., 2016] etc. The success of deep learning lies in the core that deep neural networks (DNN) has powerful ability in learning good representation from data in their raw form [Jiang et al., 2016]. For remote sensing images, it is also necessary to learn abstract representation for promoting joint interpretation of spatial-temporal images. However, unlike natural images, it is so lack of labeled data in the field of remote sensing that it is very hard to train a reliable model with certain scalability [Zhang et al., 2016a; 2016b]. Therefore, for practical purposes, it is demanding to develop unsupervised framework for joint interpretation of remote sensing data.

∗This work was supported by the National Natural Science Foundation of China (Grant no. 61422209, 61672409), and the National Program for Support of Top-notch Young Professionals of China.

†Corresponding author: gong@ieee.org

Clustering is one of the most popular unsupervised techniques often used to explore the hidden patterns and group similar structures [Shi and Malik, 2000; Hong et al., 2016; Dash et al., 2016]. However, it is very limited to perform clustering on the raw data. Deep learning outperforms others in learning good representation, but it is depended too much on supervised information during its training, while clustering may provide some reliable supervision information for training DNN in an unsupervised way [Wang et al., 2016]. Therefore, it is natural and promising to combine DNN and clustering for representation learning and special tasks, especially in the field of remote sensing where it is extremely lack of labeled data. Recently, clustering has been successfully integrated into the framework of deep learning, such as Deep Clustering Network (DCN) [Yang et al., 2016a], Deep Embedding Clustering (DEC) [Xie et al., 2016], and Variational Deep Embedding (VaDE) [Jiang et al., 2016] etc.

To effectively represent difference and measure difference degree, in this paper, a novel DRLnet and a recurrent learning framework are proposed. DRLnet firstly maps bi-temporal images into a suitable feature space to extract key information and suppress noise. Then, after automatic feature selection by the merging layer, the subsequent layers can learn more abstract DRs. By applying network forward passing and clustering, we can obtain the corresponding classification errors and clustering errors, which is used to tune network parameters in back propagation. As stated above, difference measurement, difference representation learning and unsupervised clustering are combined as a single model, i.e., DRLnet, which is driven to learn clustering-friendly and discriminative DRs for different types of changes. To strengthen its adaptability and stability, DRLnet can be extended into a recurrent framework by gradually decreasing the number of change types. In the recurrent framework, training samples would be updated and reused to learn more powerful model for CDA.

2 RelatedWork

2.1 Unsupervised Clustering

Clustering aims to group the data with similar structures into the same categories. Though many clustering methods have been proposed, such as K-means, spectral clustering [Nie et al., 2011], Gaussian mixture model (GMM) clustering [Biernacki et al., 2000] and agglomerative clustering [Gowda and Krishna, 1978] etc., unsupervised clustering still remains a fundamental challenge in the field of machine learning. In most of clustering methods, the similarity measures are limited in discovering the local patterns in data space, and thus it is hard to capture hidden and hierarchical dependencies in latent spaces [Dilokthanakul et al., 2016], while deep models can be used to encode rich latent structures hidden in raw data for improving the performance of clustering.

2.2 Deep Learning for Clustering

Recent works have demonstrated that optimizing representation learning and clustering jointly can greatly improve the performance of both. DCN [Yang et al., 2016a] combines learning good representation and clustering as a single model and optimize them jointly to find a 'K-means-friendly' space. DEC [Xie et al., 2016] was proposed to simultaneously learn representation and cluster assignments using DNN, and it works by iteratively optimizing a KL divergence based on clustering objective with a self-training target distribution. VaDE [Jiang et al., 2016] embeds the probabilistic clustering problems into a Variational Auto-Encoder

(VAE) framework, and models the data generative procedure by combining a Gaussian Mixture Model (GMM) and DNN. Additionally, Yang et al. proposed a recurrent framework for joint unsupervised learning of deep representation and image clusters, where successive operations in clustering are unfolded as steps in a recurrent process, stacked on the top of representations output by a Convolutional Neural Network (CNN) [Yang et al., 2016b].

3 DRLnet Formulation

As depicted in Fig. 1, we present a novel DRLnet and a recurrent learning framework, where DRLnet is established to learn clustering-friendly and discriminative DRs for distinguishing different types of changes. In DRLnet, unsupervised clustering and DNN are combined as a single model for learning clustering-friendly DRs without any supervision. And it starts to run with over-clustering pseudo labels, and achieves the desired number of change types by gradually decreasing the number of clusters with a recurrent way.

3.1 Loss Function of DRLnet

K-means is one of the most famous unsupervised technologies that explore the latent pattern hidden in data and group the data with similar pattern into the same categories. However, in most cases, the data distribution is not friendly to K-means clustering. Recently, DNN has show its powerful ability in representation learning, but it often needs supervised fine-tuning to greatly improve the performance on specific tasks. Inspired by this, we try to combine K-means and DNN as a single model called DRLnet for difference representation learning in CDA task. With over-clustering pseudo labels, DRLnet is driven to learn more abstract DRs friendly to desired number of clusters. The over-clustering pseudo labels may be not very reliable, but they are enough credible to avoid acquiring a trivial solution for DRLnet.

By combining DNN-based classification and K-means clustering as a single model, DRLnet can be formulated as the following loss function:

where n denotes the total number of training samples set X ∈ Rd×n , and α controls the balance between classification accuracy and clustering performance. x1i ∈ Rd×1 and x2i ∈ Rd×1 is one pair of bi-temporal patches extracted from image-pair I1 and I2, respectively, and let xi = [x1i , x2i ]. f(·) is a deep network classifier, d(·) is a difference representation extractor, and both of them share a part of parameters, and θ collects all network parameters such as weights and biases. M ∈ Rd×K is the collection of centroid with desired clusters, s0i ∈ RK0×1 is the initial pseudo labels acquired by unsupervised over-clustering, while si ∈ RK×1 is the desired assignment with respect toM, where K0 is the number of clusters in over-clustering, and K is the desired number of change types.

In Eq. (1), λi determines whether to execute K-means clustering on the difference representation of sample xi or not, and it can be computed by the following Eq. (2). This equation means that if the difference between the feature pair (h1i , h2i ) is larger than a threshold estimated by GGKI [Bazi et al., 2005], the corresponding sample xi is taken as changed sample, otherwise, it is treated as unchanged sample.

where h1i and h2i are the corresponding features of patches x1i and x2i learned by the feature extracting network, whileH1 = [h1i , } and H2 = {h2i |i = 1, 2, ..., n}, τ is the automatic threshold estimated by GGKI [Bazi et al., 2005].

DRLnet is driven to learn K-means-friendly DRs from bitemporal patches for better distinguishing different types of changes. By doing DNN-based classification and K-means clustering on the learned DRs, we can obtain classification and clustering errors, both of which are used to tune the network parameters of DRLnet by using back propagation. After training, all testing samples are fed into DRLnet and the corresponding DRs can be obtained, and then the desired CDA map can be generated by carrying out K-means on DRs.

3.2 An Recurrent Learning Framework

From the over-clustering pseudo labels to the desired number of change types, there may exists huge semantic gaps between them, which may be hard to bridge by a single saltation in number of clusters from over-clustering stage to the desired one. On the other hand, enough training samples is necessary for establishing a reliable deep network with certain capacity. Therefore, in this section, an recurrent framework is proposed to solve these two problems mentioned above. Instead of only one single saltation, the recurrent framework can achieve the desired number of change types by gradually decreasing the number of clusters from over-clustering stage to the desired one. In this procedure, the training samples are also updated and reused as the number of clusters decreases.

Suppose that the recurrent framework achieves the desired number of change types KT from over-clustering classes K0 through T timesteps, then the total loss function of the recurrent framework over all timesteps t from 1 to T can be formulated as:

For convenience, the f(x1i , x2i ; θ) and d(x1i , x2i ; θ) in Eq. (1) are written as f(xi) and d(xi), respectively. And the loss function at timestep t can be formulated as:

where all variables share the similar meanings as in Eq. (1), and sti ∈ RKt×1 represents the label of sample xi at timestep t, Mt ∈ Rd×Kt is the collection of centroids at timestep t, here Kt is the desired number of change types at timestep t. It's worth noting that the superscript t on each variable means the current timestep t, where t = {1, 2, ..., T}.

Algorithm 1 Alternating Optimization for DRLnet

Input: n pairs of bi-temporal patch pairs (x1i , x2i ), where i ∈ [1, n], known {st-1 i } and {λti }, initializedMt and {sti }. Output: Parameters θ of DRLnet at timestep t.

• 1: for iter = 1 : Iteration do
• 2: Update parameters θ by Eq. (7);
• 3: Update assignments {sti } by Eq. (8);
• 4: Update centroids {mtk } by Eq. (9);
• 5: end for
• 6: return DRLnet parameters θ at timestep t.

3.3 Optimization

The loss of the recurrent framework is formulated as Eq. (3), which needs to be minimized to achieve the desired solution. As mentioned in Section 3.2, the training samples would be updated at each timestep t. And the minimization of the loss at each timestep t is dependent on the final stage acquired at timestep (t-1). Therefore, the minimization of the total loss in Eq. (3) can be achieved by minimizing the loss function in Eq. (4) at each timestep t from 1 to T .

Algorithm 2 An Recurrent Learning Framework for CDA

Input: n pairs of bi-temporal patches (x1i , x2i ), where i ∈ [1, n], and DRLnet. C = {K0, K1, ..., Kt, ..., KT }, 1 ≤ t ≤ T , and Kt-1 > Kt, where Kt denotes the number of change types at timestep t, andKT is the targeted number of change types.

Output: Parameters θ of DRLnet and CDA map.

• 1: Obtain the initial pseudo labels {s0i } of each sample by over-clustering the stacked raw data into K0 classes: at the very beginning, all samples are taken as changed ones.
• 2: Pre-train DRLnet with over-clustering labels {s0i }.
• 3: for t = 1 : T do
• 4: Compute {λti } through Eq. (2) by applying GGKI on the learned DRs.
• 5: Initialize centers Mt and assignments {sti } at timestep t: Assign unchanged samples with the same labels and compute the centroid of their corresponding DRs, and group the DRs of changed samples into Kt classes to initialize (1 +Kt) centers and n assignments {sti }ni=1 ;
• 6: Fine-tune DRLnet to learn K-means-friendly DRs at timestep t via Algorithm 1 with known {st-1 i } and {λti }, initializedMt and {sti };
• 7: Update {sti }: Assign unchanged samples with the same labels, and group the DRs of changed samples into Kt classes with K-means clustering;
• 8: t ← t + 1;
• 9: end for
• 10: return DRLnet and CDAmap.

For each timestep t, we desire to minimize the loss function shown in Eq. (4), where three groups of parameters need to be solved, i.e., (θ, {sti }, Mt ), where {sti } is short for {sti }ni=1 . For convenience, we let

Having the DRs learned at timestep (t - 1), we assign unchanged samples with the same label and group the DRs of changed samples into Kt classes using K-means, initializing the centroids Mt and the corresponding assignments {sti }, where Kt is the desired number of change types at timestep t.

For fixed (Mt , {sti }), the network parameters θ can be updated by:

where η is the learning rate set in advance, and the partial derivatives ∂f(xi) ∂θ and ∂d(xi) ∂θ can be computed by using back propagation algorithm.

For fixed (θ, Mt ), the assignments {sti } can be updated by Eq. (8) shown as follows:

where stj,i is the j-th element of the assignment sti .

When fixing (θ, {sti }), the update of the centroid matrix Mt can be easily done as follows:

where Kt is the number of desired change types at timestep t, cik is the count of number of times this algorithm assign a sample to the cluster k before handling the next incoming sample xi, and the gradient step size 1/cik controls the learning rate of the current centroid mtk . The alternating optimization procedure of DRLnet at each timesept t is summarized in Algorithm 2.

4 Experiments

4.1 Datasets andMeasurements

Four bi-temporal remote sensing datasets are used in our experiments to assess the effectiveness of the proposed framework. All of them are cut from two large-format remote sensing images, which are acquired by GF-1 satellite at August 19th, 2013 and August 29th, 2015, respectively. Each of them is composed of four bands, i.e., red, green, blue and nearinfrared bands, and of the same spatial resolution 2m. Additionally, each pair of them has been radiometrically corrected and co-registered to make them as more comparable as possible. The overall detection accuracy (ACC) and Kappa coefficient are selected to quantitatively evaluate the proposed framework.

4.2 Experimental Setup

The proposed DRLnet is implemented based on deeplearntoolbox [Palm, 2012]. We apply learning rate as 0.005 with a momentum 0.5 to all layers of DRLnet, and the sparsity of activation on each layer is set as 0.05. During its training, DRLnet takes the batch size of 10 and we set α = 0.1. In our experiments, SLIC is applied to segment bi-temporal images independently, and then these two superpixel segmentation maps are merged to make them share the same but finer segmentation. That's, superpixel is taken as the basic analysis unit, which not only reduces the complexity of problem, but also integrates the spatial information. Fixed-size square windows centered at the center of each superpixel are used to represent them. Larger window size w would weaken their representation power on the corresponding superpixels, therefore, we set w = 3.

4.3 Performance of DRLnet on CDA

Fig. 2 shows the datasets used in our experiments and the corresponding CDA maps produced by the proposed DRLnet. As shown in Fig. 2(c), at the very beginning, all samples are taken as changed ones, and DRLnet starts to be trained with over-clustering pseudo labels. After training, the learned bitemporal feature representations can be compared to highlight changes, and then unchanged samples will be assigned with the same labels while changed samples will be grouped into less classes to achieve the targeted number of change types. As it goes, clustering-friendly and discriminative DRs can be learned by DRLnet, which is beneficial to clustering for identifying different types of changes. The intermediate CDA maps also demonstrate that, as the number of change type decreases, the real changes are detected correctly and different types of changes can be distinguished better.

Fig. 3 visualizes the distribution of the DRs learned by DRLnet on the Xi'an-1 dataset. At the very beginning, it is not easy to group DRs learned by DRLnet with overclustering, see Fig. 3(a). However, as the training goes with less number of change types, better DRs can be learned, and unchanged points and different types of changes can be easily grouped, as shown in Fig. 3(d). Clearly, these experimental results demonstrate the effectiveness of the proposed framework.

4.4 Comparison and Analysis

The proposed DRLnet is compared with a variety of baseline methods listed as follows:

• GGKI followed by K-means (Raw+KM): The classic change intensity map analysis method followed by applying KM to distinguish different types of changes.
• Stacked Autoencoder followed by GGKI and Kmeans (SAE+KM): This is a three-stage approach, SAE

is used for feature learning, and then these features are compared to highlight changes. Finally, the changes are grouped into different clusters using KM.

• DRLnet with fine-tuning only the top layers after the merging layer (DRLnet1 ).
• DRLnet with fine-tuning across all layers of it (DRLnet2 , the proposed approach).

Fig. 4 and Fig. 5 present the visual comparison between DRLnet and baseline methods on the Xi'an-1 and Xi'an-4 datasets. Compared with the baseline methods, DRLnet2 highlights most of changes happened in Xi'an-1 and Xi'an-4 datasets, meanwhile it successfully distinguish different types of changes with high precision. As specified by the red square and yellow ellipse in Fig. 6, DRLnet2 enhances the CI of changed points and suppresses that of unchanged ones, which leads to the CDA map with less noise. In Fig. 5, DRLnet2 perfectly highlights the changed region specified by the purple square while other baseline methods fail. Interestingly, as specified by the red ellipse in Fig. 5, DRLnet2 successfully suppresses unchanged region which is wrongly highlighted as changed one by other baseline methods.

Table 1 summarizes the quantitative comparison with baseline methods, and it clearly shows that DRLnet2 achieves the best performance with a significant advantage over the baseline methods both in ACC and Kappa. The success of DRLnet2 lies in fact that it combines learning DRs and clustering as a single model with a weighted parameter for detecting the changes and optimizing them in an end-to-end way. It starts to run with a reliable over-clustering, and achieves the targeted CDA goal in a recurrent fashion without any supervised information.

4.5 Clustering Performance on the Learned DRs

Fig. 6 shows the clustering performance on the DRs learned by different layers of DRLnet over the Xi'an-1 and Xi'an-4 datasets. From this figure, it is easy to find that deeper DRs has much better clustering performance. The reason lies in the fact that deeper layer has the ability to capture more abstract difference information from bi-temporal images. Compared with the ground truth maps, the clustering maps also show that deeper layer captures more accurate difference objects, while unchanged points may be grouped into different clusters because they are different ground objects in fact.

5 Conclusion

In this paper, we have presented a novel DRLnet and an recurrent learning framework for CDA in spatial-temporal remote sensing data. In DRLnet, difference measurement, difference representation learning and unsupervised clustering are combine as a single model, which can be driven to learn clustering-friendly and discriminative DRs for different types of changes without any supervision. And a recurrent learning framework is proposed to update limited training data and reuse them by gradually decreasing the number of change types from over-clustering stage to the desired one. Experimental studies demonstrate the effectiveness of the proposed DRLnet and the corresponding recurrent learning framework.