Learning deep structured network for weakly supervised change detection

1 Introduction

Identifying changes of interest in a given set of images is a fundamental task in computer vision with numerous applications in fault detection, disaster management, crop monitoring, visual surveillance, and scene analysis in general. When there are only two images available, existing approaches mostly resort to strong supervision, thus require large amounts of training data with accurate pixel-level annotations to perform pixel-level analysis. To comprehend the significant amount of effort needed for such a formidable task, we consider the example of CDnet-2014 [Wang et al., 2014], which is the largest dataset for video based change detection. This dataset required manual annotations for ∼8 billion pixel locations. Although sophisticated methods have been investigated to reduce the human effort, e.g., by expert feedback in case of ambiguity [Jain and Grauman, 2013;

∗Corresponding author: salman.khan@anu.edu.au

†This work was done when the author was at Data61/ANU.

Gueguen and Hamid, 2015], semi-automatic propagation of annotations [Badrinarayanan et al., 2013], and point-wise supervision [Russakovsky et al., 2015], acquisition of accurate and dense pixel-wise labels still remains a daunting task [Lin et al., 2014; Song et al., 2015].

Here, we address the problem of change detection within a pair of images and present a solution that uses only imagelevel labels to detect and localize changed regions (Fig. 1). Our method drastically reduces the effort required to collect annotations and provides an alternative to video change detection that requires a large number of consecutive frames to model the background scene. In many real-world applications, a continuous stream of images may not be always available due to a number of reasons such as challenging acquisition conditions, limited data storage, latency in processing, and long intervals before changes happen. For example, the analysis of aerial images for change detection, in particular for damage detection, often is formulated for a pair of images acquired at different times. Other examples where only a pair images might be available include structural defect identification, face rejuvenation tracking, and updating city street-view models.

Our algorithm jointly predicts the image-level change label and a segmentation map indicating the location of changes for a given pair of images. The central component of our method is a novel two-stream deep network model with structured outputs (Sec. 2). This model operates on a pair of images and does not need the images to be registered precisely. It can be trained with only weak image-level labels (Sec. 4.2). The network has a Directed Acyclic Graph (DAG) architecture where the initial layers are shared, while the latter part splits into two branches that make separate (but coupled) predictions for change detection and localization. In this manner, our deep network is different from the popular single-stream convolutional neural networks (CNN) for object classification [Khan et al., 2015], detection [Girshick et al., 2014; Khan et al., 2016] and semantic labeling tasks [Papandreou et al., 2015; Long et al., 2015; Pinheiro and Collobert, 2015].

In order to jointly predict the image-level and pixel-level labels, we introduce a constrained mean-field inference algorithm (Sec. 2.3), that employs a factorizable approximate posterior distribution with global linear constraints. Using a global constraint on the foreground (changed pixels) probability mass function, we suppress the bias towards the background (no-change labeled pixels) and encourage the assignment of change labels to nonidentical regions. Such global constraints enable us to derive an efficient mean-field inference procedure, while eliminating the need of approximate biases [Papandreou et al., 2015] and object based priors [Pinheiro and Collobert, 2015; Russakovsky et al., 2015]. Furthermore, based on the novel inference algorithm, we apply a variational Expectation-Maximization (EM) learning algorithm that maximizes the lower bound of the log-likelihood of image-level labels. We extensively evaluate our approach on three publicly available datasets (CDnet-2014, PCD-2015 and AICD-2012) and a custom built satellite image dataset (GASI-2015) (Sec. 4.2). Our experimental results demonstrate that the proposed approach outperforms the state-ofthe-art by a large margin (Sec. 4.3). The key contributions of our work include:

• To the best of our knowledge, this is the first work to address the weakly supervised change detection problem.
• Our proposed CNN model jointly detects and localizes changes in image pairs.
• We present a modified mean-field algorithm with additional constraints to efficiently localize changes.
• We introduce a new satellite image dataset (GASI-2015) for change detection. Furthermore, we perform a rigorous evaluation on three other relevant datasets.

2 Two-streamCNNs for Change Localization

We address the problem of joint change detection and localization with only image-level weak supervision. To this end, we propose a two-stream deep convolutional neural network model with structured outputs, which can be learned with weakly labeled image pairs. We describe our model next.

2.1 Model Overview

Given a pair of input data, which can be images or (short) video clips, our goal is to predict the categories of change events in the data pair and localize the change more precisely at the pixel level. For simplicity, we focus on the image pair scenario in the following and video clips can be processed in a similar manner.

Specifically, let each input consists of a pair of images, x = {I1 , I2 }. We associate an image-level output label vector y = {y1, y2} to indicate the occurred change events i.e., change, no-change and y1, y2 ∈ {0, 1}. It is important to note here that the no-change category (i.e., y = 0) refers to the static-background, irrelevant changes and the dynamic background change patterns while those change categories refer to changes of interest. In order to localize the change events at the pixel level, we introduce a set of binary variables h to denote the labels of individual pixel locations for each image pair x. Assume the image has m pixel locations, h = {h1, · · · , hm} ∈ {0, 1}m.

We formulate the change detection and localization problem as the joint prediction of its image-level and pixel-level change variables. To achieve this, we consider a deep structured model that defines a joint probabilistic model on y and h as P(y, h|x; θ) = 1 Z(x) exp(-E(y, h|x; θ)), where the Gibbs energy is defined as:

where Φl(y|x; θl) is the unary term for image-level label y, modeled by a CNN with parameter θl, and Φu(h|x; θu) is the unary term for pixel-level labels, modeled by a Fully Convolutional Network (FCN) with parameter θu. The pairwise energy Φp(h, y) consists of two terms, ψp and ψu, which enforces the spatial smoothness of pixel-level labels and captures the coupling between image- and pixel-level labels, respectively. The joint prediction can be formulated as inferring the MAP estimation of the model distribution,

A graphical illustration of the model is shown in Fig. 2.

2.2 Deep Network Architecture

We build the deep structured model by first introducing a twostream deep CNN for the unary terms as shown in Fig. 3. The underlying architecture of the network is similar to the VGGnet (configuration-D, the winner of the classification and localization challenge, ILSVRC'14) [Simonyan and Zisserman, 2014] but with several major differences. Most importantly, the network operates on multichannel inputs (6 channels for paired color images) and divides into two branches after the fourth pooling layer (P4). From our initial experiments (consistent with [Zagoruyko and Komodakis, 2015]), a multi-channel network performs better than a traditionally used Siamese network for paired images. The two branches compute the probability of the image-level and pixel-level labels, and therefore will be called as the classification and the segmentation branch, respectively. The segmentation branch in our architecture is similar to FCN-VGG16-16s network

[Long et al., 2015] which demonstrated state-of-the-art performance on the Pascal VOC segmentation dataset. The initial shared layers in our architecture combine the initial (essentially similar) portions of VGG and FCN networks, which results in a significant decrease in trainable parameters without any drop in performance. We now describe the details of the two branches of the network architecture.

Image-level change unary energy: The classification branch predicts the image-level label probability and has more layers to collapse the filter responses from the initial layers. Specifically, the classification branch output of the CNN architecture models Φ(y|x; θl), predicting the imagelevel change energy as:

where Fl-cnn is the deep network feature before the final softmax operator, Ws are the weight parameters shared with the segmentation branch, and Wy are the weight parameters for the classification branch only.

Pixel-level change unary energy: The segmentation branch generates a down-sampled coarse segmentation map (of size osz × osz) for each change category. After shared layers, the branch has three fully connected layers, which are implemented as convolution layers as in the FCN [Long et al., 2015]. Formally, the segmentation branch of the CNN model generates the pixel-level change label energy as follows,

where Fu-cnn denotes the segmentation branch scores of the CNN architecture before the soft-max operator and Wf are the weights for the fully connected layers.

We now describe the pairwise energy Φ(h, y|x, θp) that encodes the compatibility relations between the image-level and the pixel-level variables as well as the spatial smoothness. Specifically, on the top of the fully connected layers, we add a densely connected Conditional Random Field (CRF) to impose the spatial smoothness of the pixel labeling. Unlike the previous models [Papandreou et al., 2015], our dense CRF depends on the output label of the classification branch, and thus couples the image-level and pixel-level prediction.

Formally, we define the compatibility relations between the output variables y and h by the following energy functions,

where ψu enforces all hidden variables to be zero if the category label predicts no-change and encourages hj to take a change label otherwise:

where γ is a weight parameter and δ(xj) is the color difference between two images at pixel j. The fully-connected pairwise term ψp defines the smoothing term between the latent variables h given input features fj, fk. These energies have a functional form of the weighted Potts model in which the weight is defined using Gaussian kernels of [Kr¨ahenb¨uhl and Koltun, 2011]:

where αap, αsm are the kernel weights, µ(hj , hk) is the Potts compatibility while kap(fj, fk), ksm(fj, fk) are the appearance and smoothness kernels [Kr¨ahenb¨uhl and Koltun, 2011].

2.3 Model Inference for Change Localization

Given the two-stream CNN+CRF model, we predict the image and pixel-level change labels by inferring the MAP estimation of the joint probability model in Eq. (1). In order to compute the most likely configuration efficiently, we adopt a sequential prediction approach that first infers the image-level change label followed by the pixel-level change mask inference. Specifically, we compute the change label prediction approximately as follows,

This prioritized inference procedure allows us to compute the (more reliable) image-level label first and to run an efficient mean-field inference for the pixel-level labels only once 1 .

We now derive a constrained mean-field inference algorithm for inferring the pixel-level change labeling h. We note that the efficient mean-field algorithm [Kr¨ahenb¨uhl and Koltun, 2011] usually leads to an over-smoothing of the pixellevel labeling and assigns most of the pixels to the 'nochange' class. In this work, we incorporate an additional global constraint on the proportion of 'change' label values in the image. Unlike previous methods (e.g., [Papandreou et al., 2015; Pinheiro and Collobert, 2015; Russakovsky et al., 2015]), we enforce such constraints on the approximate probability family which allows us to derive an efficient modified mean-field procedure.

Formally, we assume the foreground label proportion to be τ , which is fixed during training by cross-validation. For each test image pair, we find K closely matching pairs from the training set using a KNN search and average their foreground label proportion to estimate τ (details in Sec. 4.3). To enforce the proportion constraint, we introduce the following factorized approximate probability family with a global constraint:

where, qj = [qj(0), qj(1)]T and the constraint implies that the overall foreground probability mass ∑j qj(1) is τ. Following [Kr¨ahenb¨uhl and Koltun, 2013], we minimize the approximate KL-divergence,

where uj is the unary term vector (including P(hj|x) and ψu(hj , y∗ , xj )) and Ψjk is the compatibility matrix computed from ψp, and C = log Z is the log partition function. We use the CCCP algorithm [Yuille and Rangarajan, 2003] to minimize DQ||P iteratively.

1 In general, we note that we can compute the MAP estimation jointly by enumerating y's values and running mean-field inference multiple times, which is less efficient.

3 EMLearning withWeak Supervision

We now consider a weakly supervised learning approach to estimate the parameters of the two-stream CNN+CRF model (Sec. 2). In particular, as the labeling of the pixel-level change pattern is tedious and impractical, we assume only imagelevel change annotations are available, which can be obtained with much less effort. Let us denote the dataset D comprising of N labeled image pairs: D = {xn , yn }1×N .

The learning objective is to maximize the log conditional likelihood and we consider a variational mean-field energy lower bound as follows,

where, EQ[·] and H(·) denote the expected value and the entropy function respectively, and Q(hn |xn , yn ) is an approximate posterior probability factorizing over {hnj } as defined in Eq. (6). In other words, the posterior probability can be expressed as the product of independent marginals: Q(hn |xn , yn ) = ∏j qn j (hnj ). We then derive a variational expectation-maximization (EM) algorithm for learning our two-stream CNN+CRF in the following, which alternately maximizes the objective function above.

3.1 Mean-field E Step

We update the approximate Q function by maximizing the objective w.r.t the Q function given the model parameter θ from the previous iteration. Note that given the model structure, this leads to a mean-field updating equation to compute q(hnj ). The updating equation requires message passing between all the hj and hk, which is computationally expensive. Efficient message passing is achieved using the high dimensional Gaussian filtering by considering the permutohedral lattice structure [Adams et al., 2010].

Given the approximate posterior marginals, we can compute the (approximate) most likely configuration of the latent variables hn ,

The marginal mode hn∗ will be used in the M step for the CNN+CRF learning.

3.2 MStep for CNN+CRF Training

Once we have the posterior marginal distribution q(hnj ) and its mode, we update the model parameters θ with the posterior mode configuration {hn∗ } and ground-truth {yn }. Specifically, we treat them as the ground-truth for the pixel and image-level labels, and learn the two-stream deep CNN+CRF in a stage-wise manner. Our stage-wise learning first estimates the parameters in the unary terms, i.e., the two deep CNNs, and then validates the parameters in the pairwise term. This strategy is similar to the piece-wise learning in the CRF literature.

We first use back-propagation to train the two branches of the deep CNN separately with the corresponding training data. More precisely, the averaged gradient from two streams is back-propagated to update the shared parameters (Ws), while the individual gradients are computed using yn and hn∗ as ground-truths to update Wy and Wf for the classification and segmentation branches, respectively. Concretely, the model parameters are updated to maximize the data likelihood as follows,

After the two-stream deep network component is trained, we estimate the parameters θp in Eq. (4) by cross-validation.

The overall EM procedure starts with an M step with an initial value of hn . We assume the initial hidden variable states (hn0 ) to be consistent with the image-level labels: hn0 = yn .

The model parameters are fine-tuned by training the twostreamCNN+CRF with those initial labels. This is important because the CNN is pre-trained for object recognition on ImageNet and therefore the estimation of change regions in the initial E-step does not generate reasonable ground-truths.

4 Experiments

4.1 CNN Implementation

The network weights are initialized from a pre-trained VGG network (on ImageNet). The network splits into two portions after the fourth pooling layer. As we need a coarse segmentation map (32 × 32) at the output of the segmentation branch, enlarged paired images of size 512×512 are fed to the CNN. Moreover, the convolution filter size in FC1 (segmentation branch) is kept to 1 × 1 (in contrast to a 7 × 7 filter size in FC1′ ) to avoid the additional decrease in resolution of the 32 × 32 output map.

The unary energies of our CRF model are defined using the CNN activations, while the Gaussian edge potentials proposed by [Kr¨ahenb¨uhl and Koltun, 2011] are used as pairwise terms. Note that changes of interest can occur in any of the two paired images, and therefore it is not desirable to remain restricted to the detection of changes in only one of the images (w.r.t the other image). For this purpose, the groundtruth with which we compare our final segmentation results include the changes in both images (see Fig. 4). During the mean-field inference step, we find the segmentation map of both images using their respective edge potentials. Subsequently, the two output maps are combined to get the final estimate of hidden variables. The resulting segmentation map is used as ground-truth during the CNN training (M step).

4.2 Datasets and Protocols

We evaluate our method on the following four datasets. All of them include pixel level change ground truth, from which we derive the image-level annotations for weakly supervised learning. The pixel-level labels are not used in the training of our deep network.

CDnet 2014 Dataset: The original video database consists of 53 videos with frame-by-frame ground-truths available for ∼ 90, 000 frames in specified regions-of-interests (ROIs). Various types of changes (e.g., shadow, object motion and motion blur) under different conditions (e.g., challenging weather, air turbulence and dynamic background) are included in this database [Wang et al., 2014]. It is also important to note that the paired images are not registered and therefore background can change across paired images [Wang et al., 2014]. A total of 91, 595 distinct image pairs are generated at random from the video sequences. In each pair, both images belong to the same video but they are captured at different time instances.

AICD 2012 Dataset: Aerial Image Change Detection (AICD) dataset [Bourdis et al., 2011] consists of 1000 pairs of large sized images (800 × 600). It is a synthetic dataset in which the images are generated using a realistic rendering engine of a computer game (Battle Station 2). A total of

100 scenes are included in this dataset containing several realworld objects including buildings, trees and vehicles. The scenes are generated under varying conditions with significant changes in viewpoint, shadows and types of changes. Because the change regions are very small in satellite/aerial images, we work on the patch level and extract 48 patches of size 122 × 122 from each image with minimal overlap. This provides a total of 24, 000 paired images, facilitating the training of a model with a large number of parameters.

GASI 2015 Dataset: Geoscience Australia Satellite Image (GASI) dataset is a custom built dataset based on the changes occurred during 1999-2015 in a ∼ 100×100 km2 area in the east of city of Melbourne in Victoria, Australia [Khan et al., 2017]. For each region of interest, we have a time lapse sequence (between 1999-2015) of surface reflectance data and the corresponding pixel quality maps. Due to the severe artifacts caused due to clouds and band saturation, the modelling of the temporal trends is very challenging. In contrast, the acquisition of paired images captured at different times is much easier.

The annotations for two types of changes are provided in the GASI dataset namely: fire and harvests. We generate pairs of image patches for 67 distinct regions of interest which were identified by experts. In total, ∼ 300 pairs are generated for each region of interest which makes a total of ∼ 20, 000 pairs. Since the raw data contains artifacts, we improved it's quality by filling data across different time instances.

There exists a large disparity among the sizes of change regions in the GASI dataset. For very large sized regions, we cropped the region bounded by a tightly fit bounding box. For small regions (mostly changes due to forest harvesting) with area < 5% of the total image area, we crop a bounding box with dimensions equal to three times that of a tightly bounded box. Since, there are large variations between the size of changes in the identified change regions, we converted all the regions to a uniform size of 224 × 224 to get a consistent segmentation map.

PCD 2015 Dataset: Panoramic change detection dataset [Sakurada and Okatani, 2015] consists of 200 pairs of panoramic images of street scenes and tsunami-hit areas. The image size is 224 × 1024, from which we extract 122 × 122 patches with a minimal overlap for training and testing. This gives us a total of 3, 600 pairs (18/panoramic image). It is important to mention that the two images are not perfectly registered. As a result, there are temporal differences in camera viewpoints, illumination and acquisition conditions.

4.3 Results

The change detection results of our approach on the four CD datasets are shown in Table 1. As a baseline, we only consider the classification branch of the CNN network initialized with the pre-trained VGG-net (configuration D, see 2-3 columns of Table 1). Paired images are fed to this network architecture and 4096 dimensional feature vectors are extracted from the FC2′ layer. A linear SVM classifier is then trained for classification using the lib-linear package [Fan et al., 2008]. On both datasets, the average precision (AP) and the overall accuracy of our approach was significantly higher than that of the baseline procedure (specifically 6.8% and 5.3% boost in AP for the CDnet and GASI datasets, respectively). As a stronger baseline, we also report performance of the network when only the fine-tuned classification branch was used (columns 4-5, Table 1). We note that our full model outperformed the results from the fine-tuned classification branch.

We report the segmentation performance of our approach in Table 2 in terms of the mean intersection over union (mIOU) score. To compare our change localization results, we report four baseline procedures. Specifically, we compare against random segmentation masks (2nd column-RS), thresholding applied to a difference map obtained from the pair of images (3rd column-DT), thresholding applied to the output from a pre-trained network (weights initialized for segmentation branch using VGG-net, 4th column-PN), thresholding applied to the output from the fine-tuned network (5th column-Th.) and the graph-cuts inference [Boykov et al., 2001] using CNN outputs as unaries and contrast based pairwise potentials with a Potts model (6th column-GC). We note that random segmentation provides a lower baseline, while our results after training with ground-truths (shown in last column, Table 2) sets an upper bar on the performance. Another important trend is that the thresholding approach and graph-cuts performances do not differ by a large margin. However, our weakly supervised approach was able to achieve significantly higher mIOU scores due to the additional potentials and constraints (Sec. 2).

We also report segmentation results on two additional baselines which use cardinality based pattern potentials (Table 3). These baselines include the higher order potential (HOP) based dense and grid CRF models of Vineet et al. [Vineet et al., 2014] and Kohli et al. [Kohli et al., 2007] respectively. For both these baselines, we define HOPs on segments generated using mean-shift segmentation. Due to absence of pixel level supervision, we use the parameters from [Kohli et al., 2007]. We note that the dense CRF model with P n HOP [Vineet et al., 2014] performs better than the grid CRF model [Kohli et al., 2007], however our deep structured prediction model outperforms both these strong baselines by a fair margin of ∼ 4 - 8% in terms of mIOU score.

The qualitative results for change localization on the CDnet-2014, GASI-2015 and PCD-2015 datasets are shown in Fig. 4. The proposed approach performed well in localizing small as well as large sized changes (e.g., 1st col, Fig. 4). Moreover, it showed good results for images acquired in varying conditions (e.g., night, snow, rainfall, dynamic background) and with different capturing devices (e.g., thermal camera, PTZ). For the CDnet-2014 dataset, it is interesting to note that our method localized several changes in the regions outside the ROIs (shown in blue color in the ground-truth). Similarly, the qualitative results indicate the good performance of our method for satellite image based change detection.

We performed an ablative study on the CDnet-2014 dataset for the change segmentation task (Table 4). The experimental results show that the localisation performance decreases without the feedback from the classification branch (whose predictions are more accurate). Moreover, since the pre-trained network is not trained to detect changes from multichannel inputs, the performance is considerably lower than that of the fine-tuned network. The difference term in the unary potential of the dense CRF and the global proportion constraint on the foreground probability mass also contributes a fair share in the final mIOU score.

During test, we use KNN to estimate an image-adaptive τ , which gives better estimate of foreground proportion and performance. We perform the KNN search using Euclidean distance on the features from the FC1 layer of the CNN model

(classification branch). We use the fast approximate KNN search method based on KD-tree which has an average complexity of at most O(log(n)). Note that for the training, we validate a fixed-value τ , which is faster than using KNN. As the pixel-level labels are unavailable in training, we set τ to a value which gives coverage of at least 15% of each image on a validation set. To compare the performance of imageadaptive τ with a fixed-value τ , we include the test segmentation scores on the CDnet-14 dataset with different fixed values of τ in Table 5. Furthermore, we evaluate the segmentation performance with different numbers of nearest neighbours used to estimate τ and notice that the best performance is achieved when the K is set to 6 (for KNN) to estimate the normalized foreground probability mass (Fig. 5). Finally, we present some error cases of our approach in Fig. 6.

5 Conclusion

This paper tackles the problem of weakly supervised change detection in paired images. We developed a novel CNN based model, which predicts change events and their location. Our approach defines a dense CRF model on top of the CNN activations and uses a modified mean-field inference procedure to enforce the compatibility between image and pixel level predictions. The proposed algorithm achieved a significant boost both in the case of detection and localisation of change events compared to strong baseline procedures. Our work is the first effort in the area of weakly supervised change detection using paired images and will find possible applications in damage detection, structural monitoring and automatic 3D model updating systems. In future, we will explore the possibility of multi-class change detection in pair of images/videos.