Densely Cascaded Shadow Detection Network via Deeply Supervised Parallel Fusion

1 Introduction

Shadow occurs frequently in natural scenes. Patterns of shadows provide important cues for estimating physical properties of the scene e.g., light source [Lalonde et al., 2010], illumination conditions [Panagopoulos et al., 2012] and scene geometry [Karsch et al., 2011]. Understanding regions of shadows can also help to improve downstream vision tasks, such as image segmentation [Ecins et al., 2014]. Moreover, removing shadows from images has many applications in computational photography [Guo et al., 2011; Qu et al., 2017]. Among these tasks, shadow detection-identify regions of shadows, is a core step. We focus on the task of shadow detection from a single image in this paper.

Finding shadows from a single image is very challenging. The problem is fundamentally ill-posed, as scene geometry, surface property and lighting conditions can not be recovered from a 2D image. Early works made use of image prior and physical models of illumination and colors [Finlayson et al., 2009; 2006]. These model based methods will fail if the underlie assumptions, such as Lambertian reflectance, are violated in natural images. To address this challenge, several recent works proposed learning-based approaches [Guo et al., 2011; Zhu et al., 2011; Tumblin and Williams, 2011; Khan et al., 2016; Vicente et al., 2018] for shadow detection. More recently, deep Convolutional Neural Networks (CNN) [Krizhevsky et al., 2012] have been exploited, and significantly advanced the performance [Vicente et al., 2016; Nguyen et al., 2017] on public benchmarks.

Our method follows the same paradigm of learning deep models for shadow detection, and forms the problem as a dense binary labeling of pixels. Our work is different from [Vicente et al., 2016; Nguyen et al., 2017] by revisiting the idea of a fully convolutional network [Shelhamer et al., 2017; Xie and Tu, 2015]. Specifically, we argue that both global context and local appearance are encoded within the hierarchy of a CNN. And we propose a novel architecture for combining these cues for shadow detection. Unlike [Vicente et al., 2016], we use a single network and do not use separate models for global image prior and local patch appearance. Unlike [Nguyen et al., 2017], we show that a binary entropy loss can outperform the adversarial loss with proper design of the network architecture. Fig 1 presents our results.

To this end, we propose a novel Densely Cascaded Deeply Supervised Parallel Fusion (DC-DSPF) network for shadow detection. Our DC-DSPF model equips an existing network [Xie and Tu, 2015] with two key components: (1) the deeply supervised parallel fusion; and (2) the densely cascaded learning. These two components are combined to obtain a better fusion of global context and local appearance in input images. The deeply supervised parallel fusion network seeks to combine both multi-scale visual features across the hierarchy of the backbone network. The densely cascaded learning scheme aims at integrating global contextual cues.

Specifically, our proposed Deeply Supervised Parallel Fusion (DSPF) stacks multiple parallel fusion units [Pinheiro et al., 2016; Wang et al., 2017b] on the backbone network. Each unit is a backward fusion branch that progressively merges feature maps from a previous unit (with the first one from the backbone network), and generates a full resolution output at the end. Loss functions are attached to all ends. By stacking multiple fusion branches together, DSPF network outputs multi-level feature maps, and learns to best fuse them in a deeply supervised manner [Lee et al., 2015].

Moreover, our densely cascaded learning scheme sticks two DSPF networks together. We concatenate the initial prediction map of the first DSPF network and the original input image as the input of the second one. We also connect the multiple intermediate predictions generated by the parallel fusion branches in the first DSPF network to the corresponding intermediate predictions in the second one [Huang et al., 2017]. In this case, multi-scale features in the previous DSPF network can be directly propagated to the next network. Therefore, the densely cascaded DSPF network can learn to better reason about the spatial context.

To verify our model, we conduct extensive experiments on the standard benchmarks. Our method outperforms the stateof-the-arts by a significant margin on the commonly used SBU [Vicente et al., 2016] and UCF [Zhu et al., 2011] datasets.

2 Densely Cascaded Network via Deeply Supervised Parallel Fusion

This section presents our Densely Cascaded Deeply Supervised Parallel Fusion (DC-DSPF) model for shadow detection. We start by introducing HED which is the basis of our model. Then we describe the key components of our proposed DC-DSPF model.

2.1 HED for ShadowDetection

Our proposed network is derived from the HED [Xie and Tu, 2015]. Thus we start by introducing the HED network. Note that we re-purpose HED for shadow region segmentation.

HED employs a deep CNN with multiple side outputs and generates per-pixel binary predictions. Specifically, HED

network computes multi-scale prediction maps at all sideoutputs and fuses these intermediate maps as the final output. HED thus leverages multi-scale representations given by the hierarchy of the network, and has been proven effective for detecting image boundaries.

Formally, we denote the training samples as {(Xn, Yn)}, where n = 1, ...,N indexes the samples. Xn is an input image and Yn = y(n) j , j = 1, ..., |Xn|, yn j ∈ {0, 1} is its ground truth shadow labels with pixel index j. We may drop the subscript n when it is clear from the context. HED is fully convolutional. And we denote the union of all convolutional weights in base network as W. HED further attaches M sideoutputs to the network at each scale m. And we denote their parameters as w = (w(1) , ..., w(M) ). w(m) can be considered as an individual linear classifier for features at scale m. For each scale, HED computes P (yj = 1|X; W, wm) = σ(ˆa(m) j ) on its activation map ˆ A(m) at all pixels with sigmoid function σ(.). Their outputs ˆ Y (m) = σ( ˆ A(m) ) forms the multi-scale outputs for shadow detection.

Fusion: These multi-scale side-output predictions are further linearly combined to generate the output prediction, as

with the fuse loss Lfuse = l(Y, ˆ Y ) as the loss between Y and ˆ Y . Therefore, the final loss function of HED is given by

where Lside(W,w) is the sum of all side-output loses, given by Lside(W, w) = ΣMm=1 l(m) (W, w(m) ).

Loss Function: HED uses weighted cross-entropy loss for each side-output, as a remedy to unbalanced samples. The loss function is given by,

where β = |Y-|/|Y | and 1 - β = |Y+|/|Y |. Y+ and Y- are the sets of mask and non-mask labels.

Enhanced HED network: We start with ImageNet pretrained VGG16 network [Simonyan and Zisserman, 2014] for HED, and use an improved version from [Hou et al., 2017] for shadow detection. Specifically, we add two additional convolutional layers at each side output of HED. Moreover, we use larger kernel size in higher layers as suggested by [Hou et al., 2017]. For the five blocks of HED, we set the kernel sizes of the newly added convolutional filters as 3x3, 3x3, 5x5, 5x5, and 7x7. We name this new version as Enhanced-HED and use it as the backbone for our proposed method.

2.2 Densely Cascaded Network via Deeply Supervised Parallel Fusion

The linear fusion of multi-scale outputs in HED does not fully capture the rich interactions between global context and local appearance. Thus, we propose a novel model that consists of two key components: (1) a deeply supervised parallel fusion network; and (2) a densely cascaded learning scheme.

Deeply Supervised Parallel Fusion network: We propose a Deeply Supervised Parallel Fusion (DSPF) network to replace the linear fusion in HED and better capture global and local cues. Fig 2 presents an overview of our proposed DSPF network. In order to combine global semantics with local appearance, DSPF network stacks multiple parallel fusion units, and learns to produce multi-level intermediate predictions in a deeply supervised manner. Then the final fusion of these intermediate predictions thus effectively combines both global semantic cues and local spatial details of input images.

More concretely, our DSPF network contains three parallel backward fusion branches. We observe that deeper layers in HED encode rich global semantic cues, while lower layers capture more local spatial details. To obtain a comprehensive fusion of these hierarchical feature maps, and inspired from [Pinheiro et al., 2016; Wang et al., 2017b], we first utilize a backward fusion branch Branch(1) to progressively combine these multi-scale features in a top-down manner. This fusion pathway starts from the deepest block of HED, and generates the first fusion map by transferring feature map of this deepest HED block with a 1 × 1 convolutional layer. The produced fusion map is then combined with the feature map in adjacent HED block via a backward fusion module. In this way, this branch progressively combines feature maps in every two adjacent blocks of HED from deeper-level layers to lower-level layers with the backward fusion module.

Specifically, we denote the fusion map from ith block as Fi. Thus, Fi+1 is the map from upper block with reduced resolution. Our backward fusion module is given by,

where Up(·) denotes the up-sampling of a feature map. Concat(·) means the concatenation of two feature maps which have the same resolution. Conv(·) refers to the 1 × 1

convolution operation. We also show the details of this backward fusion module in Fig 3. For these two adjacent feature maps, we first up-sample Fi+1 to match the resolution of Fi, then we merge these two feature maps by concatenation. Finally, another convolutional layer is employed to generate final fused feature map Fifusion of current ith block. Branch(1) runs from deeper layers to lower layers until the first block of HED by utilizing this backward fusion module recursively. Thus feature maps in higher-level layers are progressively combined with features in lower-level layers via these recursive fusion operations. Simultaneously, the feature resolution is also enlarged gradually. Global semantics in deeper layers are thus gradually integrated with spatial details in lower layers. Hence, the final fusion map of Branch(1) achieves a good fusion of the multi-scale features across the hierarchy of the base HED network. We can predict the initial response map ˆ Y (1) utilizing this final fused map which has the same resolution as the input image.

Branch(1) computes between every two adjacent blocks at multiple stages in a top-down manner. These multistage fusion maps also preserve multi-scale features, which are beneficial to obtain a better fusion of global semantics and local details in input images. We thus propose to stack another backward fusion branch Branch(2) to further fuse these multi-stage fusion maps generated by Branch(1) . Branch(2) also runs in a top-down way and combines features of every two adjacent blocks of Branch(1) via the backward fusion module until the last block of Branch(1) . Similar with Branch(1) , Branch(2) produces the second response map ˆ Y (2) utilizing the final fusion map of this branch. Furthermore, Branch(2) also generates multi-scale fusion maps at multiple stages. So we stack the third backward fusion branch Branch(3) for further fusion. Then we can obtain the third prediction ˆ Y (3) using the final fusion map.

These three intermediate shadow predictions ˆ Y (1) , ˆ Y (2) , ˆ Y (3) generated by the corresponding branches Branch(1) , Branch(2) , Branch(3) receive direct supervision frommultiple loss functions. With the help of this deep supervision scheme, these three intermediate predictions are enforced to capture multi-level cues. Then we can obtain the final shadow map by a weighted fusion of these these intermediate responses:

where the "+" means the element-wise sum. The fusion weights w(1) ,w(2) ,w(3) are learned in the training phase. The loss function of DSPF network is given by

where the Lfuse = l(Y, ˆ Y ) is the loss between ground truth Y and prediction ˆ Y , the LBranch(1) , LBranch(2) , LBranch(3) are the losses for Branch(1) , Branch(2) , Branch(3) respectively. Note that since there are more negative non-shadow pixels than positive shadow pixels, as in HED [Xie and Tu, 2015], we use the weighted cross-entropy loss for all loss functions. In this deeply supervised scheme, the final shadow map ˆ Y learns to capture rich global semantic cues and local spatial details in an input image.

Densely cascaded learning scheme: As shown in Fig 1, global contextual cues are important to segment shadows from noisy background in natural images. In order to make the model robust to various backgrounds, we propose to integrate rich contextual cues of input images into our model. Inspired by auto-context [Tu and Bai, 2010; Li et al., 2016], we propose a densely cascaded learning architecture to stack multiple DSPF networks together. This densely cascaded learning scheme specifically densely connects multiple intermediate predictions of a previous DSPF network to the next DSPF network. For a trade-off between efficiency and accuracy, we utilize two DSPF networks to form the final model via this densely cascaded learning scheme. The whole network is denoted as DC-DSPF network, and we present the details in Fig 4.

Given an input image I, the first DSPF network produces initial prediction map ˆ Y ′ , which is a weighted fusion of the three intermediate predictions ˆ Y (1)′ , ˆ Y (2)′ , ˆ Y (3)′ . The second DSPF network is staked after the first one to introduce contextual cues into our model. Specifically, ˆ Y ′ is concatenated with original input image I as the input of this second DSPF network. Moreover, these three intermediate predictions ˆ Y (1)′ , ˆ Y (2)′ , ˆ Y (3)′ of the first DSPF network are summed with three corresponding predictions ˆ Y (1)′′ , ˆ Y (2)′′ , ˆ Y (3)′′ in this second DSPF network respectively.

Moreover, the second DSPF network can obtain refined prediction ( ˜ Y ) by

Finally, ¯ Y is further summed with the initial estimation ˆ Y ′ as final output of our DC-DSPF network:

Through this densely connected cascade learning scheme, multi-scale feature maps encoded in the first DSPF network are effectively propagated to the second DSPF network. The whole network thus implicitly learns the spatial contextual cues in input images.

2.3 Implementation Details

We now describe our training details for DC-DSPF. All our models are trained using Caffe [Jia et al., 2014] as backend. We first train HED for segmenting shadow, using the public implementation of HED [Xie and Tu, 2015]. The hyperparameters, including the initial learning rate, weight decay and momentum, are set to 1e-8, 2e-4 and 0.9, respectively. Our DSPF network is initialized from the trained HED network. And our DC-DSPF is further trained on top of DSPF. The hyper-parameters of DC-DSPF are set to 1e-8, 2e-4 and 0.99 respectively for the initial learning rate, weight decay and momentum. All new convolutional layers are initialized with Gaussian random distribution with fixed mean (0.0) and variance(0.01). We apply random flipping for data augmentation during training.

3 Experiments and Results

We introduce our benchmark on shadow detection, and present two main results: (1) an ablation study of our method; and (2) a comparison of our method to the state-of-the-art methods. In addition, we demonstrate preliminary results of using the detected shadow regions for shadow removal.

3.1 Datasets and EvaluationMetrics

Datasets: We evaluate our method on two widely used benchmarks: SBU [Vicente et al., 2016] and UCF [Zhu et al., 2011]. UCF dataset consists of 245 images. Moreover, SBU dataset contains 4089 training images and 639 testing images. SBU is currently the largest and most challenging dataset for shadow detection. It covers various scenes including urban, beach, mountain, roads, parks, snow, animals, vehicles, and houses, and also contains various picture types including aerial, landscape, and close range.

Protocol: Following the evaluation protocol of [Vicente et al., 2016], we train our models on SBU training set, and evaluate the trained models on SBU testing set and UCF testing

,

(9)

where TP,FN,TN,and FP are true positives, false negatives, true negatives, and false positives, respectively. BER is widely accepted due to the unbalanced nature of shadow data: there are much fewer shadow pixels than non-shadow ones in natural images. We also report separate per-pixel Error Rates (ER) for shadow and non-shadow pixels.

3.2 Ablation Study

We first evaluate different components of our method to better understand the proposed model. All results for our ablation study is trained on SBU training set and reported on SBU testing set. Specifically, we compare the following methods and their results are summarized in Table 1.

HED and Enhanced-HED: HED and Enhanced-HED are the backbone for our network, thus we report their results as the baseline. Compared to HED, Enhanced-HED significantly reduces the error of BER by 0.9. This result confirms the benefit of larger receptive field as suggested in [Peng et al., 2017] and [Hou et al., 2017].

DSPF Network: Based on Enhanced-HED, we further add our Deeply Supervised Parallel Fusion architecture (DSPF). We also vary the number of backward fusion branches from 1 to 3, denoted as Single/Double/Triple Fusion DSPF. Single fusion reduces the error of Enhanced-HED by 0.3 BER. Double and triple further reduces the error of Single-FusionDSPF by another 0.2 and 0.5, respectively. These results demonstrate that (1) the proposed backward fusion branch is highly effective; and (2) stacking multiple fusion branches helps to improve the performance.

)

HED.

Deep Supervision: We also test the effectiveness of deep supervision. Specifically, we take the Triple-Fusion-DSPF network, remove its losses for the intermediate prediction outputs at the first two backward fusion branches, and supervise the whole network only with the loss of the third backward branch. This network, called Triple-Fusion-PF (without supervision), is significantly worse than Triple-Fusion-DSPF (0.4 drop in BER). The result shows the superiority of the deeply supervised learning [Lee et al., 2015].

DC-DSPF Network: Finally, we combine our best model (Triple-Fusion-DSPF) with densely cascaded learning scheme. This DC-Triple-Fusion-DSPF network further reduces the error rate by 0.3, reaching a BER of 4.9. To further understand this gap, we visualize the results of both methods in Fig 5. This result supports our design of cascaded learning.

3.3 Comparison to State-of-the-art Methods

We further compare our model of DC-DSPF network with three state-of-the-art shadow detection methods: StackedCNN [Vicente et al., 2016], scGAN [Nguyen et al., 2017], and DSC [Hu et al., 2018] on SBU and UCF testing set. Moreover, we include the HED network as a baseline. The results are presented in Table 2.

Surprisingly, the baseline of HED already outperforms stack-CNN and scGAN. Our DC-DSPF further improves the performance of HED and outperforms the state-of-theart methods. For BER, DC-DSPF obtains a significant error reduction by 46% and 31% than scGAN on SBU and UCF dataset, respectively. Moreover, our method can consistently reduce the error in shadow pixels (40% on SBU and 16% on UCF) and detect more none shadow pixels (51% on SBU and 39% on UCF) in comparison to scGAN. Our DC-DSPF also obtains a clear error reduction than previous best performing DSC by 0.7 BER on SBU and 0.2 BER on UCF.

We also present visual comparison between the results of our method and the state-of-the-art methods as shown in Fig 6. DC-DSPF preserves rich contextual cues and achieves more precise estimation on the easily misclassified regions where global cues should be taken into consideration.

Moreover, similar to [Hu et al., 2018], we also compare our DC-DSPF with methods for semantic segmentation (PSPNet [Zhao et al., 2017]) and saliency detection (Amulet [Zhang et al., 2017], SRM [Wang et al., 2017a]) on the task of shadow detection in Table 3. Even the baseline HED significantly surpasses PSPNet, Amulet, and SRM on SBU and UCF. Our DC-DSPF further increases the performance gap. The results demonstrate the superior performance of our method. Furthermore, these results also suggest the fundamental difference in the modeling of semantic segmentation, saliency detection and shadow detection. These three tasks are all formulated as dense labeling of pixels. However, the modeling of these tasks differs significantly. Semantic segmentation requires the reasoning of global context between objects and scene components [Zhao et al., 2017]. This is not required for saliency detection and shadow detection. In comparison to saliency detection [Zhang et al., 2017; Wang et al., 2017a], shadow detection focuses on the local properties of image regions (e.g., colors and textures) and does not require explicit modeling of objects.

3.4 Results of ShadowRemoval

Finally, we make use of our output shadow mask for the application of shadow removal in a static image. We employ the method in [Corina B. et al., 2011]-a simplified version of [Guo et al., 2011]. As [Corina B. et al., 2011] denotes that shadow detection remains a challenge for obtaining shadowfree images. The performance of shadow detection results directly influences the quality of shadow removal results. We supply the shadow mask generated by our DC-DSPF network and original image as the input of this method, and produce the shadow-free results. Qualitative examples are shown in Fig 7. These examples suggest that our precise shadow detection enables good shadow removal results.

4 Conclusion

In this work, we consider the challenging problem of shadow detection. We propose a novel DC-DSPF network that combines a DSPF network and a densely cascaded learning architecture. The DSPF network stacks multiple parallel fusion branches, and learns a comprehensive fusion of global semantic cues and local spatial details in a deeply supervised way. Moreover, the densely cascaded learning scheme helps to capture rich contextual cues. Experimental results show that our DC-DSPF network significantly outperforms stateof-the-art methods on major benchmarks.

Acknowledgements

This work is partial funded by the National Key Research and Development Program of China (Grant No.2016YFB1001005), the National Natural Science Foundation of China (Grant No.61673375, Grant No.61602485), and the Projects of Chinese Academy of Science (Grant No.QYZDB-SSW-JSC006, Grant No.173211KYSB20160008).