Bottom-up and Top-down: Bidirectional Additive Net for Edge Detection

1 Introduction

Edge detection aims to assign a label either edge pixel or nonedge pixel to each pixel given an image, which is considered as a fundamental task in computer vision and plays an important role in other higher-level tasks such as semantic segmentation [Chen et al., 2016] and salient detection [Wang et al., 2019a]. The history of researching edge detection is extremely long. Early researchers only focus on local cues such as texture gradients, colors, or other hand-crafted visual features [Martin et al., 2004]. But the lack of high-level semantic information restricts the abilities to capture accurate edges.

∗Corresponding Author

More recently, deep learning has achieved impressive results for edge detection. A series of deep learning methods have been proposed, e.g., HED [Xie and Tu, 2017], RCF [Liu et al., 2019], DeepEdge [Bertasius et al., 2015], DeepContour [Shen et al., 2015b], CEDN [Yang et al., 2016], COB [Maninis et al., 2018], CNet [Song et al., 2018], LPCB [Deng et al., 2018], AMH-Net [Xu et al., 2017], CED [Wang et al., 2019b] and BDCN [He et al., 2019]. The previous deep learning methods can be grouped into three main categories generally: 1) holistically-nested architecture, 2) top-down architecture and 3) bottom-up architecture, as shown in Fig 2.

The holistically-nested architecture is employed in HED [Xie and Tu, 2017] and [Liu et al., 2019], which extracts features from five convolutional stages of VGG-16 to obtain the side-outputs and the final result is generated by a fixed weight fusion strategy (Fig 2(a)). Generally, the low-level features are noisy but contain much spatial information, while the high-level features are coarse but carry much semantic information [Guo et al., 2019; Gao et al., 2019; Zhu et al., 2016; Shen et al., 2015a]. As shown in Fig 1, the visual edge maps from different stages and evaluation metrics verify this conclusion. Naturally, the hierarchical features are complementary for the edge detection. But a linear combination has limited ability to investigate the complex relationship between multi-level edge features since the side-outputs are isolated.

The top-down architecture in edge detection is inspired by the popular U-Net structure, in which rough high-level features carrying rich semantic information are progressively passed to lower layers (Fig 2(b)). In this way, the lost destructure tailed boundaries will be retrieved and the coarse features maps will be refined gradually in this branch. The bottom-up architecture (Fig 2(c)) is proposed in [Song et al., 2018]. The structure learns the side-outputs cumulatively, which progressively removes the detailed and noisy boundaries to enable high-resolution and accurate edge detection.

For bottom-up architecture, an ideal case is that the lowerlevel edge information is cumulatively inherited while the superfluous details are progressively abandoned. However, some useful low-level features are inevitably removed, which may cause uncorrectable effects to the resulting edge maps. This is the same case for the top-down architecture, which leads to sub-optimal results. So why don't we make full use top-down and bottom-up architectures in a collaborative manner to complement the removed information and enhance feature hierarchy? Based on that, we employ the progressive bottom-up and top-down branches to build a long-path information propagation network and refine the features in this consecutive information flow (Fig 2(d)). The combination of top-down and bottom-up architecture provides the network a chance of error-correcting, by revisiting the low-level features containing rich textual and spatial information. Furthermore, based on the observation that some features transmit superfluous or even wrong information to others as Fig 3, we incorporate channel attention mechanism into our network to select task-relevant and pivotal features.

In this paper, we propose an effective and end-to-end framework, named Bidirectional Additive Net (BAN) for image edge detection. The contributions of this paper can be summarized as: 1) We propose a progressive bottom-up and top-down information flow model for edge detection, where the feature hierarchy can be enhanced and hierarchical representations can be adaptively combined in this progressive bottom-up and top-down schemes. This architecture provides the network whit a chance of error-correcting and improves the performance gradually; 2) we propose an attended additive module to adaptively attend to each channel of the passing features so that the informative and accurate feature maps can be preserved, and the superfluous feature maps can be removed to avoid the cumulative error. To the best of our knowledge, we are the first to employ the channelwise attention mechanism for image edge detection on deep learning; and 3) experimental results show that our proposed method can improve the edge detection performance to new records and achieve state-of-the-art results on two public benchmarks: BSDS500 and NYUDv2. The ablation study also verifies the effect of each component.

2 Our Method

In this paper, we propose a novel edge detection architecture, which combines hierarchical feature maps adequately in progressive bottom-up and top-down branches. Given an input image I ∈ RH×W×3 . Our network aims to predict an accurate fused edge map Ef ∈ RH×W×1 . The framework of our architecture is shown in Fig. 4. It contains four major components: 1) pyramid feature extractor network, which extracts richer spatial features from input images by atrous pyramid architecture, 2) bottom-up branches, 3) top-down branches and 4) attended additive module (AAM). We next elaborate each of them in details.

2.1 Pyramid Feature Extractor Network

The pyramid feature extractor can be divided into two subsets: feature extractor and pyramid atrous convolution module. We denote the basic feature extractor as V , and it contains multiple convolution layers, with pooling and activation operations in five stages. Given an input image I, the basic feature extractor generates an edge feature Fi ∈ Rhi×wi×C (i = 1, 2, ..., L) in each stage followed by a 3 × 3 convolution for feature dimension reduction. L denotes the number of stages in basic feature extractor and i denotes the i-th stage.

After obtaining simplify features Fi ∈ Rhi×wi×C , a series of atrous convolution layer are employed to generate multireceptive-field edge features with the pyramid structure. We set the kernel of the convolution layers as Ar ∈ R3×3×C (r = r1, r2, ..., rK) which C denotes the output channel, r denotes the atrous rate and K denotes the number of atrous convolutions. We believe that resolution is very important in edge detection so we resize the features into input scales. This pyramid atrous convolutional operation is defined as:

where ∗ denotes the atrous convolution operation, Pi indicates the generated feature maps and U means the upsampling operations. We also add a residual shortcut between Pi and Fi in order to help gradients back propagation.

2.2 Bottom-up and Top-down Branches

Given an input image I, we generate enhanced edge features Pi ∈ RH×W×C by pyramid feature extractor in Section 2.1. Then we employ successive bottom-up and top-down branches to refine the generated edge features. The AAM in this subsection represents the attended additive module, which will be elaborated in the next subsection.

Bottom-up branch. Given a set of pyramid enhanced edge features Pi, a bottom-up branch combines the lower level features with higher level features gradually and generates fused features BUi ∈ RH×W×C (i = 1, 2, .., L - 1). For a lower level feature Pi-1 and a higher level feature Pi, we can generate fused features by:

Specially, we can observe that the final feature map carries richer information than others in a bottom-up branch. The experiments that is shown on Table 4 prove our conclusion and the effect of the bottom-up branch.

Top-down branch. Given a set of pyramid enhanced edge features Pi, a top-down branch generates features by transmitting upper-layer edge features to inferior edge features progressively. For an upper-layer feature map Pi and a lower feature map Pi-1, the top-down branch will generate features map TDi ∈ RH×W×C (i = L - 1, L - 2, ..., 1) as:

Multi branches. In this subsection, we describe the progressive bottom-up and top-down branches structure. Following this inference scheme, TDn i denotes the fused feature map in the i-th stage from the top-down branch at nth branch. In a similar way, we can define the output feature map in the i-th stage from the bottom-up and top-down branches at n-th branch as BU n i , TDn i (n = 1, 2, ...,N, i = 1, 2, ...,L - 1). The total number of branches is defined as N. The outputs of the corresponding bottom-up branch can be generated by:

The generated features from corresponding top-down branch are represented as:

2.3 Attended Additive Module

After employing the progressive bottom-up and top-down branches into our network, we can utilize multi-level features for the generation of image edge maps. In this subsection, we introduce the details of Attended Additive Module for probably fusing these features. The structure is shown in Fig 5.

In general, let f ∈ RH×W×C denote an arbitrary flowing feature map that has C channels. Firstly, we apply pooling layer to f to obtain a vector v ∈ RC , which contains the global information of f. Then two progressive 1× 1 convolution layers (the same as fully connected layer here) are employed to investigate the importance of each channel and the relationship between themselves. A sigmoid operation is employed to normalize the weight to [0, 1]. This process can be represented by the following equation:

where w denotes the channel weight, σ represents the ReLU activation function, Conv refers to convolution layer and Pool refers to pooling layer. So the AAM(·, ·) in last subsection is defined as:

Inspired by [Bertasius et al., 2015], we employ two different pooling schemes, e.g. average and max pooling, for channel reweighting.

2.4 Network Optimization

Loss function. The original cross entropy loss cannot get satisfactory performance because of the unbalance between positive and negative classes for edge detection. Following the previous work [Liu et al., 2019] [He et al., 2019], we employ a class-balanced cross entropy loss function to train the network. The loss function is defined as:

where I is an input image, and G is the final ground truth of the input image I. G+ denotes all of the edge pixels, while G- is all of the non-edge pixels. β = |G-| |G| is the ratio of edge pixels in all pixels, which is used to reweigh the positive and negative classes weights when computing loss. γ is a hyper-parameter to balance the edge and non-edge pixels ratio. W refers to all trainable parameters in our network. But the network can't generate crisp edge maps on the condition of only cross entropy loss without the non-maximum suppression (NMS). Follow the previous work, we also employ dice loss to thin the predicted edges. The ei denotes a pixel in predicted edge maps.

The mixed loss function is defined as :

where λd is a hyper-parameter that controls the trade-off between cross-entropy loss and dice loss.

Training strategies. To obtain a final edge map, we first generate a weighted-fusion output Efuse by a weighted linear combination of side-outputs En i (i = 1, 2, ...,L; n = 1, 2, ...,N). En i is generated by each Attended Additive Module followed by a 1×1×1 convolutional layer as shown in Fig 4. Inspired by the [Xie and Tu, 2017], we merge the weighted-fusion output Efuse with the side-output En i . We also supplement the edge maps generated by F1 and F5 to the final output for obtaining sufficient details and semantic information. We employ a deep supervision training strategy to train our network. It can restrain the gradients vanishing problem and guarantee a satisfactory edge prediction in each side-output, which is helpful to get a final prediction.

3 Experiments

3.1 Experiments Setup

Datasets. To evaluate our proposed method, we use two commonly used benchmark datasets: BSDS500 [Arbelaez et al., 2011] and NYUDv2 [Silberman et al., 2012]. BSDS500 benchmark dataset is an extension of the BSDS300. Follow [Xie and Tu, 2017] [Liu et al., 2019], we train our network on training and validation sets and test our method on test set. NYUDv2 is collected from a variety of indoor scenes with 1,449 RGB images and depth images. We split the whole images into two subsets: 756 for training and 654 for testing. To prevent over-fitting, we apply data augmentation strategies following [Xie and Tu, 2017].

Evaluation metrics. Following the existing works, the non-maximum suppression (NMS) is employed to generate thinned edge maps firstly. After that, we use F-measures at optimal dataset scale (ODS), optical image scale (OIS) and average precision(AP) for evaluation metrics. In the matching process, we set the maximum tolerance distance to 0.0075 for BSDS500 and 0.011 for NYUDv2 respectively.

Implementation details. We initialize our network by the VGG16 model pre-trained on ImageNet. We set the stride of pool4 to 1 and apply atrous convolution in the last stage of VGG16 to obtain reasonable resolution, which is very important for dense pixel classification. The Fuse operations in our network are implemented by addition operations. The convolution layer in our network are initialized from zeromean Gaussian distribution with 0.01 standard deviation. The hyper-parameter λd in Equation 10 is set as 300. The C,N and L are set to 21, 2 and 5 in Section 2. Following [Liu et al., 2019], λ is set to 1.1 for BSDS500 and 1.2 for NYUDv2 respectively. We employ SGD optimization during training. The learning rate, weight decay, momentum and batch size are set to 1e-7, 0.9, 2e-4 and 10, respectively. We train BSDS500 for 15k steps and NYUDv2 for 30k steps because of the different sizes of training set.

3.2 Ablation Study

The Baseline in ablation study means that the five sideoutputs are fused by a concatenation operation followed by an 1 × 1 ×1 convolution layer without information flow. We conduct experiments using BSDS500 dataset.

Different variants. Firstly we investigate the performance of different variants. We discuss two variants here: 1) first bottom-up branch and then top-down branch, 2) first topdown branch and then bottom-up branch. We call the first variant as 'v1' and another one 'v2'. We evaluate the performance of two variants and choose the 'v1' variant as our structure based on the experimental results. From Table1, we can observe that both 'v1' and 'v2' outperform the baseline, which shows the effectiveness and importance of information flow between different stages.

Different numbers of bottom-up and top-down branches. From Table 1, we can observe that the multiply top-down and bottom-up branches enhance the performance of edge detection greatly (n≤2). But when the n=3, the performance declines slightly. The reason might be that the too deep network makes the training difficult, especially for the edge detection which has extremely unbalanced and scarce training data. Even so, the performances are also outperform the baseline and single direction (n=1).

(c) ce loss (d) ce+dice loss

Figure 6: The qualitative comparison of different different loss function. 'ce loss' denotes weighted binary cross entropy loss.

Prediction-level vs. Feature-level. One of the differences between our method and [He et al., 2019] is that we employ a feature-level information flow model rather than predictionlevel. Because we think the feature maps contain richer information than edge maps. In Table 2, we compare the effect of prediction-level and feature-level information flow. From Table 2, We can draw the following conclusions: (1) The prediction-level information flow is useful for edge detection, which increases all evaluation metrics by 0.3% ODS,0.3% and 5.8% AP; (2) feature-level information flow improves the performance of edge detection significantly. Compared with the baseline, feature-level information flow performs it by 1.1% ODS, 0.9% OIS and 6.1% AP, which has greater improvement than prediction-level information flow. These prove the effectiveness of feature-level information flow mechanism and progressive bottom-up and topdown branches.

Attended additive module and different pooling schemes. In Table 2, we also discuss the attended additive module and the different pooling schemes in the module. The Featurelevel denotes that AAM without channel reweighting mechanism. As can be seen, the channel reweighting mechanism in AAM improves the performance by filtering unnecessary and noisy information and select important components on flowing features. And the different pooling schemes also improve the performance of edge detection slightly. The reason might be that the different pooling schemes capture different global information in channel reweighting, which makes the reweighting operations more robust and effective.

Side-outputs. As shown in Table 4, different stages in our networks gradually produce better performance, while the final results fuse all the side-outputs and obtain the best performance. This indicates the effect of our proposed architecture.

The loss function and training strategy. Our model is trained by a fused loss function with a deep supervision strategy. The supervisions are fed to each side-output in each branch. Tab. 3 shows that deep supervision is helpful for edge detection. The network trained by fused loss function obtains a slightly better performance. And the dice loss makes the edges crisp, which is helpful for other tasks such as optical flow and object proposal, as shown in Fig. 6.

3.3 Comparison with State-of-the-art

Comparison on BSDS500. In this subsection, we perform comparisons of our proposed BAN and other methods, including traditional methods (Canny [Canny, 1986], Pb [Martin et al., 2004], MCG [Pont-Tuset et al., 2017] and OEF [Hallman and Fowlkes, 2015]) and deep learning methods (DeepEdge [Bertasius et al., 2015], DeepContour [Shen et al., 2015b], HED [Xie and Tu, 2017], COB [Maninis et al., 2018], RCF [Liu et al., 2019], LPCB [Wang et al., 2019b], BDCN[He et al., 2019], AMH-Net [Xu et al., 2017] and CNet [Song et al., 2018]). The experimental results are summarized in Table 5. From Table 5 we can observe that our method obtains F-measure of 0.810 on ODS in single-scale input images and 0.816 of multi-scale input images, which outperforms all the previous approaches. ODS is the most important evaluation metric in edge detection which reflects the performance on the whole dataset. BAN also obtains better performance compared with CED-MS that uses multi-scale input images and ResNet-based methods such as AMH-Net and CNet on ODS. We observe that our method outperforms the others, including human for all evaluation metrics.

Comparison on NYUDv2. In this subsection, we compare our method with state of the art methods, including gpb-UCM [Arbelaez et al., 2011], OEF [Hallman and Fowlkes, 2015], gpb+NG [Gupta et al., 2013], SE [Doll´ar and Zitnick, 2015], SE+NG+ [Gupta et al., 2014], HED [Xie and Tu, 2017], RCF [Liu et al., 2019], LPCB [Deng et al., 2018] and BDCN [He et al., 2019]. In the recent work such as [Xie and Tu, 2017] and [Liu et al., 2019], the final result are obtained by averaging the results from two seperately models which are trained by RGB images and HHA images respectively. The comparison results are shown in Table 6. As shown in the results, our method outperforms other recent approaches including some ResNet-based approaches. Specifically, we can see that our method performs better than the best VGG-based approach BDCN(RGB) with an increase of 0.7% ODS and 0.7% OIS. When we combine RGB and HHA, the performance of our method sets a net record and reaches 0.773, 0.786 and 0.790. It's worth noting that the F-measure at ODS and OIS are more important evaluation metrics than AP for edge detection [Liu et al., 2019] [Deng et al., 2018]. From Table 6, we can also notice that our method outperforms ResNet-based methods: CNet and AMH-Net at F-measures.

4 Conclusion

In this paper, we propose a novel progressive bottom-up and top-down information flow model named Bidirectional Additive Net (BAN), which utilizes the feature-level information flow from bottom-up to top-down branches to enhance information propagation and obtain richer feature hierarchy. We filter the propagating information by reweighting each channel of feature maps by Attended Additive Module (AAM). Experimental results show that our proposed method outperforms the state-of-the-art on two benchmark datasets.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (Grant No. 61772116, No. 61872064, No.61632007, No. 61602049), The Open Project of Zhejiang Lab (Grant No.2019KD0AB05), Huo Yingdong Education Foundation and Sichuan Science and Technology Program (No. 2018GZDZX0032 and No. 2019JDTD0005).