ICGNet: Integration Context-based Reverse-Contour Guidance Network for Polyp Segmentation

1 Introduction

Colorectal cancer (CRC) accounts for 9.4% of cancer deaths worldwide, with nearly one million cases in 2020 [Sung et al., 2021]. Polyps are abnormal tissue growth and it is the precursor to colon cancer. If polyp can be removed before colon cancer, it can effectively reduce the incidence and mortality of colon cancer. In clinical practice, colonoscopy is the gold standard, which provides information about the location and appearance of polyps. Whereas, in the process of colonoscopy, there are missing detection and involves expensive human labour. Therefore, an automatic and accurate polyp segmentation method is needed to help doctors locate the polyp area during surgery and provide a reliable prediction for further diagnosis.

∗Corresponding Author

1 Supplementary material: https:/ICGNet.github.io/Material.pdf

With the development of deep learning in medical image analysis, deep convolutional networks have made good progress in polyp segmentation. However, automatic polyp segmentation is still a challenging task, which is shown in Figure 1(a). (1) Various sizes and irregular shapes of the polyp. (2) Due to the lack of strong contrast, the boundary between polyp and background is blurred, making it difficult to be recognized. Recently, there have been some efforts to attempt to address these issues, which can be summarized as two solutions, one used reverse information (see the blue path in Figure 1(b)) and the other used edge information (see the green path in Figure 1(b)). For example, PraNet [Fan et al., 2020] generated a global map as the initial guidance region and then used the reverse attention module to reveal more complete objects. ACSNet [Zhang et al., 2020] extracted the global context from the bottom encoder and provided a local context with edge information to deal with the diversity of the shape and size of polyp areas. Nevertheless, they all have some limitations. First, reverse information contains the unpredicted and the background region (see reverse map in Figure 1(b), the red line represents the ground truth contour), the performance of the network will be limited when introducing unwanted background noise (see final predict2 in Figure 1(b)). Second, it is difficult to improve the segmentation accuracy and there is missing detection only by using edge information as shape constraint (see final predict1 in Figure 1(b)). Finally, they fail to make full use of the local-global information of the middle layer to handle polyps that change extremely in shapes and sizes.

In the face of these challenges, the above existing methods only utilized partial information, leading to unsatisfactory results. Therefore, our basic hypothesis is that reversecontour information guide learning, local-global information adaptive learning and multi-level information fuse learning can improve polyps segmentation performance. Then, we propose an integration context-based reverse-contour guidance network (ICGNet). As shown in Figure 1(c), our approach mainly includes three parts. The green part is for solving the lack of strong contrast in the feature extraction process, through designing a novel reverse-contour guidance module (RCG) to avoid the influence caused by using reverse or contour information alone. Additionally, it compensates for boundary detail loss and considers missed detection in feature sampling. The purple part proposes an adaptive local-global context module (ALGM) to extract larger and denser features and aggerate complementary information so that adapt to polyps of various sizes and irregular shapes. Lastly, we further develop a hybrid pyramid pooling fusion module (HPPF) in the orange part to better refine the segmentation results, which capture global average and local maximum features to fuse the multi-level features generated by the decoder.

The contributions of this work are summarized as follows:

• We propose the reverse-contour guidance module to receive contour detail and reverse information to highlight the boundary needed for polyp segmentation, meanwhile focusing on unpredicted targets within the contour.
• We design the adaptive local-global context module to adaptively extract local and global information and also complementary information, which can help to segment polyps with large scale and various shapes.
• We present a hybrid pyramid pooling fusion module, which extracts global average and local maximum information to guide the fusion of multi-level features to achieve refined segmentation results.

2 RelatedWork

2.1 BoundaryMethod and Reverse Attention

To enhance boundary segmentation, [Hatamizadeh et al., 2019] used a network edge branch to consider organ boundary information. SFANet [Fang et al., 2019] applied regionboundary constraints to supervise the learning of polyp. To make up for the missing object part, [Chen et al., 2020; Fan et al., 2020] utilized reverse attention block to learn missing parts and details. However, using only edge information as shape constraint or reverse attention may cause incorrect detection and introduce unwanted noise, respectively. Therefore, [Xu et al., 2021] employed both boundary and reverse attention mechanisms. Unfortunately, it is simply concatenated, without a good focus on the complementarity of each other information. Inspired by [Zhao et al., 2019] focused on the complementarity between edge and object information, we believe that effectively implementing contour features guide reverse features can better optimize the segmentation results as well. Among them, contour information can compensate for detail loss and promote the network more attention to the unpredicted object region within the boundary.

2.2 Multi-Scale Context andMulti-Level Features Fusion

Multi-scale context is beneficial to improve the accuracy of semantic segmentation. DeepLab [Chen et al., 2017] took advantage of the ASPP module to achieve multi-scale features. Although it enhances the network's ability to make use of context information, it lacks sufficient density to handle the complex cases where polyps vary greatly in size and shape. With the help of a dense connection manner [Yang et al., 2018], information can be better mined. However, these efforts have limited the flexibility of the dilation rate and cannot learn long-range dependence. Non-local operation [Wang et al., 2018] and dilation rate of hierarchy change can help solve these issues.

The fusing multi-level features have been proven to be an effective factor in improving segmentation performance [Xie et al., 2020]. RAGCM [Wu et al., 2021] utilized the contextual relationship among channels to fuse the multi-level features. Whereas using global average pooling alone overestimates the target scales and involves too much background because it averages all pixels in feature maps. [Ru et al., 2021] adopted global max pooling to help solve this problem together. It follows that combining global information and maximum response subregion information can eliminate semantic gaps and further guide the fusion of multi-level features.

3 ProposedMethod

3.1 Overview

The overall architecture of the proposed ICGNet is shown in Figure 2. It takes polyp images as inputs and outputs the segmentation results. We adopt ResNet34 [He et al., 2016] as our encoder and it contains five blocks in total. To relieve the computation burden, they are compressed to 64 channels using a convolutional layer with 64 1x1 kernels. The model integrates three types of modules, Reverse-Contour Guidance Module (RCG), Adaptive Local-Global Context Module (ALGM) and Hybrid Pyramid Pooling Fusion Module (HPPF). The RCG receives boundary information of the first encoder layer and the output from the previous D-Block which is used as the reverse attention mechanism, making the network pay more attention to target edge and false-negative regions. The ALGM uses the output feature of the encoder as the input, and further adaptively extracts local-global context to adopt the variable scales of polyps. Subsequently, the new ALGM combines with complementary information from the previous ALGM to enhance feature representation. Finally, the HPPF fuses the last three decoder blocks to refine the segmentation result.

3.2 Reverse-Contour Guidance Module

As shown in Figure 3, the original input images have blurred boundaries that disguise areas of polyps, making the network difficult to segment them. Also, there may be the problem of missing detection in the process of segmentation (see predict map in Figure 3). In order to solve these issues, the reversecontour guidance module (RCG) is proposed. RCG includes three inputs with edge feature, reverse feature and E-Block feature. The target edge feature Fedge is extracted from the low-layer of the network because the low-level features have rich detailed information (such as boundary, texture and appearance). However, these features are submerged by background noise and are difficult to be used directly. The convolutional block attention module (CBAM) [Woo et al., 2018] is used to help remove background noise, which is shown in Figure 2. Specifically, we embed feature Fedge has two advantages. It can compensate for the loss of high-level detailed information as well as let the network focus on the region of unpredicted objects within the boundary. For feature Fedge, we apply a 3 × 3 Conv and interpolate it to the same size as E to generate feature F ′e . These operations are represented as H(·). For reverse feature input is to obtain a more complete target, the predictive map of the previous D-Block to calculate a reverse attention map and then channel-wise multiplied with the E-Block feature E ∈ RC×H×W (C, H and W are the channel, height and width) get reverse feature map R ∈ RC×H×W . Next, we concatenate the features R and F ′e through a convolution operation. This operation effectively fuses reverse and boundary information according to their contributions. The process can be formulated as follows:

where H(·) denotes convolution operation and the bilinear interpolation. C[·] refers to concatenation.

After obtaining feature f , we through MLP operations generate a boundary location information weight vector multiplied by feature f to guide it to learn the region within the boundary. The results are added with feature E. The aim is to effectively combine the enhanced edge location information with the reverse semantic context to generate more accurate and complete object features. Although we achieve good results, the algorithm is still flawed. It can be calculated as:

where MLP operations include single-channel 1×1Conv and sigmoid operation.

3.3 Adaptive Local-Global Context Module

Since the size and shape of polyps are continually changing, suffer from a challenge for feature expression in segmentation. Inspired by [Zhang et al., 2020], local and global contextual information can help the network adapt to polyps of different scales and enhance feature representation. Therefore, we design an adaptive local-global context module (ALGM) to extract local and global information from encoder features to model multi-scale receptive fields. The ALGM also helps to integrate complementary information between the middle layers and the top layers, while reducing the redundant information generated by direct addition.

As shown in Figure 4. First, the encoder feature (I ∈ RC×H×W ) serves as the input of ALGM and adjusts the channel dimension to a quarter (I ′ ∈ R(C/4)×H×W ) through 3 × 3 convolution. Then, the module is divided into two parts, the first part is the global module Gglobal, which implements a non-local operation [Wang et al., 2018] to capture the long-range spatial dependencies of the feature, and it can accurately extract the global dependencies of each locational feature. It can be calculated as:

where Fnon represents non-local operation, the feature I ′ generates global information Gglobal.

Specifically, the second part is the local module Llocal, and it aims to obtain larger and denser receptive fields to achieve predominant multiscale representation. The input feature I ′ through a series of cascaded dilated convolutions with increasing dilation rates from 1 ∗ (6 - L), 3 ∗ (6 - L) to 5 ∗ (6 - L) (for the Lth layer). The input to each dilated convolution layer is formed by concatenating the input feature with the outputs from previous convolutions. Last, the outputs Z L from the global and local context are concatenated together and combine complementary information with the previous ALGM to obtain rich multiscale difference information and a more precise feature representation. Here, we use Squeeze-and-Excitation Block [Hu et al., 2018] adaptively capture channel-wise dependencies and important feature correlations for feature Q to achieve better fusion. The calculation of Llocal and Z L as follows:

where DL dl represents the dilation rate of the (dl)th dilated convolution at the Lth layer of the network. HL dl is dilated convolution. ∑ denotes the change of the dilation rates. C refers to concatenation. The SE(Q(·)) operation through Squeeze-and-Excitation block after concatenating both global and local information and Φ(·) extracts complementary information between Lth and (L - 1)th of the ALGM.

3.4 Hybrid Pyramid Pooling FusionModule

A multi-level feature fusion strategy was applied and verified to availably improve segmentation performance [Wu et al., 2021]. The feature aggregation layer has a great influence on the quality of the segmentation result, so we design a hybrid pyramid pooling fusion module (HPPF) to achieve more effective output feature fusion in our decoder.

The HPPF is illustrated in Figure 5. Given input feature maps Di ∈ RCi×Hi×Wi from the decoder, we upsample the features to the same dimension, where C, H and W are the channel, height and width, I ∈ {1, 2, 3}. Then, the three features map is concatenated altogether to extract a large feature map F ∈ RC×H×W . Next, we use a global average pooling (GAP) to obtain global information and get the completeness of targets. Meanwhile, to solve the background problem brought by GAP, we also introduce global max pooling (GMP). GMP adopts 4 × 4 and 8 × 8 pooling to involve local maximum pixels and preserves more differentiated objects as well as less background area. And then through 1 × 1 Conv, ReLU operations to obtain three pooling feature maps F1 ∈ RC×1×1 , F2 ∈ RC/16×4×4 , F3 ∈ RC/64×8×8 . Finally, we flatten F2 and F3 into the output feature vector Z ∈ RC×1×1 is calculated by weighting three pooling feature maps.

where α is a weight factor that controls the ratio of the feature map of GAP, we set 1. F kGMP is a global max pooling feature with different kernel sizes k and F i,j GAP is a global average pooling feature. f(·) denotes the transformation with two 1D convolutions. σ(·) is a Sigmoid function to produce channel attention weighting map. After applying an element-wise multiplication to reweight the F, we can get a more refined dense prediction P based on a 3 × 3 convolution.

4 Experiments

4.1 Experiment Setup

We evaluate our proposed methods on three benchmark polyp datasets: EndoScene [V´azquez et al., 2017], KvasirSEG [Jha et al., 2020], CVC-ColonDB [Bernal et al., 2012]. EndoScene contains 912 images from 44 colonoscopy sequences which are acquired from 36 patients total. KvasirSEG has 1000 images with polyp regions, manually annotated by an experienced doctor released in 2020. CVCColonDB consists of 380 annotated frames extracted from 15 different colonoscopy sequences.

Following [Zhang et al., 2020; Fan et al., 2020], we adopt ten widely-used metrics to evaluate the model performance, including "Recall (Rec)", "Specificity (Spec)", "Precision (Prec)", "Dice Coefficient (Dice)", "Intersection-over-Union for Polyp (IoUp)", "IoU for Background (IoUb)", "Mean IoU (mIoU)" , "Accuracy (Acc)", "S-measure" and "E-measure". More details of hyperparameters and loss function are reported in the implementation details and loss function respectively of supplementary material.

4.2 Ablation Study

For the sake of evaluating the effectiveness of the three key components in our proposed method, we perform the ablation study on the Kvasir-SEG dataset. The results are presented in Table 1. The ICGNet without three modules is used as the baseline, and we continually add ALGM, RCG and HPPF to it, expressed as Baseline+ALGM, Baseline+RCG, Baseline+ALGM+RCG and Ours, respectively. When the introduction of aforementioned modules by degree, boosting Dice by 1.3%, 1.41%, 2.34% and 2.92%, also adding parameters by 0.56M, 0.24M, 1.02M, 1.15M, respectively. These experiments demonstrate that the proposed three modules are lightweight and can notably improve the segmentation performance with very few parameters.

To give a more intuitive result, we visualize the heat map of outputs in Figure S1 and the network structure for each experiment in Figure S2. Concrete analysis can be seen in supplementary material.

4.3 Comparison with State-of-the-art

To validate the effectiveness of our method, we implement learning ability and generalization capability between our method and several advanced methods, including UNet++ [Zhou et al., 2018], U-Net [Ronneberger et al., 2015], ResUNet [Zhang et al., 2018], SFANet [Fang et al., 2019], PraNet [Fan et al., 2020], ACSNet [Zhang et al., 2020] and CCBANet [Nguyen et al., 2021]. In our experiments, we conducted all competitors on the same train, validation and test sets and computing environments, meanwhile using their released code with default settings to guarantee a fair comparison.

Learning Ability

In this section, we use the Kvasir-SEG and EndoScene datasets to validate our model's learning ability. We refer to the setting in [Zhang et al., 2020] to divide the training set, validation set and test set. We resize the images to 384×288 on the EndoScene dataset and fixed size of 320×320 on the Kvasir-SEG dataset. As shown in Table 2, our proposed approach achieves competitive results on all two datasets. On the EndoScene dataset, our model achieves 87.93% Dice and 89.56% mean IoU, meanwhile achieving 92.35% Dice and 91.99% mean IoU on the Kvasir-SEG dataset. It demonstrates that our network is lightweight and has a strong learning ability to effectively segment polyps. Additionally, we can see clearly in Figure S3 that the parameters of our model are smaller than advanced models while achieving comparable performance on the Dice metric in the supplementary material.

Figure 6 illustrates some visual segmentation results of our model and the compared models in some complex situations. It is observed that 1) the UNet, UNet++ and ResUNet will fail to segment them well when polyps have highly blurry boundaries (see Figure 6 (b)-(c)) and various scales (see Figure 6 (a) and (d)). These methods neither aggregate large receptivefield features nor consider the boundary constraints, so it is difficult to fit complex situations. 2) The PraNet, ACSNet and CCBANet aggregated multi-scale features significantly improve the large scales segmentation performance, and the latter two models used boundary constraints to get better results in lack of strong contrast cases (see Figure 6 (b)-(c)). But, they still have the problems of missing segmentation (see yellow arrow in Figure 6) and the segmentation result does not fit well with the ground truth (see light blue arrow in Figure 6). 3) Our model can accurately locate and segment the polyp areas by using associative learning with three modules.

Generalization Capability

In order to adapt unseen different types of polyps in clinical scenarios, it also need to improve the predictive power of the model for unknown data. In this section, we conduct one experiment to test the model's generalizability. EndoScene is a combination of CVC612 and CVC300. We use KvasirSEG and EndoScene-CVC612 to serve as the seen dataset with 90% as training set and the remaining 10% as a validation set, while CVC-ColonDB serves as the test set. All the images are set to 352x352.

To prove the generalization capability of ICGNet, we present six main metrics results in Table 3. As can be seen, our methods have the highest values on Dice (74.97%) and mIoU (82.64%) metrics and indicate that our prediction mask and ground truth have a high degree of coincidence. In the meantime, we also present the precision-recall curves in Figure 7, where the solid red line of the proposed method contains the largest area obviously, and its performance is better than other methods. These results show that ICGNet achieves a good generalization capability. It is mainly due to the following advantages. 1) Larger and denser features are extracted through local-global information that can adaptive learning various sizes and shapes of unseen polyps. 2) Performing reverse-contour layer on the middle feature map, further provide boundary localization information and encourage the network to learn more missing features. 3) The multilevel feature fusion branch supplement the information lost during the delivery process, making the final predictions more reliable. Specific qualitative result in Figure S4 and analysis is reported in the supplementary material. To further improve the reliability of generalization capability, we also conduct two experiments on the polyp segmentation datasets. The quantitative results are shown in Table S1 of supplementary material.

4.4 Conclusion

In this paper, ICGNet is presented for automatic polyp segmentation from colonoscopic images. The reverse-contour guidance module integrates low-level edge information and reverses information to solve low contrast boundary and missing detection. Adaptive local-global context module combines the local and global information, which provides the capability to enable larger and denser receptive fields to help identify diverse scales polyp. And hybrid pyramid pooling fusion module focus on prominent areas and fewer background to fine segmentation results. Extensive experiments and ablation studies demonstrate the superiority of the proposed method and achieve new state-of-the-art performance.

Acknowledgments

This work was supported in part by the Provincial Natural Science Research Program of Higher Education Institutions of Anhui province under Grant KJ2020A0035. And then, the authors acknowledge the High-performance Computing Platform of Anhui University for providing computing resources.