Context-aware Cross-level Fusion Network for Camouflaged Object Detection

1 Introduction

Due to the purpose of survive, wild animals have developed rich camouflage abilities. In practice, they try to change their appearance to "perfectly" integrate into the surrounding environment, in order to avoid the attention from other creatures. In recent years, camouflage has attracted increasing research interest from the computer vision community [Le et al., 2019; Fan et al., 2021a; Lyu et al., 2021; Mei et al., 2021; Zhai et al., 2021]. Among various topics, camouflaged object detection (COD), which aims to identify and segment the camouflaged objects from images, is particularly popular. However, COD is a challenging task due to the nature of camouflage.

∗Co-first Authors. †Corresponding Author.

More specifically, due to the camouflage, the boundary contrast between an object and its surroundings is extremely low, leading to significant difficulties to identify/segment this object. As shown in the top row of Figure 1, it is very challenging to discover the fish from the underwater image. In addition, the camouflaged objects, most wild animals, are usually with varied appearance, e.g., size and shape, which further aggravates the difficulties of accurate COD [Le et al., 2019; Fan et al., 2021a].

To address these challenges, deep learning techniques have been adopted and shown great potentials [Le et al., 2019; Fan et al., 2021a]. In addition, a number of COD datasets were constructed for training the deep learning models. For instance, [Le et al., 2019] created the first COD dataset, called CAMO, consisting of 2500 images. However, due to the limited sample size, CAMO is unable to take full advantage of deep learning models. Recently, [Fan et al., 2021a] constructed COD10K, the first large-scale COD dataset containing 10,000 images with the consideration of various challenging camouflage attributes in the real natural environments.

Apart from datasets, these two works also contribute to the COD from the aspect of deep learning models. [Le et al., 2019] proposed an anabranch network, which performs classification to predict whether an image contains camouflaged objects, and then integrates this information into the camouflaged object segmentation task. Based on the observation that predators discover preys by first searching and then recognizing, [Fan et al., 2021a] designed a searching module and an identification module to locate rough areas and to accurately segment camouflaged objects, respectively.

Existing methods have shown promising performance in the detection of a single camouflaged object from relatively simple scenes. However, their performance degrades significantly for a number of challenging cases, such as multiple objects, occlusion, etc. As shown in Figure 1, stat-of-the-art models fail to correctly identify all camouflaged objects and show unsatisfactory performance when occlusion exits. A key solution to address these challenges is a significant large receptive field, which provides rich context information for accurate COD. In addition, the cross-level feature fusion also plays a vital role in the success of COD. However, existing works usually overlook the importance of these two key factors. It is, therefore, greatly desired for a unified COD framework that jointly considers both rich context information and effective cross-level feature fusion.

To this end, we propose a novel COD model, called Context-aware Cross-level Fusion Network (C2F-Net). We first fuse the cross-level features extracted from the backbone using an Attention-induced Cross-level Fusion Module (ACFM), which integrates the features with the guidance from a Multi-Scale Channel Attention (MSCA) component. The ACFM computes the attention coefficients from multilevel features, employs the resulting attention coefficients to refine the features, and then integrates the resulting features for the fused one. Subsequently, we propose a Dual-branch Global Context Module (DGCM) to exploit the rich context information from the fused features. The DGCM transforms the input feature into multi-scaled ones with two parallel branches, employs the MSCA components to compute the attention coefficients, and integrates the features by considering the attention coefficients. The ACFM and DGCM are organized in a cascaded manner at two stages, from high-level to low-level. The final DGCM predicts the segmentation map of the camouflaged object(s) from an image. In summary, the main contributions of this paper are as follows:

• We propose a novel COD model, C2F-Net, which integrates the cross-level features with the consideration of rich global context information.
• We propose a context-aware module, DGCM, which exploits global context information from the fused features. Our DGCM is capable of capturing valuable context information, which is the key factor for improving COD accuracy.
• We integrate the cross-level features with an effective fusion module, ACFM, which integrates the features with the valuable attention cues provided by MSCA.
• Extensive experiments on three benchmark datasets demonstrate that our C2F-Net outperforms 14 state-ofthe-art models in the terms of four evaluation metrics.

2 RelatedWork

In this section, we discuss the two types of works that are closely related to our model, i.e., camouflaged object detection and context-aware deep learning.

2.1 Camouflaged Object Detection

As an emerging field, COD has gained increasing attention in recent years. Early works rely on visual features, such as color, texture, motion, and gradient [Mondal, 2020]. In practice, the techniques relying on a single feature are unable to provide promising performance, therefore [Jiang, 2011] integrated different features for improved performance. However, limited by hand-crafted features, these methods are only suitable for relatively simple scenarios and tend to fail in realworld applications. To address this, deep learning techniques have been employed for COD by taking advantage of deep features automatically learned by the network from extensive training images, which are more generic and effective than hand-crafted features. For instance, [Li et al., 2017] identified the camouflaged object with a region proposal network. [Yan et al., 2020] observed that flipping the image provides vital cues for identifying the camouflaged object, therefore a two-stream MirrorNet was proposed with the original and flipped images as inputs for COD. [Lamdouar et al., 2020] exploited the motion information to identify camouflaged objects in videos with a deep learning framework, consisting of two components, i.e., a differentiable registration module to highlight object boundaries and a motion segmentation module with memory that discovers moving regions. This method overcomes the limitations of traditional motion information based methods that tend to fail when the target is stationary. For COD application, polyp segmentation has also received widespread attention. [Fan et al., 2020] proposed a parallel reverse attention network (PraNet), which first predicts rough regions and then refines the boundaries.

2.2 Context-aware Deep Learning

The contextual information plays a crucial role in object segmentation tasks due to its great capability of enhancing the feature representation for improved performance. Efforts have been dedicated to enriching the contextual information. For instance, PSPNet [Zhao et al., 2017] obtains rich context information by establishing multi-scale representations around each pixel. [Chen et al., 2018] employed different dilated convolutions to construct ASPP to achieve context information acquisition. DANet [Fu et al., 2019] utilizes nonlocal modules [Wang et al., 2018] to extract contextual information. CCNet [Huang et al., 2019b] obtains dense contextual information by cyclically cross-attention modules. In the field of salient object detection, [Zhang et al., 2018] utilized a multi-scale context-aware feature extraction module to allow multi-level feature maps to obtain rich context information. PoolNet [Liu et al., 2019], which is equipped with a pyramid pool module, employs a global guidance module to obtain the location information of underlying salient targets for different feature layers. GCFANet [Chen et al., 2020] utilizes a global context flow module to transfer features containing global semantics to feature mapping at different stages to improve the integrity of the detection of prominent objects. LCANet [Tan et al., 2020] adaptively integrates the local area context and the global scene context in a coarse-to-fine structure to enhance local context features.

3 ProposedMethod

In this section, we first outline the overall architecture of the proposed C2F-Net. Then, we provide the details of the two proposed modules. Finally, we give the overall loss function for the proposed model.

3.1 Overall Architecture

Figure 2 shows the overall architecture of the proposed C2FNet, which fuses context-aware cross-level features to improve the camouflaged object detection performance. Specifically, we adopt Res2Net-50 [Gao et al., 2019] to extract its features at five different layers, denoted as fi (i = 1, 2, . . . , 5). Then, a receptive field block (RFB) is adopted to expand the receptive field for capturing richer features in a specific layer. We utilize the same setting in [Gao et al., 2019], and the RFB component includes five branches bk (k = 1, 2, . . . , 5). In each branch, the first convolutional layer has dimensions 1 × 1 to reduce the channel size to 64. This is followed by two layers, i.e., a (2k-1)×(2k-1) convolutional layer and a 3×3 convolutional layer with a specific dilation rate (2k - 1) when k > 2. The first four branches are concatenated and then their channel size is reduced to 64 using a 1 × 1 convolutional operation. Then, the fifth branch is added and the whole module is fed to a ReLU activation function to obtain the final feature. After that, we propose an Attention-induced Cross-level Fusion Module (ACFM) to integrate multi-scale features, and a Dual-branch Global Context Module (DGCM) to mine multi-scale context information within the fused features. Finally, we obtain the prediction result for camouflaged object detection. We give the details of each key component below.

3.2 Attention-induced Cross-level FusionModule

There are natural differences among different types of camouflaged objects. Besides, the size of similar camouflaged objects may also vary greatly, due to the observation distance and their relative location to the surrounding environment. In other words, for a captured image with camouflaged object(s), the proportion of the camouflaged object(s) often varies. To address these challenges, we propose an ACFM via introducing a Multi-Scale Channel Attention (MSCA) [Dai et al., 2020] mechanism to effectively fuse cross-level features, which can exploit multi-scale information to alleviate scale changes. The MSCA has strong adaptability to different scale targets, and its structure is shown in Figure 2. MSCA is based on a double-branch structure, in which one branch uses global average pooling to obtain global contexts to emphasize globally distributed large objects, and the other branch maintains the original feature size to obtain local contexts to avoid small objects being ignored. Different from other multi-scale attention mechanisms, MSCA uses point convolution in two branches to compress and restore features along the channel dimension, thereby aggregating multi-scale channel context information. Features at different levels have different contributions to the task, and fusion of features at multiple levels can complement each other to obtain a comprehensive feature expression.

As discussed in [Wu et al., 2019], low-level features with greater spatial resolution demand more computing resources than high-level features, but contribute less to the performance of deep integration models. Motivated by this observation, we only carry out ACFM in high-level features. Specifically, we call fi (i = 3, 4, 5) as high-level features, and the cross-level fusion process can be described as follows:

whereMrepresents the MSCA, Fa and Fb represent the two input features. ⊕ represents the initial fusion of Fa and Fb, that is, Fb is up-sampled twice and then added element-wisely with Fa. ⊗ represents element-wise multiplication. And (1-M(Fa ⊎Fb)) corresponds to the dotted line in Figure 2. After that, the feature Fab is fed into a 3 × 3 convolutional layer, followed by batch normalization and a ReLU activation function. Finally, we obtain the cross-level fused feature F .

3.3 Dual-branch Global Context Module

We have adopted the proposed ACFM to fuse multi-scale features at different levels, which introduces the multi-scale channel attention mechanism to obtain informative attentionbased fused features. Besides, global context information is critical for improving the camouflaged object detection performance. Thus, we propose a Dual-branch Global Context Module (DGCM) to exploit rich global context information within the fused cross-level features. Specifically, the output F ∈ RW×H×C of the ACFM is fed to two branches though a convolutional operation and average pooling, respectively. We can obtain the sub features Fc ∈ RW×H×C and Fp ∈ R W2 × H2 ×C . To learn multi-scale attention-based feature representations, Fc and Fp are first fed to the MSCA component. Element-wise multiplication is adopted to fuse the outputs of MSCA and the corresponding features (Fc or Fp), and then we obtain Fcm ∈ RW×H×C and Fpm ∈ R W2 × H2 ×C . After that, we directly fuse the features from the two branches using an addition operation to obtain Fcpm. Finally, a residual structure is used to fuse F and Fcpm to get F ′ . The above process can be described as follows:

where C, P, M, and U represent the convolution, average pooling, MSCA, and upsampling operations, respectively. It is worth noting that the proposed DGCM is used to enhance the fusion features of ACFM, which makes our model can adaptively extract multi-scale information from a specific level during the training phase. The detailed structure is shown in Figure 2.

3.4 Loss Function

The binary cross-entropy loss (LBCE) is used to independently calculate the loss of each pixel to form a pixel restriction on the network. In order to make up for its deficiency of ignoring the global structure, [Qin et al., 2019] introduced IoU loss(LIoU) to focus on the global structure to form a global restriction on the network. These losses are treated equally to all pixels, ignoring the differences between pixels. Based on this, [Wei et al., 2020] improved the above two losses into the weighted binary cross-entropy loss (LwBCE ) and IoU loss (LwIoU ). By calculating the difference between the center pixel and its surrounding environment, each pixel is assigned a different weight, so that the hard pixels can be obtained more attention. In summary, the loss function of our model is defined as: L = LwIoU + LwBCE . The network prediction result P is up-sampled to have the same size with ground-truth map G. Thus, the total loss for the proposed model can be formulated as: Ltotal = L(G, P ).

4 Experiments

In this section, we first provide the implementation details, datasets and evaluation metrics. Then, we provide quantitative and qualitative comparison results, and present results of ablation studies to validate the effectiveness of key modules.

4.1 Implementation Details

We implement our model using PyTorch and adopt Res2Net50 [Gao et al., 2019] pretrained on ImageNet as our network backbone. We carry out a multi-scale training strategy {0.75, 1, 1.25} instead of data augmentation and resize all input images to 352×352. We utilize the AdaX [Li et al., 2020] optimization algorithm to optimize the overall parameters by setting the initial learning rate as 1e - 4. Accelerated by an NVIDIA Tesla P40 GPU, the whole network takes about 4 hours to converge over 40 epochs for training with a batch size of 32. During training, the learning rate decays 10 times after 30 epochs.

4.2 Datasets

We conduct experiments on three public benchmark datasets for camouflaged object detection. The details of each dataset are as follows:

• CHAMELEON [Fan et al., 2021a] contains 76 camouflaged images, which are collected through the Google search engine with the keyword "camouflage animals".
• CAMO [Le et al., 2019] has 1,25k images (1k for training, 0.25k for testing), including 8 categories.
• COD10K [Fan et al., 2021a] is currently the largest camouflaged object detection dataset. It consists of 5,066 camouflaged images (3040 for training, 2026 for testing), which is divided into 5 super-classes and 69 subclasses. This dataset also provides high-quality fine annotation, reaching the level of matting.

Training/Testing Sets. Following the training setting in [Fan et al., 2021a], for both CAMO and COD10K datasets, we use the default training sets. Then, we evaluate our model and all compared methods on the whole CHAMELEON dataset and the test sets of CAMO and COD10K.

4.3 EvaluationMetrics

To comprehensively compare our proposed model with other state-of-the-art methods, we utilize four popular metrics to evaluate the COD performance. · Mean absolute error (MAE, M) [Perazzi et al., 2012] evaluates the average pixellevel relative error between the normalized prediction and the ground truth. · Weighted F-measure (F w β ) [Margolin et al., 2014] is an overall performance measurement that synthetically considers both weighted precision and weighted recall. · S-measure (Sα) [Fan et al., 2017] computes the object-aware and region-aware structure similarities between the prediction and the ground truth. · E-measure (Eφ) [Fan et al., 2021b] is based on the human visual perception mechanism to evaluate the overall and local accuracy of COD.

4.4 Performance Comparison

We compare the proposed method with 14 state-of-the-art COD methods, including FPN [Lin et al., 2017], MaskRCNN [He et al., 2017], PSPNet [Zhao et al., 2017], UNet++ [Zhou et al., 2018], PiCANet [Liu et al., 2018], MSRCNN [Huang et al., 2019a], BASNet [Qin et al., 2019], PFANet [Zhao and Wu, 2019], CPD [Wu et al., 2019], HTC [Chen et al., 2019], EGNet [Zhao et al., 2019], ANetSRM [Le et al., 2019], SINet [Fan et al., 2021a] and MirrorNet [Yan et al., 2020]. For a fair comparison, the results of the first 13 methods are taken from [Fan et al., 2021a]. Besides, the results of MirrorNet based on ResNet50 are taken from https://sites.google.com/view/ltnghia/research/camo.

Quantitative Evaluation. Table 1 summarizes the quantitative results of different COD methods on the three benchmark datasets. On the three datasets, the four evaluation indicators all have better performance than previous methods. Specifically, compared with the ResNet50-based SINet specifically designed for COD tasks, Sα increased by 4.54% on average, Eφ increased by 8.71% on average, and F w β increased by 18.35% on average.

Qualitative Evaluation. Figure 3 shows the qualitative results of different COD methods. Examples are from the five super-classes of COD10K from top to bottom: aquatic, terrestrial, flying, amphibious, and others. Compared with other cutting-edge models, our model can achieve better visual effects by detecting more accurate and complete camouflaged objects with rich details.

4.5 Ablation Study

In order to verify the effectiveness of each key module, we designed three ablation experiments, and the results are shown in Table 2. In No.1 (Basic) experiment, we removed all ACFMs and DGCMs while keeping RFB modules, and then simply fused the features from the last three layers via a summation operation. In No.2 (Basic+ACFM) experiment, we removed the DGCMs and directly connected the two ACFMs. In No.3 (Basic+DGCM) experiment, we replaced the ACFMs with the combined operation of upsampling and then adding, and the DGCMs remained unchanged. No.4 (Basic+ACFM+DGCM) is the complete model, which is consistent with the structure of Figure 2. Besides, we replaced all the MSCAs in the whole network with the convolution layers to verify the effectiveness of MSCA, i.e., "MSCA→Conv" in Table 3.

Effectiveness of ACFM. We investigate the importance of the ACFM. From Table 2, we observe that No.2 (Basic + ACFM) outperforms No.1 (Basic), clearly showing that the attention-induced cross-level fusion module is necessary for improving performance.

Effectiveness of DGCM. We further investigate the contribution of the DGCM. We observe that No.3 (Basic + DGCM) improves No.1 (Basic) performance on the three benchmark

(a) Image

(b) GT

(c) Ours

(d) SINet

(e) EGNet

(f) CPD

(g) BASNet

(h) MSRCNN

Figure 3: Qualitative results of our model and five state-of-the-art methods (i.e., SINet [Fan et al., 2021a], EGNet [Zhao et al., 2019],

CPD [Wu et al., 2019], BASNet [Qin et al., 2019], and MSRCNN [Huang et al., 2019a]).

datasets. These improvements suggest that introducing dualbranch global context module can enable our model to accurately detect objects.

Effectiveness of ACFM & DGCM. To assess the combination of the ACFM and DGCM, we evaluate the performance of No.4. As shown in Table 2, our model is generally better than other settings. The visual comparison results in Figure 4 also show that the complete structure is more conducive to detecting camouflaged objects.

Effectiveness of MSCA. To validate the effectiveness of MSCA, we utilize a convolution layer instead of it (denoted "MSCA→Conv"). The comparison results are shown in Table 3. It can be observed that C2F-Net outperforms "MSCA→Conv" on three datasets in terms of two evaluation metrics. In particular, the use of MSCA significantly improves the F w β score by a largemargin of 1.9%on theCAMOTest dataset.

5 Conclusion

In this paper, we propose a novel network C2F-Net for camouflaged object detection, which integrates the cross-level features with the consideration of rich global context information. We propose a context-aware module, DGCM, which exploits global context information from the fused features. We integrates the cross-level features with an effective fusion module, ACFM, which integrates the features with the consideration of valuable attention cues provided by MSCA. Extensive experimental results on three benchmark datasets demonstrate that the proposed model outperforms other stateof-the-art methods.