A Consistency and Integration Model with Adaptive Thresholds for Weakly Supervised Object Localization

1 Introduction

Object detection and localization based on the fully supervised network [Liu et al., 2016; Bochkovskiy et al., 2020; Wang et al., 2023] has achieved great success in recent years, but requires costly bounding box annotations. Weakly supervised object localization (WSOL) determines the position and size of objects through only image-level labels, which attracts wide attention due to its low cost and labor savings.

As a pioneering work, [Zhou et al., 2016] aggregated features from the last convolutional layer in classification networks to learn the class activation map (CAM) for localizing objects. The classification network tends to focus on the most discriminative regions, while the localization task requires the extraction of the entire area of objects. The range of activation regions in CAM is significantly smaller than the actual range of objects, leading to poor localization accuracy. Thus, how to expand the activation areas of objects has become an urgent problem to be solved.

∗Corresponding author.

Many subsequent CAM-based works used adversarial erasing [Kumar Singh and Jae Lee, 2017; Zhang et al., 2018a; Choe and Shim, 2019; Mai et al., 2020], spatial relationship activation [Xue et al., 2019; Zhang et al., 2020; Guo et al., 2021], and foreground prediction map [Xie et al., 2021; Wu et al., 2022; Xie et al., 2022] to alleviate the problem of insufficient activation. However, the essence of the above methods is to obtain the local features first and then conduct the activation diffusion, which cannot solve the problem of the restricted activation regions due to local representations captured by the limited receptive field of the convolutional neural network (CNN).

In recent years, Vision Transformer (ViT) [Dosovitskiy et al., 2020] has achieved amazing results in computer vision due to the excellent extraction capability of long-distance feature dependencies. ViT used the multiple transformer blocks with a multi-head self-attention mechanism to successfully extract global features for classification. TS-CAM [Gao et al., 2021] first applied the transformer structure [Touvron et al., 2021] in WSOL, which learned the long-distance dependencies between pixels and alleviated the partial activation problem in CNN networks. However, most of the current researches consider using the deep features in the network for object localization, while ignoring the shallow features that contain rich image details. Deep features preserve the most discriminative information for classification but lose object details (e.g., boundaries and object parts insignificant for classification), as shown in Figure 1(b). Moreover, most existing methods [Bai et al., 2022] segment the localization maps through a fixed threshold for all input images, which may lead to an inappropriate localization result (smaller or bigger one), as shown in Figure 1(c).

Based on the above analysis, we proposed a novel method, namely the Consistency and Integration model with Adaptive Thresholds (CIAT), which promotes and integrates the consistency of shallow and deep features and derives an adaptive threshold for the detection of object regions. CIAT contains a new consistency and integration module of shallow-deep features (CISD) and an online adaptive threshold (OAT) for the final localization. CISD designs an efficient shallow feature extraction module and uses shallow-deep feature consistency to determine the attention value for each class channel of the semantic feature map, forcing the model to explore both the weak response regions and high response areas, as shown in Figure 1(d). OAT is adopted to estimate the segmentation threshold of each localization map, which aims to predict closer bounding boxes to the ground truths, as shown in Figure 3. OAT segments the foreground and background from the localization map and determines an adaptive threshold (i.e., the decision boundary) via Bayesian decision theory. The theoretical threshold is approximately realized by the combination of the mean of foreground and background centers and an adjustment item for compensation, segmenting localization maps with different activation distributions.

Our main contributions can be summarized as:

• We propose the new Consistency and Integration Model with Adaptive Thresholds (CIAT) for WSOL task, which learns accurate bounding boxes to greatly improve the localization performance.
• We introduce a simple consistency and integration module of shallow-deep features (CISD) that integrates the advantages of different features to enhance the entire object perception capability of the model.
• We design an online adaptive threshold (OAT) to obtain an approximation of the theoretical threshold, adaptively and effectively detecting more complete object regions in different images.
• To validate the effectiveness of the proposed methods, we perform a series of experiments on two challenging datasets and have achieved better results than some representative methods.

2 RelatedWork

2.1 CNN-basedMethods forWSOL

As a representative method of WSOL, [Zhou et al., 2016] observed that the global average pooling (GAP) layer [Lin et al., 2014] can be applied for localizing discriminative regions in images. They inserted a GAP layer behind the last convolution layer and aggregated deep features to generate the class activation map (CAM) for object localization. To expand the activation regions in CAM, HaS [Kumar Singh and Jae Lee, 2017] randomly erased a certain amount of patches with high activation in the input images, forcing the network to capture non-salient features. ACoL [Zhang et al., 2018a] and ADL [Choe and Shim, 2019] erased some discriminative patches in feature maps to further expand the activation regions. MEIL [Mai et al., 2020] utilized two parallel CNN networks with shared weights while simultaneously exploring both class-specific regions and non-salient areas.

Besides the erasing methods described above, DANet [Xue et al., 2019] proposed a divergent activation approach to get better localization maps. I2C [Zhang et al., 2020] introduced the intra-class similarity of different images to obtain the complete localization maps. SLT-Net [Guo et al., 2021] divided WSOL into two independent sub-tasks and strengthened the learning tolerance to semantic mistakes to improve the localization performance. ORNet [Xie et al., 2021] used low-level features instead of high-level features in the classification network to learn localization results with clearer boundaries. BAS [Wu et al., 2022] and C2AM [Xie et al., 2022] directly applied a generator to generate a foreground prediction map (FPM) for localization and constrain the FPM through some appropriate loss functions.

Although local features have been well studied in these CNN-based methods, CNNs are prone to capture partial semantic features and the entire object regions cannot be well activated due to the lack of global feature perception.

2.2 Transformer-basedMethods forWSOL

CNN-based WSOL approaches inevitably focus on the discriminative regions of objects, while the transformer-based methods do well in extracting the global features of input images. ViT [Dosovitskiy et al., 2020] adopted the self-attention mechanism to capture the long-range feature dependencies, which performed well in image classification tasks. Inspired by ViT, TS-CAM [Gao et al., 2021] applied the transformer structure [Touvron et al., 2021] in WSOL for the first time, which used attention maps from patches to avoid partial activation. Then based on the TS-CAM, LCTR [Chen et al., 2022] incorporated cross-patch information and used local features to enhance the local perception capability to weak response regions. SCM [Bai et al., 2022] mainly considered enhancing spatial and semantic correlation of patch tokens through the graph diffusion method. LCAR [Pan et al., 2023] proposed a dynamic aggregation network that replaced the post-processing of threshold segmentation to get closer predictions. TAFormer [Meng et al., 2023] learned the classagnostic foreground maps to get complete object localization. CATR [Chen et al., 2023] enhanced the category awareness of self-attention maps by learning category-aware representations for specific objects.

Despite the progress, these methods based on transformer structure ignore the shallow features in the first few layers of the neural network, which contain rich spatial information. An intuitive observation is that the shallow semantic feature has better spatial resolution while the deep semantic feature has better class discrimination ability. Our proposed method aims to force the model to explore both the detailed features and global features.

3 Methodology

In this section, we first give the overview of our proposed CIAT and then provide a detailed description of CISD and OAT. The modules are combined with the transformer structure into a joint optimization localization framework, as shown in Figure 2.

3.1 Overview

For ViT [Dosovitskiy et al., 2020], an input image I ∈ RH×W×3 is split and then linearly projected into to N = h×w patch tokens xp ∈ R1×D of size P ×P , where D is the feature dimension of each token, h = H/P and w = W/P . To extract global information from tokens, an extra class token xclass ∈ R1×D is added. Before these patch tokens are fed to L transformer blocks, each of which contains a multihead attention layer and a multilayer perceptron (MLP) block, the initial input token sequence X0 is expressed as:

where Epos is a position embedding.

In the l-th transformer block, the output feature is denoted as Xl ∈ R(N+1)×D and self-attention mechanism is used to get an attention matrix Al ∈ R(N+1)×(N+1) between tokens, which is formulated as:

where Ql and Kl are the queries and keys obtained by linear operation. Nh is the number of heads and ⊤ is a transpose operator. We extract the attention vector Al∗ ∈ R1×N of the class token from Al and then generate the final attention vector A∗ by:

which aggregates long-range feature dependency from each transformer block.

Differ from ViT, TS-CAM [Gao et al., 2021] and SCM [Bai et al., 2022] use the N patch tokens of the L-th transformer block for classification. The specific process is to reshape them into feature maps F ∈ RD×h×w and to calculate a semantic map S ∈ RC×h×w by a 3×3 convolution layer with C filters, where C is the number of categories. In the following subsection, we introduce a spatial-awareness class attention to preserve the spatial consistency in semantic feature generation process. Finally, semantic map S is fed to a global average pooling (GAP) layer [Lin et al., 2014] and a softmax layer to predict the classification probability p ∈ R1×C . The loss function is defined as:

where pˆ c is the class probability of the correct class ˆ c.

In the localization map generation phase, we first extract a semantic-agnostic map fromA∗ that contains long-distance dependencies and then couple it with S that contains local information to learn the localization map Mc for each class c. The coupling procedure is formulated as:

(5)

where Γ(·) indicates the reshape operator which coverts the vector (R1×N ) to the map (Rh×w ). ⊗ denotes element-wise multiplication operation. The localization mapMc is adjusted to the same size as the original image by linear interpolation. Based on Mc, we model the segmentation foreground and the background as a binary classification problem and determine the threshold value based on Bayesian decision theory for final object bounding box prediction.

3.2 Consistency and IntegrationModule of Shallow-deep Features

CISD computes the spatial-awareness class attention through a nonlinear transform of the consistency between shallow and deep semantic features. In Figure 2, the shallow feature preserving spatial locality is first generated via a simple convolution operation with almost no increase in computational complexity. Then its consistency with the deep semantic feature processed by the multi-layer transformer incorporating global information is calculated for each class channel.

Shallow Feature Extraction. We reshape the token sequence Xl from l-th (l is a small value) transformer block to the token feature map Fl ∈ RD×h×w . Each channel of Fl focuses on different features of input images. And we reag-

gregate these information by one 3 × 3 convolutional layer with 1 filter to generate Rl ∈ R1×h×w , which contains rich

image details and is calculated by:

where F d l denotes the d-th feature map, k ∈ R1×D×3×3 is the convolution kernel, k1,d is a 3 × 3 kernel map indexed by 1 and d, and * is the convolution operator.

Shallow-deep Feature Consistency as Attention. We aim to fully capture channel-wise dependencies and keep spatialsemantic relations across different features with simple convolution operation. To fulfill these objectives, we calculate the similarity vector between Rl and each channel feature map of S as channel attention. Then we adopt a sigmoid activation function to learn a non-mutually-exclusive relationship as opposed to one-hot activation. The calculation procedure is formulated as:

where Ψ denotes the reshape function which converts the 3D vector Rk×h×w to the 2D vector Rk×N (k ∈ {1, C}), and × denotes the matrix multiplication.

Shallow-deep Feature Integration. C ls ∈ RC×1 introduces the consistency information between the shallow features and the class-specific deep features. We use a channelwise multiplication to couple S with C ls as well as remaining the original semantic map. Finally, S can be written as:

where · refers to the channel-wise multiplication. Instead of changing the structure of network or adding additional loss functions, the new semantic map integrates the class-agnostic shallow features that help extract whole object regions and the class-specific deep features that emphasize the discriminative parts. Thus, the shallow-deep feature integration makes the network focus more on features, which contribute to localization effectively.

3.3 Online Adaptive Threshold

Existing methods [Gao et al., 2021; Bai et al., 2022] use the same threshold to segment the localization maps of all input images. However, the objects and background values of localization maps may vary widely in different images. In the experiments, we observe that in some localization maps objects are located incorrectly, whose predictions have less than 50% IoU value with the ground-truth bounding boxes due to inappropriate thresholds. Some localization results are significantly smaller when a higher threshold is used, as presented in Figure 3(a). And the other localization results are significantly larger with a lower threshold in Figure 3(c).

The fixed threshold method (FTM) [Zhang et al., 2018b] is only able to adjust the localization results of one situation above. Thus, we propose an effective online adaptive threshold (OAT) to adaptively raise the thresholds for images with larger predicted localization and lower the thresholds for images with smaller predictions. We intuitively consider following the minimum error thresholding (MET) approach [Kittler and Illingworth, 1986] to compute a theoretical threshold for each localization map.

Let Mc be divided into a foreground set M fg c and a background M bg c . The deep learning-based classification model aims to enhance the semantic map of objects and suppress the semantic map of backgrounds. Thus, it is reasonable to assume that the foreground pixels and the background pixels both follow a Gaussian distribution. Specifically, the probability density function of foreground and background values are p(x) = N(µfg, σ2 fg ) and q(x) = N(µbg, σ2 bg ), where µfg and σfg are the mean and standard deviation in M fg c , µbg and σbg are the mean and standard deviation in M bg c .

Then we define θfg is the foreground pixel ratio and the background pixel ratio is θbg, where | · | is the cardinal number of a set, θfg = |Mfg|/|Mc| and θbg = |Mbg|/|Mc|. The probability of erroneously classifying an object point as a background point is P1, which is defined as:

Similarly, P2 denotes the probability that a background point is misclassified as an object point, which is defined as:

Thus, the total classification error P is the sum of P1 and P2. Our optimization goal is to minimize P. Let ∂P ∂τ = 0, we can get:

where τt is the theoretical threshold to divide the two classes. Equation 11 can be converted to the following quadratic equation through some appropriate operations:

where A = σ2 bg - σ2 fg , B = 2(µbgσ2 fg - µfgσ2 bg ), C = µ2fg σ2 bg - µ2bg σ2 fg + 2σ2 fg σ2 bg ln (σfgθbg/σbgθfg).

Due to the binary classification of foregrounds and backgrounds, we simply let σfg = σbg = σ, i.e., the foregrounds and backgrounds have the same variance. Then the theoretical threshold is defined as:

where the first term τe = (µfg +µbg)/2 is seen as an empirical threshold and the second term τa = (σ2 /∆) ln (θbg/θfg) can be seen as an adjustment. Here ∆ = µfg -µbg and σ2 /∆ is approximately independent from τe.

Let the mean value of τe for all images be τ0. When θbg > θfg, τa is positive (i.e., τe < τt) and τ0 - τe > 0 due to more background pixels. When θbg < θfg, τa is negative (i.e., τe > τt) and τ0 - τe < 0 due to more foreground pixels. It can concluded that τ0-τe and τa have the same sign. Thus we utilize λ(τ0-τe) to approach τa and the theoretical threshold is realized by:

where λ is a hyperparameter to control the adjustment.

4 Experiments

4.1 Experimental Settings

Datasets. We test the performance of our proposed methods on two challenging datasets: CUB-200-2011 [Wah et al., 2011] and ILSVRC [Russakovsky et al., 2015]. CUB-2002011 is a small fine-grained dataset of 200 classes, with 5, 994 images used for training and 5, 794 images used for testing. For ILSVRC, we choose the subset of 1, 000 classes containing about 1.2 million training images and 50, 000 validation images. We train the model using only the image-level labels, and the bounding box labels are only used to evaluate the localization performance of the model.

Evaluation Metrics. Following some representative methods [Russakovsky et al., 2015; Zhou et al., 2016], we adopt Top-1/Top-5 classification accuracy (Top-1/Top-5 Cls.) for classification and adopt Top-1/Top-5 localization accuracy (Top-1/Top-5 Loc.), GT-known localization accuracy (GTknown Loc.) for localization. Specifically, Top-1/Top-5 Cls.

CUB

ILSVRC

Figure 4: Visual localization results comparing with different methods on CUB-200-2011 and ILSVRC datasets. Note that the ground-truth bounding boxes are in red, the predictions are in green, and IoU values (%) are shown in white text.

is right, which means that the Top-1/Top-5 predicted category contains the correct image category label. GT-known Loc. is considered correct if only the predicted bounding boxes have over 50% IoU with at least one of the ground-truth boxes. Top-1/Top-5 Loc. is correct only when Top-1/Top-5 Cls. and GT-known Loc. are both right.

Implementation Details. We adopt SCM [Bai et al., 2022] as our baseline, which uses the Deit-S backbone [Touvron et al., 2021] pretrained on ILSVRC [Russakovsky et al., 2015]. During the training phase, each input image is resized to 256 × 256 pixels and randomly cropped to 224 × 224 pixels. Then we use one 3×3 convolution layer initialized following He's approach [He et al., 2015] instead of MLP head for classification. We choose AdamW [Loshchilov and Hutter, 2019] with ϵ = 1e - 8, β1 = 0.9, β2 = 0.99 and weight decay of 5e-4 to train the model. On CUB-200-2011, we train the model with a learning rate of 5e-5 and a batch size of 256 for 60 epochs. On ILSVRC, the training process lasts 40 epochs with a learning rate of 1e-6 and a batch size of 512.

4.2 Comparison with the State-of-the-Arts

Comparison on ILSVRC. To demonstrate the effectiveness of our methods, we first compare the performance with some state-of-the-art approaches on the more challenging million-level ILSVRC dataset. As reported in Table 1 and Table 2, CIAT achieves both the highest localization and classification accuracy on each evaluation metric, and further highlights the potential of transformers for WSOL. Notably, we surpass recent transformer-based works like CATR [Chen et al., 2023] by large margins of 2.9%, 2.8%, and 1.3% on Top-1 Loc, GT-k. Loc, and Top-1 Cls, respectively.

Comparison on CUB-200-2011. Then, we compare the performance with state-of-the-art methods on CUB-200-2011 dataset. As shown in Table 3, CIAT achieves a remarkable performance of 97.1% on GT-k. Loc and Top-5 Loc of 92.2%, outperforming all transformer-based and CNN-based methods. Moreover, we achieve comparable results with state-ofthe-art studies on Top-1 Loc, with a slightly lower result than LCTR [Chen et al., 2022] and CATR [Chen et al., 2023].

Visualization Comparison. To further demonstrate the effectiveness of CIAT, we visualize the localization results on two datasets in Figure 4. Compared to TS-CAM and SCM, our approach succeeds in exploring more complete range of object regions containing both the weak and high response regions. For instance, in Figure 4(e) and Figure 4(f), our methods force the model to aggregate more detailed features, such as the tails of these animals, while others fail to capture them. Similarly, in Figure 4(a) and Figure 4(b), our methods contribute to filtering out the background noise in complex environment, leading to more accurate localization results.

4.3 Ablation Studies

First, we show the accuracy results with different configurations in Table 4. It is shown that CISD greatly increases the localization performance on the ILSVRC validation set, compared to the baseline method. The results indicate that keeping the semantic-spatial information consistency between shallow features and deep features is important for localization. During the post-processing time, OAT is used to obtain the closer prediction bounding boxes, which further brings an improvement for localization. We get the best results when applying both CISD and OAT.

Then, we explore the design details of each method to max-

𝛌

Figure 5: Performance analyses of hyperparameter λ in OAT.

imize their performance in a series of experiments. Table 5 shows results when extracting features from different blocks. Notably, we observe a downward trend on these metrics when using deeper features. These results indicate that CISD performs best when extracting shallow features from the 0-th block due to its more image details and spatial information. Moreover, to evaluate OAT's performance with other methods, we select TS-CAM [Gao et al., 2021], SCM [Bai et al., 2022], and CIAT to testify OAT. As shown in Table 6, we adopt OAT instead of the commonly used fixed threshold method (FTM) for each method and all get a 0.5% - 0.9% GT-known Loc improvement.

Finally, we perform the sensitivity analysis of hyperparameters through extensive experiments. Our methods are simple with fewer hyperparameters and there is no need to design additional loss functions and balance each loss. Figure 5 further shows the effect of the hyperparameter λ, which is used to control the adjustment. For the same λ, our methods get much higher localization results than TS-CAM and SCM. For a method, a larger or smaller λ may lead to a decline in the results and the performance is worst when λ = 1.

5 Conclusion

In this paper, we propose the Consistency and Integration model with Adaptive Thresholds (CIAT), a novel approach for weakly supervised object localization. CIAT consists of two components, which improve the object activation and the foreground segmentation, respectively. The first component is a consistency and integration module of shallow-deep features (CISD), which significantly brings an improvement for object localization by maintaining semantic-spatial information consistency between shallow features and deep features. The other component is an online adaptive threshold (OAT) that utilizes prior information of each localization map to compute a robust threshold for a more accurate prediction. Extensive experiments demonstrate that our proposed methods effectively improve the performance of representative transformer-based methods for WSOL.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Grant No. 62176271) and the Guangdong Basic and Applied Basic Research Foundation (Grant No. 2024A1515011692).