Strengthening Layer Interaction via Dynamic Layer Attention

1 Introduction

Numerous studies have highlighted the significance of enhancing interaction among hierarchical layers in Deep Convolutional Neural Networks (DCNNs), which have made substantial progress across various tasks. For instance, ResNet [He et al., 2016] introduced a straightforward and highly effective implementation by incorporating skip connections between two consecutive layers. DenseNet [Huang et al., 2017] further improved inter-layer interaction by recycling information from all previous layers. Meanwhile, attention mechanisms are playing an increasingly crucial role in DCNNs. The

∗Corresponding author.

(a) Static Layer Attention

(b) Dynamic Layer Attention

Figure 1: The comparison between static and dynamic layer attention architecture.

evolution of attention mechanisms in DCNNs has progressed through several stages, including channel attention ([Hu et al., 2018], [Wang et al., 2020a]), spatial attention ([Carion et al., 2020], [Wang et al., 2018]), branch attention ([Srivastava et al., 2015], [Li et al., 2019]), and temporal attention ([Xu et al., 2017], [Chen et al., 2018a]).

Recently, attention mechanisms have been successfully applied to another direction, (e.g., DIANet [Huang et al., 2020], RLANet [Zhao et al., 2021], MRLA [Fang et al., 2023]), indicating the feasibility of strengthening interaction among layers through attention mechanisms. Compared to simple interaction like those in ResNet and DenseNet, the introduction of attention mechanisms makes the interaction among layers more closely effective. DIANet employed a parameter-shared LSTM module along the network's depth to facilitate interaction among layers. RLANet proposed a layer aggregation structure to reuse features of previous layers for enhancing layer interaction. MRLA first introduced the concept of layer attention, treating each feature as a token to learn useful information from others by attention mechanisms.

However, we have identified a common drawback in existing layer attention mechanisms: they are applied in a static manner, limiting inter-layer information interaction. In channel and spatial attention, for the input x ∈ RC×H×W , tokens are input to the attention module, all of which are generated from x concurrently. However, in existing layer attention,

𝑶𝒍

Layer Attention Module

𝑶𝒍

• (a) Static Layer Attention
• (b) Dynamic Layer Attention

Figure 2: Visualization of attention scores from stage 3 of the ResNet-56 between static and the proposed dynamic layer attention on CIFAR-100 dataset.

features generated from different time steps are treated as tokens and passed into the attention module, as shown in Figure 1(a). Since tokens generated earlier do not change once produced, the input tokens are relatively static, leading to a reduction in information interaction between the current layer and previous layers. Figure 2(a) visualizes the MRLA attention scores from stage 3 of ResNet-56 trained on CIFAR-100. When the first 5 layers reuse information from previous layers through static layer attention, the key values from only one specific layer are activated, with almost zero attention assigned to other layers. This observation verifies that static layer attention compromises the efficiency of information interaction among layers.

To solve the static problem of layer attention, we propose a novel Dynamic Layer Attention (DLA) architecture to improve information flow among layers, where the information of previous layers can be dynamically modified during the feature interaction. As shown in Figure 2(b), during the reutilization of information from preceding layers, the attention of the current feature undergo a shift from exclusively focusing on a particular layer to incorporating information from various layers. DLA facilitates a more thorough exploitation of information, enhancing inter-layer information interaction efficiency. Experimental results demonstrate the effectiveness of the proposed DLA architecture, outperforming other stateof-the-art methods in image recognition and object detection tasks. The contributions of this paper are summarized as follows:

• We propose a novel DLA architecture, which consists of dual paths, where the forward path extracts context feature among layers using a Recurrent Neural Network (RNN) and the backward path refreshes the original feature at each layer using these shared context representation.
• A novel RNN block, named Dynamic Sharing Unit (DSU), is proposed to be a suitable component for DLA. It effectively promotes the dynamic modification of information within DLA and demonstrates commendable performance in the layer-wise information integration as well.

2 RelatedWork

Layer Interaction. Layer interaction has consistently been an intriguing aspect of DCNNs. Initially, the implementation of layer interaction was relatively simple. ResNet [He et al., 2016] introduced skip connections between consecutive layers, mitigating issues of gradient vanishing and exploding to some extent. DenseNet [Huang et al., 2017] further enhanced layer interaction by reusing information generated from all preceding layers. In U-Net [Ronneberger et al., 2015], a commonly utilized architecture in medical segmentation, the encoder and decoder are connected through skip connections to improve feature extraction and achieve higher accuracy.

Recent studies have explored effective methods for implementing layer interaction. DREAL [Li and Chen, 2020] optimized parameters by introducing arbitrary attention modules and employed a Long Short-Term Memory (LSTM) [Hochreiter and Schmidhuber, 1997] to incorporate previous attention weights. The update of parameters for both LSTM and attention layers was achieved through deep reinforcement learning. DIANet [Huang et al., 2020] incorporated a LSTM module at the layer level, constructing a DIA block shared by all layers in the entire network to facilitate inter-layer interaction. RealFormer [He et al., 2020] integrated residual connections between adjacent attention modules, adding the attention scores from the previous layer to the current one. RLANet [Zhao et al., 2021] introduced a lightweight recurrent layer aggregation module to describe how information from previous layers can be efficiently reused for better feature extraction in the current layer. [Zhao et al., 2022] proposed a straightforward and versatile approach to strike a balance between effectively utilizing neural network information and maintaining high computational efficiency. By seamlessly integrating features from preceding layers, it foster effective interaction of information. MRLA [Fang et al., 2023] treated the features of each layer as tokens, enabling interaction among different hierarchical layers through an attention mechanism, further strengthening the interaction among layers.

Dynamic Network Architecture. Dynamic networks represent a type of neural network structure whose topology or parameters can dynamically change during runtime, providing the network with considerable flexibility or other benefits. [Wang et al., 2020b] proposed a universally adaptable inference framework for the majority of DCNNs, with costs that can be easily dynamically adjusted without additional training. RANet [Yang et al., 2020] introduced an adaptive network that could effectively reduce the spatial redundancy involved in inferring high-resolution inputs. [Han et al., 2021] have shown that dynamic neural networks can enhance the efficiency of deep networks. CondenseNetv2 [Yang et al., 2021] utilized a Sparse Feature Reactivation (SFR) module to reactivate pruned features from CondenseNet [Huang et al., 2018], thereby enhancing feature utilization efficiency. To the best of our knowledge, this paper is the first attempt to build a dynamic network architecture for strengthening layer interaction.

3 Dynamic Layer Attention

We will start by reconsidering the current layer attention architecture and elucidating its static nature. Subsequently, we will introduce the Dynamic Layer Attention (DLA). Finally, tanh

X



Linear

we will present an enhanced RNN plugin block named the Dynamic Sharing Unit (DSU), integrated within the DLA architecture.

3.1 Rethinking Layer Attention

Layer attention was recently defined by [Fang et al., 2023] and is illustrated in Figure 1(a), where the attention mechanism enhances layer interaction. [Fang et al., 2023] focused on reducing the computational cost of layer attention and proposed the Recurrent Layer Attention (RLA) architecture. In RLA, features from distinct layers are treated as tokens and undergo computations, ultimately producing attention output. Let the feature output of the l-th layer be xl ∈ RC×W×H . The vectors Ql , Kl , and V l can be calculated as follows:

where fq is a mapping function to extract information from the l-th layer, while fk and fv are corresponding mapping functions intended to extract information from the 1st to lth layers, respectively. The attention output ol is given as follows:

where Dk serves as the scaling factor. A lightweight version of RLA was proposed to recurrently update the attention output ol as follows:

where λlo is a learnable vector and ⊙ indicates the elementwise multiplication. With the multi-head structure design, Multi-head RLA (MRLA) is introduced.

3.2 Motivation

MRLA successfully integrated the attention mechanism to enhance layer interaction, effectively addressing computational costs. However, when MRLA is applied in the l-th layer, a preceding feature output xm (m < l) has already been generated in the m-th layer, with no subsequent changes. Consequently, the information processed by MRLA comprises fixed features from the previous layers. In contrast, widely used attention-based models, such as channel attention [Hu et al., 2018; Wang et al., 2020a], spatial attention [Carion et al., 2020; Huang et al., 2019], and Transformers [Vaswani et al., 2017; Liu et al., 2021; Chen et al., 2023; Jiao et al., 2023], pass tokens into the attention module generated simultaneously. Applying the attention module between freshly generated tokens ensures that each token consistently learns up-to-date features. Therefore, we categorize MRLA as a static layer attention mechanism, limiting interaction between the current layer and shallower layers.

In a general self-attention mechanism, the feature xm serves two purposes: conveying essential information and representing context. The essential information extracted at the current layer distinguishes it from that at other layers. Meanwhile, the context representation captures changes and evolution of features along the temporal axis, a critical aspect in determining feature freshness. In the general attention mechanism, essential information is generated at each layer, and the context representation is transferred to the next layer for calculating the attention output. In contrast, in layer attention, once tokens are generated, attention is calculated with a fixed context representation, diminishing the efficiency of the attention mechanism. Therefore, this paper aims to establish a novel method to restore the context representation, ensuring that the information fed into the layer attention is consistently dynamic.

3.3 Dynamic Layer Attention Architecture

To address the static issue of MRLA, we propose the use of a dynamic updating rule to extract the context representation and promptly update features at previous layers, resulting in a Dynamic Layer Attention (DLA) architecture. As illustrated in Figure 1(b), DLA consists of dual paths: forward and backward. In the forward path, a Recurrent Neural Network (RNN) is employed for context feature extraction. Let the RNN block be denoted as "Dyn", and the initial context as c0 , respectively. c0 is randomly initialized. Given an input xm ∈ RC×W×H where m < l, a Global Average Pooling (GAP) is applied to extract global features at m-th layer as follows:

𝒄𝒎

𝒉𝒎



X



Linear

𝒄𝒎

𝒉𝒎

The context representation is extracted as follows:

where θl represents the shared trainable parameters of "Dyn". Once the context cl is calculated, the features of each layer are simultaneously updated in the backward path as follows:

Referring to Equation (5), the forward context feature extraction is a step-by-step process with a computation complexity of O(n). Meanwhile, the feature updating in Equation (6) can be performed in parallel, resulting in a computation complexity of O(1). After updating xm, the basic version of DLA use Equation (2) to compute the layer attention, abbreviated as DLA-B. For the lightweight version of DLA, Simply update ol-1 and then use Equation (3) to obtain DLA-L.

Computation Efficiency. DLA possesses several advantages in its structural design. Firstly, global information is condensed to compute context information, a utility that has been validated in [Huang et al., 2020]. Secondly, DLA employs shared parameters within a RNN block. Thirdly, the context cl is separately fed into the feature maps at each layer in parallel. Both the forward and backward paths share the same parameters throughout the entire network. Finally, we introduce an efficient RNN block for calculating context representation, which will be elucidated in the following subsection. With these efficiently designed structural rules, the computation cost and network capacity are guaranteed.

3.4 Dynamic Sharing Unit

LSTM, as depicted in Figure 2(a), is designed for processing sequential data and learning temporal features, enabling it to capture and store information over long sequences. However, the fully connected linear transformation in LSTM significantly increases the network capacity when embedding LSTM as the recurrent block in DLA. To mitigate this capacity increase, a variant LSTM block named the DIA unit was proposed by [Huang et al., 2020], as illustrated in Figure 2(b). Before feeding data into the network, DIA first utilizes a linear transformation followed by a ReLU activation function to reduce the input dimension. Additionally, DIA replaces the Tanh function with a Sigmoid function at the output layer.

LSTM and DIA generate two outputs, comprising a hidden vector hm and a cell state vector cm. Typically, hm is used as the output vector, and cm serves as the memory vector. DLA is exclusively focused on extracting context characteristics from different layers, where the RNN block has no duty to transform its internal state feature to the outside. Consequently, we discard the output gate and merge the memory and hidden vector by omitting the hm symbol. The proposed simplified RNN block is named Dynamic Sharing Unit (DSU). The workflow of the DSU is illustrated in Figure 2(c). Specifically, before adding cm-1 and ym, we first normalize cm-1 using an activation function σ(·). Here, we opt for the Sigmoid function (σ(z) = 1/(1+e-z )). Therefore, the input to DSU was compressed as follows:

The hidden transformation, the input gate, and the forget gate can be represented by the following formula:

Subsequently, we obtain

To decrease the network parameters, let W1 ∈ R Cr ×2C and W2 ∈ RC× Cr , where r is the reduction ratio. DSU reduces the parameters to 5C 2 /r, which is fewer than 8C 2 of LSTM and 10C 2 /r of DIA.

4 Experiments

4.1 Image Classification

Experimental Setup. We conducted experiments on the CIFAR-10, CIFAR-100, and ImageNet-1K datasets using ResNets [He et al., 2016] as the backbone network for image classification. For the CIFAR-10 and CIFAR-100 datasets, we employed standard data augmentation strategies [Huang et al., 2016]. The training process involved random horizontal flipping of images, padding each side by 4 pixels, and then randomly cropping to 32×32. Normalization with mean and standard deviation adjustment was implemented, and training hyperparameters such as batch size, initial learning rate, and weight decay followed the recommendations of the original ResNets [He et al., 2016]. To address hyperparameter uncertainty, we conducted five runs of experiments. For the ImageNet-1K dataset, we adopted the same data augmentation strategy and hyperparameter settings outlined in [He et al., 2016] and [He et al., 2019]. During training, images were randomly cropped to 224×224 with horizontal flipping. In the testing phase, images were resized to 256×256, then centrally cropped to a final size of 224×224. The optimization process utilized an SGD optimizer with a momentum of 0.9 and weight decay of 1e-4. The initial learning rate was set to 0.1 and decreased according to the MultiStepLR schedule over 100 epochs for a batch size of 256, consistent with the ResNet approach [He et al., 2016]. Meanwhile, the reduction ratio r was set to 4 for the CIFAR-10 and CIFAR-100 datasets, and 20 for the ImageNet-1K dataset, consistent with the settings in DIANet [Huang et al., 2020].

Results on CIFAR. The experimental results, showcasing Accuracy±Std, are presented in Table 1. These results underscore the significant superiority of our DLA-B and DLA-L models over other challenging networks, including SE [Hu et al., 2018], ECA [Wang et al., 2020a], DIANet [Huang et al., 2020], MRLA [Fang et al., 2023] on the CIFAR-10 and CIFAR-100 datasets. In comparison with the baselines, DLA-L's top-1 accuracy on CIFAR-10 surpasses ResNets by 1.32%, 1.60%, and 1.62%, and even outperforms them on CIFAR-100 by 4.96%, 2.94%, and 3.41%. Furthermore, both DLA-B and DLA-L outperform MRLA-B and MRLA-L, which are typical static layer attention models. It is noteworthy that DLA-L-20 (embedding ResNet-20) achieves fewer parameters than ResNet-56 while maintaining comparable top-1 accuracy on the CIFAR-10 (92.96% vs. 92.95%) and CIFAR-100 (72.26% vs. 72.32%) datasets. Furthermore, DLA-L-56 outperforms ResNet-110 by 1.51% and 2.46% on CIFAR-10 and CIFAR-100 datasets, respectively. As depicted in Figure 4, ResNet, DIANet, and MRLA-L exhibit a relatively slow growth in capabilities with increasing network depth. In contrast, the proposed DLA demonstrates a faster increase in test accuracy on the CIFAR-10 and CIFAR100 datasets as the network depth increases. This observation verifies that strengthening layer interaction through DLA is more beneficial in a deeper network structure.

(a) Testing acc. on CIFAR-10 (b) Testing acc. on CIFAR-100

Figure 4: Comparison of testing accuracy of different models with different depths.

Results on ImageNet-1K. We compare the DLA-L architecture with other challenging methods using ResNets as baselines. Experimental results, shown in Table 2, indicate that our model significantly outperforms other models. Firstly, our DLA-L exhibits an increase of 1.9% and 1.5% in top-1 accuracy to ResNet-50 and ResNet-101, respectively.

Then, when compared to the channel attention method, DLAL-50 has 0.9M fewer parameters than SE-50 and CBAM-50 [Woo et al., 2018], but its top-1 accuracy is higher by 1.3% and 0.7%. DLA-L-101 has 1.5M fewer parameters than SE101 and CBAM-101 [Woo et al., 2018] while achieving a 1.3% and 0.4% top-1 accuracy increase, respectively. Meanwhile, when compared to the lightweight model, DLA-L-50 and DLA-L-101 remains a 0.5% and 0.2% higher top-1 accuracy than ECA-50 and ECA-101. DLA-L also outperforms other popular layer interaction models. With 1.2M fewer parameters than DIANet-50 , DLA-L-50 achieves a 0.8% higher top-1 accuracy than DIANet-50. Although DLA-L introduces more parameters compared to RLAg [Zhao et al., 2021], it achieves a 0.8% and 0.4% higher top-1 accuracy than RLAg50 and RLAg-101, respectively. Additionally, DLA-L also surpasses recent models such as A2 [Chen et al., 2018b], AA [Bello et al., 2019], and GC [Cao et al., 2019]. Finally, in comparison with static layer attention, DLA-L exhibits an increase of 0.3% and 0.2% in top-1 accuracy to MRLA-L50 and MRLA-L-101 with a tolerable increase in the number of parameters. In summary, through comparisons with well-known channel attention models, layer interaction models, and various other challenging models, we have validated that our proposed DLA architecture serves as a more effective layer interaction model, outperforming other models in the domain of image classification.

4.2 Object Detection

Experimental Setup. In the context of object detection, our approach was evaluated on the COCO2017 dataset using the Faster R-CNN [Ren et al., 2015] and Mask R-CNN [He et al., 2017] frameworks as detectors. The implementations of all detectors were carried out using the MMDetection toolkit [Chen et al., 2019], following the default settings. In the preprocessing stage, the shorter side of input images was resized to 800 pixels. The optimization process employed SGD with a weight decay of 1e-4, momentum of 0.9, and a batch size of 8. The models underwent training for a total of 12 epochs, starting with an initial learning rate of 0.01. Learning rate adjustments occurred at the 8th and 11th epochs, with a reduction by a factor of 10 each time.

Results on COCO2017. As shown in Table 3, when utilizing Faster R-CNN as the detectors, our proposed DLAL demonstrates a remarkable improvement in average precision (AP) of 4.2% and 3.6% on ResNet-50 [He et al., 2016] and ResNet-101 [He et al., 2016], respectively, which validates the outstanding capability of the DLA architecture in enhancing object detection performance. Notably, DLA-L outperforms other models, when compared to the challenging channel attention block, DLA-L-50 exhibits similar parameter counts to SE-50 [Hu et al., 2018] but achieves a higher AP by 2.9% and a 3.3% improvement on AP75. When compared to representative layer interaction models, DLA-L continues to exhibit excellence. Despite introducing more parameters compared to RLAg [Zhao et al., 2021], there is an excellent increase in AP by 1.8% and 1.1% on DLA-L-50 and DLAL-101, respectively. Meanwhile, in contrast to a static layer attention model, MRLA-L [Fang et al., 2023], our DLA-L achieves a respective increase of 0.2% and 0.3% in AP. When utilizing Mask R-CNN as the detector, our DLA-L also outperforms the aforementioned models. Additionally, it surpasses NL [Wang et al., 2018], GC [Cao et al., 2019], BA [Zhao et al., 2022], and other models, showcasing the remarkable potential of the DLA architecture in facilitating dynamic modification in inter-layer information, even with the introduction of a tolerable parameter increment.

5 Ablation Study

5.1 Evaluating Different RNN Blocks in DLA-L

In order to validate the effectiveness of our proposed DSU block in implementing dynamic layer attention, we incorporated the original RNN block, DIA block and LSTM block as plugins into DLA to replace DSU block for comparative experiments. Due to limitations in computational resources, our ablation experiments were conducted using ResNet-110 as the baseline on CIFAR-100. Each experiment was run five times, and the results were expressed in the form of Accuracy ± Std.

As illustrated in Table 4, the dynamic layer attention implemented by the four RNN blocks exhibits varying performance. The original RNN blocks show the poorest performance, with even a little top-1 accuracy increase than the baseline. LSTM block and the DIA block demonstrate comparable performance, outperforming baseline by 2.32% and 2.60%, respectively. However, the LSTM block has an additional parameter count of 0.47M compared to the DIA block, making it impractical applying LSTM in DLA. On the other hand, our proposed DSU block, utilizing even fewer parameters, achieves superior results. In comparison to the DIA block, we have 0.07M fewer parameters, leading to a 0.81% improvement in top-1 accuracy. Therefore, it can be inferred that our proposed DSU is the most effective block among existing RNN blocks for implementing DLA.

5.2 Evaluating the Introduced σ Function in DSU

Compared to other RNN blocks, our DSU block first normalizes cm-1 using an activation function, where we employ the Sigmoid function σ(z) = 1 1+e-z for this purpose. To contrast the role of the σ function utilized in DSU, we will substitute it with various activation functions, including Identity mapping, Tanh, and ReLU functions.

As shown in Table 5, we conducted experiments on the CIFAR-100 dataset using DLA-L-56 and DLA-L-110. Firstly, we observed in DSU that normalizing cm-1 is essential compared to Identity Mapping. The inclusion of an activation function resulted in significantly higher top-1 accuracy compared to Identity mapping. Secondly, among various activation functions, the Sigmoid function has been proven effective in scaling cm-1 to the range of (0, 1), facilitating better fusion with ym as input for the DSU block. Experimental results confirm this observation, achieving impressive top-1 accuracy of 75.46% and 76.41% on DLA-L-56 and DLA-L110, respectively, far surpassing the baseline.

5.3 Evaluating DSU Block in Layer-wise Information Integration

We also attempt to evaluate that whether the proposed DSU block could be deployed to other layer interaction methods. [Huang et al., 2020] introduced a Layer-wise Information Integration (LII) architecture that shared a RNN block throughout different network layers. And DIANet serves as a simple and effective form of LII architecture, demonstrating notable achievements in image classification. We evaluated the performance of the proposed DSU, LSTM, and DIA blocks, when integrating them into the LII. Additionally, we employed ResNets as the backbone and conduct experimental comparisons on the CIFAR-100 dataset.

As shown in Table 6, the experimental results demonstrate that the RNN blocks exhibits favorable performance when applied to LII. On the CIFAR-100 dataset, our DSU block shows improvements of 1.91% and 2.14% on ResNet-56 and ResNet-110, respectively. While the introduced parameters are 0.04M smaller than DIA block, DSU block outperforms DIA block by 0.36% on ResNet-56. Meanwhile, DSU block performs comparably with DIA block on ResNet-110. On the contrary, LSTM block has the highest number of parameters but performs the worst. Therefore, it could be concluded that DSU block is also a challenging method in LII, which could achieve comparable results to DIA block in LII architecture while having fewer parameters.

6 Conclusion

In this paper, we first unveil the inherent static nature of existing layer attention mechanisms and analyze their limitations in facilitating feature extraction through layer interaction. To address these limitations and restore the dynamic features of attention mechanisms, we propose and construct a framework called Dynamic Layer Attention (DLA). For implementing DLA, we design a novel RNN block, named Dynamic Sharing Unit (DSU). Experimental evaluations in the domains of image classification and object detection demonstrate that our framework outperforms static layer attention significantly in promoting layer interaction.

Acknowledgments

This work was supported by the National Major Science and Technology Projects of China under Grant 2018AAA0100201, the National Natural Science Foundation of China under Grant 62206189, and the China Postdoctoral Science Foundation under Grant 2023M732427.