{"type":"ScholarlyArticle","title":"Strengthening Layer Interaction via Dynamic Layer Attention","authors":[{"name":"Kaishen Wang","slug":"kaishen-wang","given":"Kaishen","family":"Wang","affiliation":"College of Computer Science , Sichuan University"},{"name":"Xun Xia","slug":"xun-xia","given":"Xun","family":"Xia","affiliation":"Clinical Medical College and"},{"name":"Jian Liu","slug":"jian-liu","given":"Jian","family":"Liu","affiliation":"Clinical Medical College and"},{"name":"Zhang Yi","slug":"zhang-yi","given":"Zhang","family":"Yi","affiliation":"College of Computer Science , Sichuan University; The First Affiliated Hospital of Chengdu Medical College"},{"name":"Tao He","slug":"tao-he","given":"Tao","family":"He","affiliation":"College of Computer Science , Sichuan University"}],"abstract":"In recent years, employing layer attention to enhance interaction among hierarchical layers has proven to be a significant advancement in building network structures. In this paper, we delve into the distinction between layer attention and the general attention mechanism, noting that existing layer attention methods achieve layer interaction on fixed feature maps in a static manner. These static layer attention methods limit the ability for context feature extraction among layers. To restore the dynamic context representation capability of the attention mechanism, we propose a Dynamic Layer Attention (DLA) architecture. The DLA comprises dual paths, where the forward path utilizes an improved recurrent neural network block, named Dynamic Sharing Unit (DSU), for context feature extraction. The backward path updates features using these shared context representations. Finally, the attention mechanism is applied to these dynamically refreshed feature maps among layers. Experimental results demonstrate the effectiveness of the proposed DLA architecture, outperforming other state-of-the-art methods in image recognition and object detection tasks. Additionally, the DSU block has been evaluated as an efficient plugin in the proposed DLA architecture. The code is available at https://github.com/tunantu/Dynamic-Layer-attention.","doi":"10.24963/ijcai.2024/561","url":"https://ijcai.org/proceedings/2024/561","pdf":"https://ijcai.org/proceedings/2024/0561.pdf","markdown":"https://ijcai.org/proceedings/2024/561.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/561","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":561,"pages":{"first":5073,"last":5081},"keywords":["Machine Learning: ML: Theory of deep learning","Computer Vision: CV: Representation learning","Machine Learning: ML: Attention models","Computer Vision: CV: Recognition (object detection, categorization)"],"videos":[],"references":[{"position":1,"raw":"[Bello et al., 2019] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V Le. Attention aug- mented convolutional networks. In ICCV, pages 3286- 3295, 2019.","title":"Attention augmented convolutional networks","authors":"Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, Quoc V Le","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cao et al., 2019] Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, and Han Hu. Gcnet: Non-local networks meet squeeze-excitation networks and beyond. In ICCV, pages 0-0, 2019.","title":"Gcnet: Non-local networks meet squeeze-excitation networks and beyond","authors":"Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, Han Hu","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Carion et al., 2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pages 213-229, 2020.","title":"End-to-end object detection with transformers","authors":"Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2018a] Dapeng Chen, Hongsheng Li, Tong Xiao, Shuai Yi, and Xiaogang Wang. Video person re- identification with competitive snippet-similarity aggrega- tion and co-attentive snippet embedding. In CVPR, pages 1169-1178, 2018.","title":"Video person reidentification with competitive snippet-similarity aggregation and co-attentive snippet embedding","authors":"Chen","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2018b] Yunpeng Chen, Yannis Kalantidis, Jian- shu Li, Shuicheng Yan, and Jiashi Feng. Aˆ2-nets: Double attention networks. NeurIPS, 2018.","title":"Aˆ2-nets: Double attention networks","authors":"Chen","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2019] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155, 2019.","title":"Mmdetection: Open mmlab detection toolbox and benchmark","authors":"Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2023] Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xiaokang Yang, and Fisher Yu. Dual aggre- gation transformer for image super-resolution. In ICCV, pages 12312-12321, 2023.","title":"Dual aggregation transformer for image super-resolution","authors":"Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xiaokang Yang, Fisher Yu","year":"2023","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fang et al., 2023] Yanwen Fang, Yuxi Cai, Jintai Chen, Jingyu Zhao, Guangjian Tian, and Guodong Li. Cross- layer retrospective retrieving via layer attention. arXiv preprint arXiv:2302.03985, 2023.","title":"Crosslayer retrospective retrieving via layer attention","authors":"Yanwen Fang, Yuxi Cai, Jintai Chen, Jingyu Zhao, Guangjian Tian, Guodong Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Han et al., 2021] Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, pages 7436-7456, 2021.","title":"Dynamic neural networks: A survey","authors":"Han","year":"2021","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, pages 2961-2969, 2017.","title":"Kaiming He, Georgia Gkioxari","authors":"He","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2019] Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In CVPR, pages 558-567, 2019.","title":"Bag of tricks for image classification with convolutional neural networks","authors":"Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, Mu Li","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2020] Ruining He, Anirudh Ravula, Bhargav Kanagal, and Joshua Ainslie. Realformer: Transformer likes residual attention. arXiv preprint arXiv:2012.11747, 2020.","title":"Ruining He, Anirudh Ravula","authors":"He","year":"2020","venue":"Realformer: Transformer likes residual attention","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, pages 1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hu et al., 2018] Jie Hu, Li Shen, and Gang Sun. Squeeze- and-excitation networks. In CVPR, pages 7132-7141, 2018.","title":"Squeezeand-excitation networks","authors":"Jie Hu, Li Shen, Gang Sun","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Huang et al., 2016] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In ECCV, pages 646-661, 2016.","title":"Deep networks with stochastic depth","authors":"Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, Kilian Q Weinberger","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Huang et al., 2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely con- nected convolutional networks. In CVPR, pages 4700- 4708, 2017.","title":"Densely connected convolutional networks","authors":"Gao Huang, Zhuang Liu, Laurens Van Der Maaten, Kilian Q Weinberger","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Huang et al., 2018] Gao Huang, Shichen Liu, Laurens Van der Maaten, and Kilian Q Weinberger. Condensenet: An efficient densenet using learned group convolutions. In CVPR, pages 2752-2761, 2018.","title":"Condensenet: An efficient densenet using learned group convolutions","authors":"Gao Huang, Shichen Liu, Laurens Van Der Maaten, Kilian Q Weinberger","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Huang et al., 2019] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cc- net: Criss-cross attention for semantic segmentation. In ICCV, pages 603-612, 2019.","title":"Ccnet: Criss-cross attention for semantic segmentation","authors":"Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, Wenyu Liu","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Huang et al., 2020] Zhongzhan Huang, Senwei Liang, Mingfu Liang, and Haizhao Yang. Dianet: Dense-and- implicit attention network. In AAAI, pages 4206-4214, 2020.","title":"Dianet: Dense-andimplicit attention network","authors":"Huang","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Jiao et al., 2023] Jiayu Jiao, Yu-Ming Tang, Kun-Yu Lin, Yipeng Gao, Jinhua Ma, Yaowei Wang, and Wei-Shi Zheng. Dilateformer: Multi-scale dilated transformer for visual recognition. IEEE Transactions on Multimedia, 2023.","title":"Dilateformer: Multi-scale dilated transformer for visual recognition","authors":"Jiayu Jiao, Yu-Ming Tang, Kun-Yu Lin, Yipeng Gao, Jinhua Ma, Yaowei Wang, Wei-Shi Zheng","year":"2023","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Li and Chen, 2020] Duo Li and Qifeng Chen. Deep rein- forced attention learning for quality-aware visual recogni- tion. In ECCV, pages 493-509, 2020.","title":"Deep reinforced attention learning for quality-aware visual recognition","authors":"Chen ; ] Duo Li, Qifeng Li, Chen","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Li et al., 2019] Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. In CVPR, pages 510-519, 2019.","title":"Selective kernel networks","authors":"Xiang Li, Wenhai Wang, Xiaolin Hu, Jian Yang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. NeurIPS, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234-241, 2015.","title":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Srivastava et al., 2015] Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. Training very deep networks. NeurIPS, 2015.","title":"Training very deep networks","authors":"K Rupesh, Klaus Srivastava, Jürgen Greff, Schmidhuber","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2018] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, pages 7794-7803, 2018.","title":"Non-local neural networks","authors":"Xiaolong Wang, Ross Girshick, Abhinav Gupta, Kaiming He","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2020a] Qilong Wang, Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu. Eca- net: Efficient channel attention for deep convolutional neu- ral networks. In CVPR, pages 11534-11542, 2020.","title":"Ecanet: Efficient channel attention for deep convolutional neural networks","authors":"Wang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2020b] Yulin Wang, Kangchen Lv, Rui Huang, Shiji Song, Le Yang, and Gao Huang. Glance and focus: a dynamic approach to reducing spatial redundancy in image classification. NeurIPS, pages 2432-2444, 2020.","title":"Glance and focus: a dynamic approach to reducing spatial redundancy in image classification","authors":"Wang","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Woo et al., 2018] Sanghyun Woo, Jongchan Park, Joon- Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In ECCV, pages 3-19, 2018.","title":"Cbam: Convolutional block attention module","authors":"Sanghyun Woo, Jongchan Park, Joon-Young Lee, In So Kweon","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Xu et al., 2017] Shuangjie Xu, Yu Cheng, Kang Gu, Yang Yang, Shiyu Chang, and Pan Zhou. Jointly attentive spatial-temporal pooling networks for video-based person re-identification. In ICCV, pages 4733-4742, 2017.","title":"Jointly attentive spatial-temporal pooling networks for video-based person re-identification","authors":"Shuangjie Xu, Yu Cheng, Kang Gu, Yang Yang, Shiyu Chang, Pan Zhou","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yang et al., 2020] Le Yang, Yizeng Han, Xi Chen, Shiji Song, Jifeng Dai, and Gao Huang. Resolution adaptive networks for efficient inference. In CVPR, pages 2369- 2378, 2020.","title":"Resolution adaptive networks for efficient inference","authors":"Le Yang, Yizeng Han, Xi Chen, Shiji Song, Jifeng Dai, Gao Huang","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Yang et al., 2021] Le Yang, Haojun Jiang, Ruojin Cai, Yulin Wang, Shiji Song, Gao Huang, and Qi Tian. Condensenet v2: Sparse feature reactivation for deep networks. In CVPR, pages 3569-3578, 2021.","title":"Condensenet v2: Sparse feature reactivation for deep networks","authors":"Le Yang, Haojun Jiang, Ruojin Cai, Yulin Wang, Shiji Song, Gao Huang, Qi Tian","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhao et al., 2021] Jingyu Zhao, Yanwen Fang, and Guodong Li. Recurrence along depth: Deep convolu- tional neural networks with recurrent layer aggregation. NeurIPS, pages 10627-10640, 2021.","title":"Recurrence along depth: Deep convolutional neural networks with recurrent layer aggregation","authors":"Jingyu Zhao, Yanwen Fang, Guodong Li","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhao et al., 2022] Yue Zhao, Junzhou Chen, Zirui Zhang, and Ronghui Zhang. Ba-net: Bridge attention for deep convolutional neural networks. In ECCV, pages 297-312, 2022.","title":"Ba-net: Bridge attention for deep convolutional neural networks","authors":"Yue Zhao, Junzhou Chen, Zirui Zhang, Ronghui Zhang","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}