{"type":"ScholarlyArticle","title":"CoAtFormer: Vision Transformer with Composite Attention","authors":[{"name":"Zhiyong Chang","slug":"zhiyong-chang","given":"Zhiyong","family":"Chang","affiliation":"Peking University"},{"name":"Mingjun Yin","slug":"mingjun-yin","given":"Mingjun","family":"Yin","affiliation":"The University of Melbourne"},{"name":"Yan Wang","slug":"yan-wang","given":"Yan","family":"Wang","affiliation":""}],"abstract":"Transformer has recently gained significant attention and achieved state-of-the-art performance in various computer vision applications, including image classification, instance segmentation, and object detection. However, the self-attention mechanism underlying the transformer leads to quadratic\r\ncomputational cost with respect to image size,limiting its widespread adoption in state-of-the-art vision backbones. In this paper we introduce an efficient and effective attention module\r\nwe call Composite Attention. It features parallel branches, enabling the modeling of various global dependencies. In each composite attention module, one branch employs a dynamic channel attention module to capture global channel dependencies, while the other branch utilizes an efficient spatial attention module to extract long-range spatial interactions. In addition, we effectively blending composite attention module with convolutions, and accordingly develop a simple hierarchical vision backbone, dubbed CoAtFormer, by simply repeating the basic building block over multiple stages. Extensive experiments show our CoAtFormer achieves state-of-the-art results on various different tasks. Without any pre-training and extra data, CoAtFormer-Tiny, CoAtFormer-Small, and CoAtFormer-Base achieve 84.4%, 85.3%, and 85.9% top-1 accuracy on ImageNet-1K with 24M, 37M, and 73M parameters, respectively. Furthermore, CoAtFormer also consistently outperform prior work in other vision tasks such as object detection, instance segmentation, and semantic segmentation. When further pretraining on the larger dataset ImageNet-22k, we achieve 88.7% Top-1 accuracy on ImageNet-1K","doi":"10.24963/ijcai.2024/68","url":"https://ijcai.org/proceedings/2024/68","pdf":"https://ijcai.org/proceedings/2024/0068.pdf","markdown":"https://ijcai.org/proceedings/2024/68.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/68","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":68,"pages":{"first":614,"last":622},"keywords":["Machine Learning: ML: Deep learning architectures","Computer Vision: CV: Representation learning"],"videos":[],"references":[{"position":1,"raw":"[Cai and Vasconcelos, 2017] Zhaowei Cai and Nuno Vas- concelos. Cascade r-cnn: Delving into high quality ob- ject detection. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.","title":"Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection","authors":"Vasconcelos Cai","year":"2017","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2017] Yunpeng Chen, Jianan Li, Huaxin Xiao, Xiaojie Jin, Shuicheng Yan, and Jiashi Feng. Dual path networks. In Neural Information Processing Systems, 2017.","title":"Dual path networks","authors":"Yunpeng Chen, Jianan Li, Huaxin Xiao, Xiaojie Jin, Shuicheng Yan, Jiashi Feng","year":"2017","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chu et al., 2021] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, and Chunhua Shen. Conditional positional encodings for vision transformers. In International Con- ference on Learning Representations, 2021.","title":"Conditional positional encodings for vision transformers","authors":"Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Chunhua Shen","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K Li, Li Fei-Fei","year":"2009","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In North American Chapter of the Association for Computational Linguistics, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"North American Chapter of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ding et al., 2022] Mingyu Ding, Bin Xiao, Noel C. F. Codella, Ping Luo, Jingdong Wang, and Lu Yuan. Davit: Dual attention vision transformers. ArXiv, abs/2204.03645, 2022.","title":"Davit: Dual attention vision transformers","authors":"Mingyu Ding, Bin Xiao, C F Noel, Ping Codella, Jingdong Luo, Lu Wang, Yuan","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dong et al., 2021] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.","title":"Cswin transformer: A general vision transformer backbone with cross-shaped windows","authors":"Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, Baining Guo","year":"2021","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv, abs/2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[El-Nouby et al., 2021] Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Ar- mand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Syn- naeve, Jakob Verbeek, and Hervé Jégou. Xcit: Cross- covariance image transformers. In Neural Information Processing Systems, 2021.","title":"Xcit: Crosscovariance image transformers","authors":"El-Nouby","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hatamizadeh et al., 2023] Ali Hatamizadeh, Hongxu Yin, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Global context vision transformers. In International Conference on Machine Learning. PMLR, 2023.","title":"Global context vision transformers","authors":"Ali Hatamizadeh, Hongxu Yin, Greg Heinrich, Jan Kautz, Pavlo Molchanov","year":"2023","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2015] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.","title":"Deep residual learning for image recognition","authors":"X Kaiming He, Shaoqing Zhang, Jian Ren, Sun","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2017a] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask r-cnn. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 42, 2017.","title":"Mask r-cnn","authors":"He","year":"2017","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2017b] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask r-cnn. 2017.","title":"Kaiming He, Georgia Gkioxari","authors":"He","year":"2017","venue":"Mask r-cnn","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hendrycks and Gimpel, 2016] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv: Learn- ing, 2016.","title":"Gaussian error linear units (gelus)","authors":"Gimpel Hendrycks, Dan Hendrycks, Kevin Gimpel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Howard et al., 2017] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mo- bilenets: Efficient convolutional neural networks for mo- bile vision applications. ArXiv, abs/1704.04861, 2017.","title":"Mobilenets: Efficient convolutional neural networks for mobile vision applications","authors":"Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, Hartwig Adam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hu et al., 2017] Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu. Squeeze-and-excitation networks. 2018 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, 2017.","title":"Squeeze-and-excitation networks","authors":"Jie Hu, Li Shen, Samuel Albanie, Gang Sun, Enhua Wu","year":"2017","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. ArXiv, abs/1502.03167, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Jiao et al., 2023] Jiayu Jiao, Yu-Ming Tang, Kun-Yu Lin, Yipeng Gao, Jinhua Ma, Yaowei Wang, and Wei-Shi Zheng. Dilateformer: Multi-scale dilated transformer for visual recognition. IEEE Transaction on Multimedia, 2023.","title":"Dilateformer: Multi-scale dilated transformer for visual recognition","authors":"Jiayu Jiao, Yu-Ming Tang, Kun-Yu Lin, Yipeng Gao, Jinhua Ma, Yaowei Wang, Wei-Shi Zheng","year":"2023","venue":"IEEE Transaction on Multimedia","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kipf and Welling, 2016] Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. ArXiv, 2016.","title":"Semi-supervised classification with graph convolutional networks","authors":"Thomas Kipf, Max Welling","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge J. Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In European Conference on Com- puter Vision, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge J Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence Zitnick","year":"2014","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lin et al., 2023] Wei-Shiang Lin, Ziheng Wu, Jiayu Chen, Jun Huang, and Lianwen Jin. Scale-aware modulation meet transformer. ArXiv, abs/2307.08579, 2023.","title":"Scale-aware modulation meet transformer","authors":"Wei-Shiang Lin, Ziheng Wu, Jiayu Chen, Jun Huang, Lianwen Jin","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. 2021 IEEE/CVF International Confer- ence on Computer Vision (ICCV), 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":"IEEE/CVF International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Liu et al., 2022] Zhuang Liu, Hanzi Mao, Chaozheng Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.","title":"A convnet for the 2020s","authors":"Zhuang Liu, Hanzi Mao, Chaozheng Wu, Christoph Feichtenhofer, Trevor Darrell, Saining Xie","year":"2022","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Qin et al., 2020] Zequn Qin, Pengyi Zhang, Fei Wu, and Xi Li. Fcanet: Frequency channel attention networks. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2020.","title":"Fcanet: Frequency channel attention networks","authors":"Qin","year":"2020","venue":"IEEE/CVF International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross B. Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 39, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sandler et al., 2018] Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottle- necks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.","title":"Mobilenetv2: Inverted residuals and linear bottlenecks","authors":"Mark Sandler, Andrew G Howard, Menglong Zhu, Andrey Zhmoginov, Liang-Chieh Chen","year":"2018","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sun et al., 2017] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Kumar Gupta. Revisiting unreason- able effectiveness of data in deep learning era. 2017 IEEE International Conference on Computer Vision (ICCV), 2017.","title":"Revisiting unreasonable effectiveness of data in deep learning era","authors":"Chen Sun, Abhinav Shrivastava, Saurabh Singh, Abhinav Kumar Gupta","year":"2017","venue":"IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Szegedy et al., 2014] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, D. Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E Reed, D Dragomir Anguelov, Vincent Erhan, Andrew Vanhoucke, Rabinovich","year":"2014","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Szegedy et al., 2015] Christian Szegedy, Vincent Van- houcke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.","title":"Rethinking the inception architecture for computer vision","authors":"Szegedy","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Tan and Le, 2019] Mingxing Tan and Quoc V. Le. Efficient- net: Rethinking model scaling for convolutional neural networks. ArXiv, abs/1905.11946, 2019.","title":"Efficientnet: Rethinking model scaling for convolutional neural networks","authors":"Mingxing Tan, V Quoc, Le","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Touvron et al., 2020] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablay- rolles, and Herv'e J'egou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, 2020.","title":"Training data-efficient image transformers & distillation through attention","authors":"Touvron","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Tu et al., 2022] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Conrad Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In European Conference on Computer Vision, 2022.","title":"Maxvit: Multi-axis vision transformer","authors":"Tu","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Information Processing Systems, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam M Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wang et al., 2017] X. Wang, Ross B. Girshick, Abhi- nav Kumar Gupta, and Kaiming He. Non-local neural net- works. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.","title":"Non-local neural networks","authors":"Wang","year":"2017","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Wang et al., 2021] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A ver- satile backbone for dense prediction without convolutions. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021.","title":"Pyramid vision transformer: A versatile backbone for dense prediction without convolutions","authors":"Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao","year":"2021","venue":"IEEE/CVF International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Wang et al., 2023] Wenxiao Wang, Wei Chen, Qibo Qiu, Long Chen, Boxi Wu, Binbin Lin, Xiaofei He, and Wei Liu. Crossformer++: A versatile vision transformer hing- ing on cross-scale attention. IEEE Transactions on Pattern Analysis and Machine Intelligence, TPAMI, 2023.","title":"Crossformer++: A versatile vision transformer hinging on cross-scale attention","authors":"Wenxiao Wang, Wei Chen, Qibo Qiu, Long Chen, Boxi Wu, Binbin Lin, Xiaofei He, Wei Liu","year":"2023","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Woo et al., 2018] Sanghyun Woo, Jongchan Park, Joon- Young Lee, and In-So Kweon. Cbam: Convolutional block attention module. In European Conference on Computer Vision, 2018.","title":"Cbam: Convolutional block attention module","authors":"Sanghyun Woo, Jongchan Park, Joon-Young Lee, In-So Kweon","year":"2018","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Xiao et al., 2018] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. ArXiv, abs/1807.10221, 2018.","title":"Unified perceptual parsing for scene understanding","authors":"Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, Jian Sun","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Yang et al., 2021] Huanrui Yang, Hongxu Yin, Pavlo Molchanov, Hai Li, and Jan Kautz. Nvit: Vision trans- former compression and parameter redistribution. ArXiv, abs/2110.04869, 2021.","title":"Nvit: Vision transformer compression and parameter redistribution","authors":"Yang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Yang et al., 2022a] Chenglin Yang, Siyuan Qiao, Qihang Yu, Xiaoding Yuan, Yukun Zhu, Alan Loddon Yuille, Hartwig Adam, and Liang-Chieh Chen. Moat: Alternat- ing mobile convolution and attention brings strong vision models. ArXiv, abs/2210.01820, 2022.","title":"Moat: Alternating mobile convolution and attention brings strong vision models","authors":"Yang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Yang et al., 2022b] Jianwei Yang, Chunyuan Li, Xiyang Dai, and Jianfeng Gao. Focal modulation networks, 2022.","title":"Focal modulation networks","authors":"Yang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Yang et al., 2023] Chenhongyi Yang, Jiarui Xu, Shalini De Mello, Elliot J. Crowley, and Xiaolong Wang. GPViT: A High Resolution Non-Hierarchical Vision Transformer with Group Propagation. 2023.","title":"GPViT: A High Resolution Non-Hierarchical Vision Transformer with Group Propagation","authors":"Chenhongyi Yang, Jiarui Xu, Shalini De Mello, Elliot J Crowley, Xiaolong Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Yao et al., 2023] Ting Yao, Yehao Li, Yingwei Pan, Yu Wang, Xiao-Ping Zhang, and Tao Mei. Dual vision transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10870-10882, 2023.","title":"Dual vision transformer","authors":"Ting Yao, Yehao Li, Yingwei Pan, Yu Wang, Xiao-Ping Zhang, Tao Mei","year":"2023","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Zhang et al., 2023] Xiaosong Zhang, Yunjie Tian, Lingxi Xie, Wei Huang, Qi Dai, Qixiang Ye, and Qi Tian. Hivit: A simpler and more efficient design of hierarchical vi- sion transformer. In International Conference on Learning Representations, 2023.","title":"Hivit: A simpler and more efficient design of hierarchical vision transformer","authors":"Xiaosong Zhang, Yunjie Tian, Lingxi Xie, Wei Huang, Qi Dai, Qixiang Ye, Qi Tian","year":"2023","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Zheng et al., 2020] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H. S. Torr, and Li Zhang. Rethinking semantic segmentation from a sequence-to- sequence perspective with transformers. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.","title":"Rethinking semantic segmentation from a sequence-tosequence perspective with transformers","authors":"Zheng","year":"2020","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Zhou et al., 2017] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.","title":"Scene parsing through ade20k dataset","authors":"Zhou","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Zhu et al., 2020] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. ArXiv, abs/2010.04159, 2020.","title":"Deformable detr: Deformable transformers for end-to-end object detection","authors":"Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, Jifeng Dai","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}