{"type":"ScholarlyArticle","title":"LeMeViT: Efficient Vision Transformer with Learnable Meta Tokens for Remote Sensing Image Interpretation","authors":[{"name":"Wentao Jiang","slug":"wentao-jiang","given":"Wentao","family":"Jiang","affiliation":"Wuhan University"},{"name":"Jing Zhang","slug":"jing-zhang","given":"Jing","family":"Zhang","affiliation":"The University of Sydney {jiang wentao , wangzengmao , dubo ,"},{"name":"Di Wang","slug":"di-wang","given":"Di","family":"Wang","affiliation":"Wuhan University"},{"name":"Qiming Zhang","slug":"qiming-zhang","given":"Qiming","family":"Zhang","affiliation":"The University of Sydney {jiang wentao , wangzengmao , dubo ,"},{"name":"Zengmao Wang","slug":"zengmao-wang","given":"Zengmao","family":"Wang","affiliation":"Wuhan University"},{"name":"Bo Du","slug":"bo-du","given":"Bo","family":"Du","affiliation":"Wuhan University"}],"abstract":"Due to spatial redundancy in remote sensing images, sparse tokens containing rich information are usually involved in self-attention (SA) to reduce the overall token numbers within the calculation, avoiding the high computational cost issue in Vision Transformers. However, such methods usually obtain sparse tokens by hand-crafted or parallel-unfriendly designs, posing a challenge to reach a better balance between efficiency and performance. Different from them, this paper proposes to use learnable meta tokens to formulate sparse tokens, which effectively learn key information meanwhile improving the inference speed. Technically, the meta tokens are first initialized from image tokens via cross-attention. Then, we propose Dual Cross-Attention (DCA) to promote information exchange between image tokens and meta tokens, where they serve as query and key (value) tokens alternatively in a dual-branch structure, significantly reducing the computational complexity compared to self-attention. By employing DCA in the early stages with dense visual tokens, we obtain the hierarchical architecture LeMeViT with various sizes. Experimental results in classification and dense prediction tasks show that LeMeViT has a significant 1.7 × speedup, fewer parameters, and competitive performance compared to the baseline models, and achieves a better trade-off between efficiency and performance. The code is released at https://github.com/ViTAE-Transformer/LeMeViT.","doi":"10.24963/ijcai.2024/103","url":"https://ijcai.org/proceedings/2024/103","pdf":"https://ijcai.org/proceedings/2024/0103.pdf","markdown":"https://ijcai.org/proceedings/2024/103.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/103","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":103,"pages":{"first":929,"last":937},"keywords":["Computer Vision: CV: Segmentation","Computer Vision: CV: Representation learning","Computer Vision: CV: Recognition (object detection, categorization)"],"videos":[],"references":[{"position":1,"raw":"[Bazi et al., 2021] Yakoub Bazi, Laila Bashmal, Mohamad M Al Rahhal, Reham Al Dayil, and Naif Al Ajlan. Vi- sion transformers for remote sensing image classification. Remote Sensing, 13(3):516, 2021.","title":"Vision transformers for remote sensing image classification","authors":"Bazi","year":"2021","venue":"Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bolya and Hoffman, 2023] Daniel Bolya and Judy Hoff- man. Token merging for fast stable diffusion. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4598-4602, 2023.","title":"Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion","authors":"Hoffman Bolya","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bolya et al., 2022] Daniel Bolya, Cheng-Yang Fu, Xiao- liang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In The Eleventh International Conference on Learning Represen- tations, 2022.","title":"Token merging: Your vit but faster","authors":"Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, Judy Hoffman","year":"2022","venue":"The Eleventh International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Carion et al., 2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.","title":"End-to-end object detection with transformers","authors":"Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko","year":"2020","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2021a] Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi- scale vision transformer for image classification. In Pro- ceedings of the IEEE/CVF international conference on computer vision, pages 357-366, 2021.","title":"Crossvit: Cross-attention multiscale vision transformer for image classification","authors":"Chen","year":"2021","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2021b] Hao Chen, Zipeng Qi, and Zhenwei Shi. Remote sensing image change detection with transform- ers. IEEE Transactions on Geoscience and Remote Sens- ing, 60:1-14, 2021.","title":"Remote sensing image change detection with transformers","authors":"Chen","year":"2021","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2021c] Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end explo- ration. Advances in Neural Information Processing Sys- tems, 34:19974-19988, 2021.","title":"Chasing sparsity in vision transformers: An end-to-end exploration","authors":"Chen","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chiang and Cholak, 2022] David Chiang and Peter Cholak. Overcoming a theoretical limitation of self-attention. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7654-7664, 2022.","title":"Overcoming a theoretical limitation of self-attention","authors":"Cholak ; ] David Chiang, Peter Chiang, Cholak","year":"2022","venue":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Deng et al., 2021] Peifang Deng, Kejie Xu, and Hong Huang. When cnns meet vision transformer: A joint framework for remote sensing scene classification. IEEE Geoscience and Remote Sensing Letters, 19:1-5, 2021.","title":"When cnns meet vision transformer: A joint framework for remote sensing scene classification","authors":"Deng","year":"2021","venue":"IEEE Geoscience and Remote Sensing Letters","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Repre- sentations, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Grainger et al., 2023] Ryan Grainger, Thomas Paniagua, Xi Song, Naresh Cuntoor, Mun Wai Lee, and Tianfu Wu. Paca-vit: learning patch-to-cluster attention in vision transformers. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 18568-18578, 2023.","title":"Paca-vit: learning patch-to-cluster attention in vision transformers","authors":"Ryan Grainger, Thomas Paniagua, Xi Song, Naresh Cuntoor, Mun Wai Lee, Tianfu Wu","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Han et al., 2023] Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5961-5971, 2023.","title":"Flatten transformer: Vision transformer using focused linear attention","authors":"Han","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kong et al., 2022] Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, et al. Spvit: Enabling faster vision transformers via latency-aware soft token pruning. In European conference on computer vision, pages 620- 640. Springer, 2022.","title":"Spvit: Enabling faster vision transformers via latency-aware soft token pruning","authors":"Kong","year":"2022","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lebedev et al., 2018] MA Lebedev, Yu V Vizilter, OV Vy- golov, Vladimir A Knyaz, and A Yu Rubis. Change de- tection in remote sensing images using conditional ad- versarial networks. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 42:565-571, 2018.","title":"Change detection in remote sensing images using conditional adversarial networks","authors":"Lebedev","year":"2018","venue":"The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2023] Yanyu Li, Ju Hu, Yang Wen, Georgios Evan- gelidis, Kamyar Salahi, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Rethinking vision transformers for mo- bilenet size and speed. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16889-16900, 2023.","title":"Rethinking vision transformers for mobilenet size and speed","authors":"Yanyu Li, Ju Hu, Yang Wen, Georgios Evangelidis, Kamyar Salahi, Yanzhi Wang, Sergey Tulyakov, Jian Ren","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF in- ternational conference on computer vision, pages 10012- 10022, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2022] Liu Liu, Zheng Qu, Zhaodong Chen, Feng- bin Tu, Yufei Ding, and Yuan Xie. Dynamic sparse atten- tion for scalable transformer acceleration. IEEE Transac- tions on Computers, 71(12):3165-3178, 2022.","title":"Dynamic sparse attention for scalable transformer acceleration","authors":"Liu Liu, Zheng Qu, Zhaodong Chen, Fengbin Tu, Yufei Ding, Yuan Xie","year":"2022","venue":"IEEE Transactions on Computers","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Long et al., 2021] Yang Long, Gui-Song Xia, Shengyang Li, Wen Yang, Michael Ying Yang, Xiao Xiang Zhu, Liangpei Zhang, and Deren Li. On creating benchmark dataset for aerial image interpretation: Reviews, guid- ances, and million-aid. IEEE Journal of selected topics in applied earth observations and remote sensing, 14:4205- 4230, 2021.","title":"On creating benchmark dataset for aerial image interpretation: Reviews, guidances, and million-aid","authors":"Yang Long, Gui-Song Xia, Shengyang Li, Wen Yang, Michael Ying Yang, Xiao Xiang Zhu, Liangpei Zhang, Deren Li","year":"2021","venue":"IEEE Journal of selected topics in applied earth observations and remote sensing","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mehta and Rastegari, 2021] Sachin Mehta and Mohammad Rastegari. Mobilevit: Light-weight, general-purpose, and mobile-friendly vision transformer. In International Con- ference on Learning Representations, 2021.","title":"Mobilevit: Light-weight, general-purpose, and mobile-friendly vision transformer","authors":"Sachin Mehta, Mohammad Rastegari","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Srinivas et al., 2021] Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16519-16529, 2021.","title":"Bottleneck transformers for visual recognition","authors":"Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, Ashish Vaswani","year":"2021","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tang et al., 2021] Shitao Tang, Jiahui Zhang, Siyu Zhu, and Ping Tan. Quadtree attention for vision transformers. In International Conference on Learning Representations, 2021.","title":"Quadtree attention for vision transformers","authors":"Shitao Tang, Jiahui Zhang, Siyu Zhu, Ping Tan","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Touvron et al., 2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablay- rolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In In- ternational conference on machine learning, pages 10347-10357. PMLR, 2021.","title":"Training data-efficient image transformers & distillation through attention","authors":"Touvron","year":"2021","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tu et al., 2022] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In European conference on computer vision, pages 459-479. Springer, 2022.","title":"Maxvit: Multi-axis vision transformer","authors":"Tu","year":"2022","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2021] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A ver- satile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF international conference on computer vision, pages 568-578, 2021.","title":"Pyramid vision transformer: A versatile backbone for dense prediction without convolutions","authors":"Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao","year":"2021","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2022a] Di Wang, Jing Zhang, Bo Du, Gui- Song Xia, and Dacheng Tao. An empirical study of remote sensing pretraining. IEEE Transactions on Geoscience and Remote Sensing, 2022.","title":"An empirical study of remote sensing pretraining","authors":"Wang","year":"2022","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2022b] Di Wang, Qiming Zhang, Yufei Xu, Jing Zhang, Bo Du, Dacheng Tao, and Liangpei Zhang. Advancing plain vision transformer toward remote sens- ing foundation model. IEEE Transactions on Geoscience and Remote Sensing, 61:1-15, 2022.","title":"Advancing plain vision transformer toward remote sensing foundation model","authors":"Wang","year":"2022","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2022c] Pichao Wang, Xue Wang, Fan Wang, Ming Lin, Shuning Chang, Hao Li, and Rong Jin. Kvt: k- nn attention for boosting vision transformers. In European conference on computer vision, pages 285-302. Springer, 2022.","title":"Kvt: knn attention for boosting vision transformers","authors":"Wang","year":"2022","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2022d] Wei Wang, Chen Tang, Xin Wang, and Bin Zheng. A vit-based multiscale feature fusion approach for remote sensing image segmentation. IEEE Geoscience and Remote Sensing Letters, 19:1-5, 2022.","title":"A vit-based multiscale feature fusion approach for remote sensing image segmentation","authors":"Wang","year":"2022","venue":"IEEE Geoscience and Remote Sensing Letters","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2022e] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):415-424, 2022.","title":"Pvt v2: Improved baselines with pyramid vision transformer","authors":"Wang","year":"2022","venue":"Computational Visual Media","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xia et al., 2018] Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Mar- cello Pelillo, and Liangpei Zhang. Dota: A large-scale dataset for object detection in aerial images. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.","title":"Dota: A large-scale dataset for object detection in aerial images","authors":"Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, Liangpei Zhang","year":"2018","venue":"The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Xia et al., 2022] Zhuofan Xia, Xuran Pan, Shiji Song, Li Er- ran Li, and Gao Huang. Vision transformer with de- formable attention. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 4794-4803, 2022.","title":"Vision transformer with deformable attention","authors":"Xia","year":"2022","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Xiao et al., 2018] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV), pages 418-434, 2018.","title":"Unified perceptual parsing for scene understanding","authors":"Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, Jian Sun","year":"2018","venue":"Proceedings of the European conference on computer vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Xie et al., 2021] Xingxing Xie, Gong Cheng, Jiabao Wang, Xiwen Yao, and Junwei Han. Oriented r-cnn for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3520-3529, 2021.","title":"Oriented r-cnn for object detection","authors":"Xingxing Xie, Gong Cheng, Jiabao Wang, Xiwen Yao, Junwei Han","year":"2021","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Xu et al., 2021] Yufei Xu, Qiming Zhang, Jing Zhang, and Dacheng Tao. Vitae: Vision transformer advanced by ex- ploring intrinsic inductive bias. Advances in neural infor- mation processing systems, 34:28522-28535, 2021.","title":"Vitae: Vision transformer advanced by exploring intrinsic inductive bias","authors":"Yufei Xu, Qiming Zhang, Jing Zhang, Dacheng Tao","year":"2021","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2021] Jianrong Zhang, Hongwei Zhao, and Jiao Li. Trs: Transformers for remote sensing scene clas- sification. Remote Sensing, 13(20):4143, 2021.","title":"Trs: Transformers for remote sensing scene classification","authors":"Jianrong Zhang, Hongwei Zhao, Jiao Li","year":"2021","venue":"Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhang et al., 2022] Qiming Zhang, Yufei Xu, Jing Zhang, and Dacheng Tao. Vsa: Learning varied-size window at- tention in vision transformers. In European conference on computer vision, pages 466-483. Springer, 2022.","title":"Vsa: Learning varied-size window attention in vision transformers","authors":"Qiming Zhang, Yufei Xu, Jing Zhang, Dacheng Tao","year":"2022","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhang et al., 2023] Qiming Zhang, Yufei Xu, Jing Zhang, and Dacheng Tao. Vitaev2: Vision transformer ad- vanced by exploring inductive bias for image recognition and beyond. International Journal of Computer Vision, 131(5):1141-1162, 2023.","title":"Vitaev2: Vision transformer advanced by exploring inductive bias for image recognition and beyond","authors":"Qiming Zhang, Yufei Xu, Jing Zhang, Dacheng Tao","year":"2023","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhang et al., 2024] Qiming Zhang, Jing Zhang, Yufei Xu, and Dacheng Tao. Vision transformer with quadrangle at- tention. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 2024.","title":"Vision transformer with quadrangle attention","authors":"Qiming Zhang, Jing Zhang, Yufei Xu, Dacheng Tao","year":"2024","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhu et al., 2023] Lei Zhu, Xinjiang Wang, Zhanghan Ke, Wayne Zhang, and Rynson WH Lau. Biformer: Vision transformer with bi-level routing attention. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10323-10333, 2023.","title":"Biformer: Vision transformer with bi-level routing attention","authors":"Lei Zhu, Xinjiang Wang, Zhanghan Ke, Wayne Zhang, Rynson Wh Lau","year":"2023","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}