{"type":"ScholarlyArticle","title":"RaMLP: Vision MLP via Region-aware Mixing","authors":[{"name":"Shenqi Lai","slug":"shenqi-lai","given":"Shenqi","family":"Lai","affiliation":""},{"name":"Xi Du","slug":"xi-du","given":"Xi","family":"Du","affiliation":"Kiwi Tech"},{"name":"Jia Guo","slug":"jia-guo","given":"Jia","family":"Guo","affiliation":""},{"name":"Kaipeng Zhang","slug":"kaipeng-zhang","given":"Kaipeng","family":"Zhang","affiliation":"Shanghai AI Laboratory"}],"abstract":"Recently, MLP-based architectures achieved impressive results in image classification against CNNs and ViTs. However, there is an obvious limitation in that their parameters are related to image sizes, allowing them to process only fixed image sizes. Therefore, they cannot directly adapt dense prediction tasks (e.g., object detection and semantic segmentation) where images are of various sizes. Recent methods tried to address it but brought two new problems, long-range dependencies or important visual cues are ignored. This paper presents a new MLP-based architecture, Region-aware MLP (RaMLP), to satisfy various vision tasks and address the above three problems. In particular, we propose a well-designed module, Region-aware Mixing (RaM). RaM captures important local information and further aggregates these important visual clues. Based on RaM, RaMLP achieves a global receptive field even in one block. It is worth noting that, unlike most existing MLP-based architectures that adopt the same spatial weights to all samples, RaM is region-aware and adaptively determines weights to extract region-level features better. Impressively, our RaMLP outperforms state-of-the-art ViTs, CNNs, and MLPs on both ImageNet-1K image classification and downstream dense prediction tasks, including MS-COCO object detection, MS-COCO instance segmentation, and ADE20K semantic segmentation. In particular, RaMLP outperforms MLPs by a large margin (around 1.5% Apb or 1.0% mIoU) on dense prediction tasks. The training code could be found at https://github.com/xiaolai-sqlai/RaMLP.","doi":"10.24963/ijcai.2023/111","url":"https://ijcai.org/proceedings/2023/111","pdf":"https://ijcai.org/proceedings/2023/0111.pdf","markdown":"https://ijcai.org/proceedings/2023/111.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/111","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":111,"pages":{"first":999,"last":1007},"keywords":["Computer Vision: CV: Recognition (object detection, categorization)","Computer Vision: CV: Representation learning"],"videos":[],"references":[{"position":1,"raw":"[Brown et al., 2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Win- ter, Christopher Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few- shot learners. In NeurIPS, 2020.","title":"Language models are fewshot learners","authors":"B Tom, Benjamin Brown, Nick Mann, Melanie Ryder, Jared Subbiah, Prafulla Kaplan, Arvind Dhariwal, Pranav Neelakantan, Girish Shyam, Amanda Sastry, Sandhini Askell, Ariel Agarwal, Gretchen Herbert-Voss, Tom Krueger, Rewon Henighan, Aditya Child, Daniel M Ramesh, Jeffrey Ziegler, Clemens Wu, Christopher Winter, Mark Hesse, Eric Chen, Mateusz Sigler, Scott Litwin, Benjamin Gray, Jack Chess, Christopher Clark, Sam Berner, Alec Mccandlish, Ilya Radford, Dario Sutskever, Amodei","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2022] Shoufa Chen, Enze Xie, Chongjian Ge, Runjian Chen, Ding Liang, and Ping Luo. Cyclemlp: A mlp-like architecture for dense prediction. In ICLR, 2022.","title":"Cyclemlp: A mlp-like architecture for dense prediction","authors":"Chen","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chu et al., 2021a] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In NeurIPS, 2021.","title":"Twins: Revisiting the design of spatial attention in vision transformers","authors":"Chu","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chu et al., 2021b] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision trans- formers. arXiv:2102.10882, 2021.","title":"Conditional positional encodings for vision transformers","authors":"Chu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Fei-Fei Li","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understand- ing. In NAACL-HLT, 2019.","title":"BERT: pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Diao et al., 2022] Qishuai Diao, Yi Jiang, Bin Wen, Jia Sun, and Zehuan Yuan. Metaformer: A unified meta framework for fine-grained recognition. arXiv:2203.02751, 2022.","title":"Metaformer: A unified meta framework for fine-grained recognition","authors":"Qishuai Diao, Yi Jiang, Bin Wen, Jia Sun, Zehuan Yuan","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dong et al., 2022] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vi- sion transformer backbone with cross-shaped windows. In CVPR, 2022.","title":"Cswin transformer: A general vision transformer backbone with cross-shaped windows","authors":"Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, Baining Guo","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale. In ICLR, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guo et al., 2021] Meng-Hao Guo, Zheng-Ning Liu, Tai- Jiang Mu, and Shi-Min Hu. Beyond self-attention: Ex- ternal attention using two linear layers for visual tasks. arXiv:2105.02358, 2021.","title":"Beyond self-attention: External attention using two linear layers for visual tasks","authors":"Meng-Hao Guo, Zheng-Ning Liu, Tai-Jiang Mu, Shi-Min Hu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Guo et al., 2022] Jianyuan Guo, Yehui Tang, Kai Han, Xinghao Chen, Han Wu, Chao Xu, Chang Xu, and Yunhe Wang. Hire-mlp: Vision MLP via hierarchical rearrange- ment. In CVPR, 2022.","title":"Hire-mlp: Vision MLP via hierarchical rearrangement","authors":"Guo","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2016a] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"He","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2016b] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In ECCV, 2016.","title":"Identity mappings in deep residual networks","authors":"He","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[He et al., 2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In ICCV, 2017.","title":"Mask R-CNN","authors":"He","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hou et al., 2021] Qibin Hou, Zihang Jiang, Li Yuan, Ming- Ming Cheng, Shuicheng Yan, and Jiashi Feng. Vision permutator: A permutable mlp-like architecture for visual recognition. arXiv:2106.12368, 2021.","title":"Vision permutator: A permutable mlp-like architecture for visual recognition","authors":"Hou","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In ICML, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kirillov et al., 2019] Alexander Kirillov, Ross B. Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In CVPR, 2019.","title":"Panoptic feature pyramid networks","authors":"Alexander Kirillov, Ross B Girshick, Kaiming He, Piotr Dollár","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2021] Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool. Localvit: Bringing locality to vision transformers. arXiv:2104.05707, 2021.","title":"Localvit: Bringing locality to vision transformers","authors":"Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, Luc Van Gool","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lian et al., 2022] Dongze Lian, Zehao Yu, Xing Sun, and Shenghua Gao. AS-MLP: an axial shifted MLP architec- ture for vision. In ICLR, 2022.","title":"AS-MLP: an axial shifted MLP architecture for vision","authors":"Lian","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge J. Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: com- mon objects in context. In ECCV, 2014.","title":"Microsoft COCO: common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge J Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence Zitnick","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lin et al., 2017] Tsung-Yi Lin, Priya Goyal, Ross B. Gir- shick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.","title":"Focal loss for dense object detection","authors":"Tsung-Yi Lin, Priya Goyal, Ross B Girshick, Kaiming He, Piotr Dollár","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Liu et al., 2022] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. arXiv:2201.03545, 2022.","title":"A convnet for the 2020s","authors":"Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, Saining Xie","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabi- novich. Going deeper with convolutions. In CVPR, 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Szegedy et al., 2016] Christian Szegedy, Vincent Van- houcke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.","title":"Rethinking the inception architecture for computer vision","authors":"Szegedy","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Szegedy et al., 2017] Christian Szegedy, Sergey Ioffe, Vin- cent Vanhoucke, and Alexander A. Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In AAAI, 2017.","title":"Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A. Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning","authors":"Szegedy","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tolstikhin et al., 2021] Ilya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mlp-mixer: An all-mlp architecture for vi- sion. arXiv:2105.01601, 2021.","title":"Mlp-mixer: An all-mlp architecture for vision","authors":"O Ilya, Neil Tolstikhin, Alexander Houlsby, Lucas Kolesnikov, Xiaohua Beyer, Thomas Zhai, Jessica Unterthiner, Andreas Yung, Daniel Steiner, Jakob Keysers, Mario Uszkoreit, Alexey Lucic, Dosovitskiy","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Touvron et al., 2021a] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. Resmlp: Feedforward net- works for image classification with data-efficient training. arXiv:2105.03404, 2021.","title":"Resmlp: Feedforward networks for image classification with data-efficient training","authors":"Touvron","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Touvron et al., 2021b] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablay- rolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.","title":"Training data-efficient image transformers & distillation through attention","authors":"Touvron","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wang et al., 2021] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A ver- satile backbone for dense prediction without convolutions. In ICCV, 2021.","title":"Pyramid vision transformer: A versatile backbone for dense prediction without convolutions","authors":"Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wang et al., 2022a] Guangting Wang, Yucheng Zhao, Chuanxin Tang, Chong Luo, and Wenjun Zeng. When shift operation meets vision transformer: An extremely simple alternative to attention mechanism. In AAAI, 2022.","title":"When shift operation meets vision transformer: An extremely simple alternative to attention mechanism","authors":"Wang","year":"2022","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wang et al., 2022b] Ziyu Wang, Wenhao Jiang, Yiming Zhu, Li Yuan, Yibing Song, and Wei Liu. Dynamixer: A vision MLP architecture with dynamic mixing. In ICML, 2022.","title":"Dynamixer: A vision MLP architecture with dynamic mixing","authors":"Wang","year":"2022","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wu et al., 2021a] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In ICCV, 2021.","title":"Cvt: Introducing convolutions to vision transformers","authors":"Wu","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wu et al., 2021b] Yu-Huan Wu, Yun Liu, Xin Zhan, and Ming-Ming Cheng. P2T: pyramid pooling transformer for scene understanding. arXiv:2106.12011, 2021.","title":"P2T: pyramid pooling transformer for scene understanding","authors":"Wu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Yang et al., 2021] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal attention for long-range interactions in vision trans- formers. In NeurIPS, 2021.","title":"Focal attention for long-range interactions in vision transformers","authors":"Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, Jianfeng Gao","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Yuan et al., 2021] Li Yuan, Yunpeng Chen, Tao Wang, Wei- hao Yu, Yujun Shi, Zi-Hang Jiang, Francis E.H. Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In ICCV, 2021.","title":"Tokens-to-token vit: Training vision transformers from scratch on imagenet","authors":"Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis E H Tay, Jiashi Feng, Shuicheng Yan","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhou et al., 2019] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ADE20K dataset. IJCV, 2019.","title":"Semantic understanding of scenes through the ADE20K dataset","authors":"Zhou","year":"2019","venue":"IJCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}