{"type":"ScholarlyArticle","title":"SWAT: Spatial Structure Within and Among Tokens","authors":[{"name":"Kumara Kahatapitiya","slug":"kumara-kahatapitiya","given":"Kumara","family":"Kahatapitiya","affiliation":"Stony Brook University"},{"name":"Michael S. Ryoo","slug":"michael-s-ryoo","given":"Michael. S.","family":"Ryoo","affiliation":"Stony Brook University"}],"abstract":"Modeling visual data as tokens (i.e., image patches) using attention mechanisms, feed-forward networks or convolutions has been highly effective in recent years. Such methods usually have a common pipeline: a tokenization method, followed by a set of layers/blocks for information mixing, both within and among tokens. When image patches are converted into tokens, they are often flattened, discarding the spatial structure within each patch. As a result, any processing that follows (eg: multi-head self-attention) may fail to recover and/or benefit from such information. In this paper, we argue that models can have significant gains when spatial structure is preserved during tokenization, and is explicitly used during the mixing stage. We propose two key contributions: (1) Structure-aware Tokenization and, (2) Structure-aware Mixing, both of which can be combined with existing models with minimal effort. We introduce a family of models (SWAT), showing improvements over the likes of DeiT, MLP-Mixer and Swin Transformer, across multiple benchmarks including ImageNet classification and ADE20K segmentation. Our code is available at github.com/kkahatapitiya/SWAT.","doi":"10.24963/ijcai.2023/106","url":"https://ijcai.org/proceedings/2023/106","pdf":"https://ijcai.org/proceedings/2023/0106.pdf","markdown":"https://ijcai.org/proceedings/2023/106.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/106","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":106,"pages":{"first":956,"last":964},"keywords":["Computer Vision: CV: Recognition (object detection, categorization)","Computer Vision: CV: Representation learning"],"videos":[],"references":[{"position":1,"raw":"[Arnab et al., 2021] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. ViViT: A Video Vision Transformer. In ICCV, October 2021.","title":"ViViT: A Video Vision Transformer","authors":"Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, Cordelia Schmid","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bello, 2020] Irwan Bello. LambdaNetworks: Modeling long-range Interactions without Attention. In ICLR, 2020.","title":"LambdaNetworks: Modeling long-range Interactions without Attention","authors":"Irwan Bello","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cao et al., 2021] Jiezhang Cao, Yawei Li, Kai Zhang, and Luc Van Gool. Video Super-Resolution Transformer. arXiv:2106.06847, 2021.","title":"Video Super-Resolution Transformer","authors":"Jiezhang Cao, Yawei Li, Kai Zhang, Luc Van Gool","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Carion et al., 2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-End Object Detection with Transformers. In ECCV. Springer, 2020.","title":"End-to-End Object Detection with Transformers","authors":"Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Caron et al., 2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers. In ICCV, October 2021.","title":"Emerging Properties in Self-Supervised Vision Transformers","authors":"Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2018] Liang-Chieh Chen, Yukun Zhu, George Papan- dreou, Florian Schroff, and Hartwig Adam. Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. In ECCV, 2018.","title":"Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation","authors":"Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, Hartwig Adam","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2021a] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Ar- avind Srinivas, and Igor Mordatch. Decision Transformer: Rein- forcement Learning via Sequence Modeling. URL Workshop in ICML, 2021.","title":"Decision Transformer: Reinforcement Learning via Sequence Modeling","authors":"Chen","year":"2021","venue":"URL Workshop in ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2021b] Shoufa Chen, Enze Xie, Chongjian Ge, Ding Liang, and Ping Luo. CycleMLP: A MLP-like Architecture for Dense Prediction. arXiv:2107.10224, 2021.","title":"CycleMLP: A MLP-like Architecture for Dense Prediction","authors":"Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2022] Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu. Mobile- former: Bridging mobilenet and transformer. In CVPR, 2022.","title":"Mobileformer: Bridging mobilenet and transformer","authors":"Chen","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chu et al., 2021] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chun- hua Shen. Twins: Revisiting the design of spatial attention in vision transformers. NeurIPS, 34, 2021.","title":"Twins: Revisiting the design of spatial attention in vision transformers","authors":"Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, Chunhua Shen","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dai et al., 2021] Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. UP-DETR: Unsupervised Pre-training for Object Detection with Transformers. In CVPR, 2021.","title":"UP-DETR: Unsupervised Pre-training for Object Detection with Transformers","authors":"Zhigang Dai, Bolun Cai, Yugeng Lin, Junying Chen","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Dai et al., 2022] Rui Dai, Srijan Das, Kumara Kahatapitiya, Michael S Ryoo, and Francois Bremond. MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection. In CVPR, 2022.","title":"MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection","authors":"Rui Dai, Srijan Das, Kumara Kahatapitiya, Francois Michael S Ryoo, Bremond","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Dauphin et al., 2017] Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language Modeling with Gated Convolu- tional Networks. In ICML. PMLR, 2017.","title":"Language Modeling with Gated Convolutional Networks","authors":"Angela Yann N Dauphin, Michael Fan, David Auli, Grangier","year":"2017","venue":"ICML. PMLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR. IEEE, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR. IEEE","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. Association for Compu- tational Linguistics, June 2019.","title":"BERT: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. 2021.","title":"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Duke et al., 2021] Brendan Duke, Abdalla Ahmed, Christian Wolf, Parham Aarabi, and Graham W Taylor. SSTVOS: Sparse Spa- tiotemporal Transformers for Video Object Segmentation. In CVPR, 2021.","title":"SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentation","authors":"Brendan Duke, Abdalla Ahmed, Christian Wolf, Parham Aarabi, Graham W Taylor","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[d'Ascoli et al., 2021] Stéphane d'Ascoli, Hugo Touvron, Matthew L Leavitt, Ari S Morcos, Giulio Biroli, and Lev- ent Sagun. ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases. In ICML. PMLR, 2021.","title":"ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases","authors":null,"year":"2021","venue":"ICML. PMLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Esser et al., 2021] Patrick Esser, Robin Rombach, and Bjorn Om- mer. Taming Transformers for High-Resolution Image Synthesis. In CVPR, 2021.","title":"Taming Transformers for High-Resolution Image Synthesis","authors":"Patrick Esser, Robin Rombach, Bjorn Ommer","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Fan et al., 2021] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Fe- ichtenhofer. Multiscale Vision Transformers. In ICCV, October 2021.","title":"Multiscale Vision Transformers","authors":"Bo Haoqi Fan, Karttikeya Xiong, Yanghao Mangalam, Zhicheng Li, Jitendra Yan, Christoph Malik, Feichtenhofer","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Fu et al., 2019a] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual Attention Network for Scene Segmentation. In CVPR, 2019.","title":"Dual Attention Network for Scene Segmentation","authors":"Fu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Fu et al., 2019b] Jun Fu, Jing Liu, Yuhang Wang, Yong Li, Yongjun Bao, Jinhui Tang, and Hanqing Lu. Adaptive Context Network for Scene Parsing. In ICCV, 2019.","title":"Adaptive Context Network for Scene Parsing","authors":"Fu","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Graham et al., 2021] Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. LeViT: A Vision Transformer in ConvNet's Clothing for Faster Inference. In ICCV, October 2021.","title":"LeViT: A Vision Transformer in ConvNet's Clothing for Faster Inference","authors":"Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, Matthijs Douze","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Guo et al., 2021] Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. PCT: Point Cloud Transformer. Computational Visual Media, 7(2), 2021.","title":"PCT: Point Cloud Transformer","authors":"Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Shi-Min Ralph R Martin, Hu","year":"2021","venue":"Computational Visual Media","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Guo et al., 2022] Meng-Hao Guo, Cheng-Ze Lu, Zheng-Ning Liu, Ming-Ming Cheng, and Shi-Min Hu. Visual Attention Network. arXiv:2202.09741, 2022.","title":"Visual Attention Network","authors":"Meng-Hao Guo, Cheng-Ze Lu, Zheng-Ning Liu, Ming-Ming Cheng, Shi-Min Hu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Han et al., 2021] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in Transformer. In NeurIPS, 2021.","title":"Transformer in Transformer","authors":"Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, Yunhe Wang","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016.","title":"Deep Residual Learning for Image Recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Heo et al., 2021] Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking Spatial Dimensions of Vision Transformers. In ICCV, October 2021.","title":"Rethinking Spatial Dimensions of Vision Transformers","authors":"Heo","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[LeCun et al., 2015] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep Learning. Nature, 521(7553), 2015.","title":"Deep Learning","authors":"Yann Lecun, Yoshua Bengio, Geoffrey Hinton","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Trans- former: Hierarchical Vision Transformer Using Shifted Windows. In ICCV, October 2021.","title":"Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Liu et al., 2022] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. In CVPR, 2022.","title":"A ConvNet for the 2020s","authors":"Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, Saining Xie","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Nagrani et al., 2021] Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention Bottle- necks for Multimodal Fusion. NeurIPS, 34, 2021.","title":"Attention Bottlenecks for Multimodal Fusion","authors":"Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, Chen Sun","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Naseer et al., 2021] Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming- Hsuan Yang. Intriguing Properties of Vision Transformers. In NeurIPS, 2021. [OpenMMLab, 2020] OpenMMLab. MMSegmentation: OpenMM- Lab Semantic Segmentation Toolbox and Benchmark. https: //github.com/open-mmlab/mmsegmentation, 2020. Accessed: 2023-01-18.","title":"Intriguing Properties of Vision Transformers","authors":"Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang","year":"2020","venue":"OpenMMLab. MMSegmentation: OpenMM-Lab Semantic Segmentation Toolbox and Benchmark","doi":null,"url":"https://github.com/open-mmlab/mmsegmentation,2020","internal_url":null},{"position":34,"raw":"[Peng et al., 2021] Zhiliang Peng, Wei Huang, Shanzhi Gu, Lingxi Xie, Yaowei Wang, Jianbin Jiao, and Qixiang Ye. Conformer: Lo- cal Features Coupling Global Representations for Visual Recogni- tion. In ICCV, October 2021.","title":"Conformer: Local Features Coupling Global Representations for Visual Recognition","authors":"Zhiliang Peng, Wei Huang, Shanzhi Gu, Lingxi Xie, Yaowei Wang, Jianbin Jiao, Qixiang Ye","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Radosavovic et al., 2020] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In CVPR, 2020.","title":"Designing network design spaces","authors":"Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, Piotr Dollár","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Ranasinghe et al., 2022] Kanchana Ranasinghe, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, and Michael S Ryoo. Self-supervised Video Transformer. In CVPR, 2022.","title":"Self-supervised Video Transformer","authors":"Kanchana Ranasinghe, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, Michael S Ryoo","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Ranftl et al., 2021] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision Transformers for Dense Prediction. In ICCV, 2021.","title":"Vladlen Koltun. Vision Transformers for Dense Prediction","authors":"Ranftl","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Ryoo et al., 2021] Michael S Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. TokenLearner: Adaptive Space-Time Tokenization for Videos. In NeurIPS, 2021.","title":"TokenLearner: Adaptive Space-Time Tokenization for Videos","authors":"Michael S Ryoo, Anurag Piergiovanni, Mostafa Arnab, Anelia Dehghani, Angelova","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Shang et al., 2022] Jinghuan Shang, Kumara Kahatapitiya, Xiang Li, and Michael S Ryoo. StARformer: Transformer with State- Action-Reward Representations. In ECCV, 2022.","title":"StARformer: Transformer with State-Action-Reward Representations","authors":"Shang","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Tan and Le, 2019] Mingxing Tan and Quoc Le. EfficientNet: Re- thinking Model Scaling for Convolutional Neural Networks. In ICML. PMLR, 2019.","title":"EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks","authors":"Le ; Tan, Mingxing Tan, Quoc Le","year":"2019","venue":"ICML. PMLR","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Tang et al., 2021] Chuanxin Tang, Yucheng Zhao, Guangting Wang, Chong Luo, Wenxuan Xie, and Wenjun Zeng. Sparse MLP for Image Recognition: Is Self-Attention Really Necessary? arXiv:2109.05422, 2021.","title":"Sparse MLP for Image Recognition: Is Self-Attention Really Necessary","authors":"Chuanxin Tang, Yucheng Zhao, Guangting Wang, Chong Luo, Wenxuan Xie, Wenjun Zeng","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Tolstikhin et al., 2021] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jes- sica Yung, Andreas Peter Steiner, Daniel Keysers, Jakob Uszkor- eit, et al. MLP-Mixer: An All-MLP Architecture for Vision. In NeurIPS, 2021.","title":"MLP-Mixer: An All-MLP Architecture for Vision","authors":"Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Peter Steiner, Daniel Keysers, Jakob Uszkoreit","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Touvron et al., 2021a] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gau- tier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al. ResMLP: Feedforward networks for image classification with data-efficient training. arXiv:2105.03404, 2021.","title":"ResMLP: Feedforward networks for image classification with data-efficient training","authors":"Touvron","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Touvron et al., 2021b] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML. PMLR, 2021.","title":"Training data-efficient image transformers & distillation through attention","authors":"Touvron","year":"2021","venue":"ICML. PMLR","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Trockman and Kolter, 2022] Asher Trockman and J Zico Kolter. Patches are all you need? arXiv:2201.09792, 2022.","title":"Patches are all you need","authors":"Kolter ; Trockman, J Asher Trockman, Kolter Zico","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In NeurIPS, 2017.","title":"Attention Is All You Need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Wang et al., 2018] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local Neural Networks. In CVPR, 2018.","title":"Non-local Neural Networks","authors":"Xiaolong Wang, Ross Girshick, Abhinav Gupta, Kaiming He","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Wang et al., 2021] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction Without Convolutions. In ICCV, October 2021.","title":"Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction Without Convolutions","authors":"Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Wightman, 2019] Ross Wightman. Pytorch Image Models. https: //github.com/rwightman/pytorch-image-models, 2019. Accessed: 2023-01-18.","title":"Pytorch Image Models","authors":"Ross Wightman","year":"2023","venue":null,"doi":null,"url":"https://github.com/rwightman/pytorch-image-models,2019","internal_url":null},{"position":50,"raw":"[Wu et al., 2021] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. CvT: Introducing Convolutions to Vision Transformers. In ICCV, October 2021.","title":"CvT: Introducing Convolutions to Vision Transformers","authors":"Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, Lei Zhang","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Xiao et al., 2018] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified Perceptual Parsing for Scene Under- standing. In ECCV, 2018.","title":"Unified Perceptual Parsing for Scene Understanding","authors":"Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, Jian Sun","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Xiao et al., 2021] Tete Xiao, Piotr Dollar, Mannat Singh, Eric Mintun, Trevor Darrell, and Ross Girshick. Early Convolutions Help Transformers See Better. In NeurIPS, 2021.","title":"Early Convolutions Help Transformers See Better","authors":"Tete Xiao, Piotr Dollar, Mannat Singh, Eric Mintun, Trevor Darrell, Ross Girshick","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Xie et al., 2017] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, 2017.","title":"Aggregated residual transformations for deep neural networks","authors":"Xie","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Xie et al., 2021] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. SegFormer: Simple and Efficient Design for Semantic Segmentation with Transform- ers. In NeurIPS, 2021.","title":"SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers","authors":"Xie","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":55,"raw":"[Yang et al., 2021a] Guanglei Yang, Hao Tang, Mingli Ding, Nicu Sebe, and Elisa Ricci. Transformer-Based Attention Networks for Continuous Pixel-Wise Prediction. In ICCV, 2021.","title":"Transformer-Based Attention Networks for Continuous Pixel-Wise Prediction","authors":"Yang","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":56,"raw":"[Yang et al., 2021b] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal Self- attention for Local-Global Interactions in Vision Transformers. In NeurIPS, 2021.","title":"Focal Selfattention for Local-Global Interactions in Vision Transformers","authors":"Yang","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":57,"raw":"[Yin et al., 2020] Minghao Yin, Zhuliang Yao, Yue Cao, Xiu Li, Zheng Zhang, Stephen Lin, and Han Hu. Disentangled Non-Local Neural Networks. In ECCV. Springer, 2020.","title":"Disentangled Non-Local Neural Networks","authors":"Minghao Yin, Zhuliang Yao, Yue Cao, Xiu Li, Zheng Zhang, Stephen Lin, Han Hu","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":58,"raw":"[Yu et al., 2022] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In CVPR, 2022.","title":"Metaformer is actually what you need for vision","authors":"Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, Shuicheng Yan","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":59,"raw":"[Yuan et al., 2021a] Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating Convolution Designs Into Visual Transformers. In ICCV, October 2021.","title":"Incorporating Convolution Designs Into Visual Transformers","authors":"Yuan","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":60,"raw":"[Yuan et al., 2021b] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis E.H. Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-Token ViT: Training Vision Trans- formers From Scratch on ImageNet. In ICCV, October 2021.","title":"Tokens-to-Token ViT: Training Vision Transformers From Scratch on ImageNet","authors":"Yuan","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":61,"raw":"[Yue et al., 2021] Xiaoyu Yue, Shuyang Sun, Zhanghui Kuang, Meng Wei, Philip H.S. Torr, Wayne Zhang, and Dahua Lin. Vision Transformer With Progressive Sampling. In ICCV, October 2021.","title":"Vision Transformer With Progressive Sampling","authors":"Xiaoyu Yue, Shuyang Sun, Zhanghui Kuang, Meng Wei, H S Philip, Wayne Torr, Dahua Zhang, Lin","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":62,"raw":"[Zhai et al., 2021] Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, and Josh Susskind. An Attention Free Transformer. arXiv:2105.14103, 2021.","title":"An Attention Free Transformer","authors":"Zhai","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":63,"raw":"[Zhang et al., 2021] Zizhao Zhang, Han Zhang, Long Zhao, Ting Chen, and Tomas Pfister. Aggregating Nested Transformers. arXiv:2105.12723, 2021.","title":"Aggregating Nested Transformers","authors":"Zizhao Zhang, Han Zhang, Long Zhao, Ting Chen, Tomas Pfister","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":64,"raw":"[Zhao et al., 2020] Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring Self-attention for Image Recognition. In CVPR, 2020.","title":"Exploring Self-attention for Image Recognition","authors":"Zhao","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":65,"raw":"[Zhao et al., 2021] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point Transformer. In ICCV, 2021.","title":"Point Transformer","authors":"Hengshuang Zhao, Li Jiang, Jiaya Jia, Vladlen Philip Hs Torr, Koltun","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":66,"raw":"[Zhou et al., 2019] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic Understanding of Scenes Through the ADE20K Dataset. IJCV, 127(3), 2019.","title":"Semantic Understanding of Scenes Through the ADE20K Dataset","authors":"Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, Antonio Torralba","year":"2019","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":67,"raw":"[Zhu et al., 2020] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xi- aogang Wang, and Jifeng Dai. Deformable DETR: Deformable Transformers for End-to-End Object Detection. In ICLR, 2020.","title":"Deformable DETR: Deformable Transformers for End-to-End Object Detection","authors":"Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, Jifeng Dai","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}