{"type":"ScholarlyArticle","title":"Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attention","authors":[{"name":"Xiangcheng Liu","slug":"xiangcheng-liu","given":"Xiangcheng","family":"Liu","affiliation":"Peking University"},{"name":"Tianyi Wu","slug":"tianyi-wu","given":"Tianyi","family":"Wu","affiliation":"Baidu Autonomous Driving Technology Department (ADT)"},{"name":"Guodong Guo","slug":"guodong-guo","given":"Guodong","family":"Guo","affiliation":"Institute of Deep Learning , Baidu Research"}],"abstract":"Vision transformer has emerged as a new paradigm in computer vision, showing excellent performance while accompanied by expensive computational cost. Image token pruning is one of the main approaches for ViT compression, due to the facts that the complexity is quadratic with respect to the token number, and many tokens containing only background regions do not truly contribute to the final prediction. Existing works either rely on additional modules to score the importance of individual tokens, or implement a fixed ratio pruning strategy for different input instances. In this work, we propose an adaptive sparse token pruning framework with a minimal cost. Specifically, we firstly propose an inexpensive attention head importance weighted class attention scoring mechanism. Then, learnable parameters are inserted as thresholds to distinguish informative tokens from unimportant ones. By comparing token attention scores and thresholds, we can discard useless tokens hierarchically and thus accelerate inference. The learnable thresholds are optimized in budget-aware training to balance accuracy and complexity, performing the corresponding pruning configurations for different input instances. Extensive experiments demonstrate the effectiveness of our approach. Our method improves the throughput of DeiT-S by 50% and brings only 0.2% drop in top-1 accuracy, which achieves a better trade-off between accuracy and latency than the previous methods.","doi":"10.24963/ijcai.2023/136","url":"https://ijcai.org/proceedings/2023/136","pdf":"https://ijcai.org/proceedings/2023/0136.pdf","markdown":"https://ijcai.org/proceedings/2023/136.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/136","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":136,"pages":{"first":1222,"last":1230},"keywords":["Machine Learning: ML: Attention models","Computer Vision: CV: Recognition (object detection, categorization)"],"videos":[],"references":[{"position":1,"raw":"137 (70%) 34%) Method Resolution Acc (%)","title":"Method Resolution Acc","authors":null,"year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"GFLOPs DeiT-B [Touvron et al., 2021a] 224 81.8 17.5","title":null,"authors":"Gflops Deit, -B [ Touvron","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"DyViT-B [Rao et al., 2021] 224 81.3 (-0.5) 11","title":null,"authors":"Dyvit-B [ Rao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"2 EViT-DeiT-B [Liang et al., 2022] 224 81.3 (-0.5) 11","title":null,"authors":"Evit-Deit-B [ Liang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"5 IA-RED2 [Pan et al., 2021] 224 80.3 (-1.5) 11.8 AS-DeiT-B 224 81.4 (-0.4) 11","title":"IA-RED2","authors":"Pan","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"DeiT-B [Touvron et al., 2021a] 384 82.9 49.4 IA-RED2 [Pan et al., 2021] 384 81.9 (-1.0) 34.7 AS-DeiT-B 384 82.7 (-0.2)","title":"IA-RED2","authors":"Deit-B [touvron","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Carion et al., 2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pages 213-229, 2020.","title":"End-to-end object detection with transformers","authors":"Carion","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2021] Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. Advances in Neural Information Processing Systems, 34, 2021.","title":"Chasing sparsity in vision transformers: An end-to-end exploration","authors":"Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, Zhangyang Wang","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chu et al., 2021] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. Advances in Neural Infor- mation Processing Systems, 34, 2021.","title":"Twins: Revisiting the design of spatial attention in vision transformers","authors":"Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, Chunhua Shen","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Represen- tations (ICLR), 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Frankle and Carbin, 2018] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2018.","title":"The lottery ticket hypothesis: Finding sparse, trainable neural networks","authors":"Carbin Frankle, Jonathan Frankle, Michael Carbin","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Goyal et al., 2020] Saurabh Goyal, Anamitra Roy Choud- hury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sabharwal, and Ashish Verma. Power-bert: Accelerat- ing bert inference via progressive word-vector elimination. In International Conference on Machine Learning, pages 3690-3699. PMLR, 2020.","title":"Power-bert: Accelerating bert inference via progressive word-vector elimination","authors":"Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sabharwal, Ashish Verma","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Han et al., 2015] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.","title":"Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding","authors":"Han","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[He et al., 2018] Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accel- erating deep convolutional neural networks. arXiv preprint arXiv:1808.06866, 2018.","title":"Soft filter pruning for accelerating deep convolutional neural networks","authors":"Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, Yi Yang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2018/309"},{"position":17,"raw":"[Jiang et al., 2021] Zihang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Yujun Shi, Xiaojie Jin, Anran Wang, and Jiashi Feng. All tokens matter: Token labeling for training better vision transformers. arXiv preprint arXiv:2104.10858, 2021.","title":"All tokens matter: Token labeling for training better vision transformers","authors":"Jiang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kim et al., 2021] Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. Learned token pruning for transformers. arXiv preprint arXiv:2107.00910, 2021.","title":"Learned token pruning for transformers","authors":"Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, Kurt Keutzer","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2016] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710, 2016.","title":"Pruning filters for efficient convnets","authors":"Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liang et al., 2022] Youwei Liang, Chongjian GE, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Evit: Expediting vision transformers via token reorganizations. In International Conference on Learning Representations (ICLR), 2022.","title":"Evit: Expediting vision transformers via token reorganizations","authors":"Youwei Liang, G E Chongjian, Zhan Tong, Yibing Song, Jue Wang, Pengtao Xie","year":"2022","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2017] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slim- ming. In ICCV, 2017.","title":"Learning efficient convolutional networks through network slimming","authors":"Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, Changshui Zhang","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yix- uan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV), pages 10012-10022, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu et al., 2022a] Kai Liu, Tianyi Wu, Cong Liu, and Guodong Guo. Dynamic group transformer: A general vision transformer backbone with dynamic group attention. arXiv preprint arXiv:2203.03937, 2022.","title":"Dynamic group transformer: A general vision transformer backbone with dynamic group attention","authors":"Liu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2022/166"},{"position":24,"raw":"[Liu et al., 2022b] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.","title":"A convnet for the 2020s","authors":"Liu","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Luo et al., 2017] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural net- work compression. In Proceedings of the IEEE interna- tional conference on computer vision, pages 5058-5066, 2017.","title":"Thinet: A filter level pruning method for deep neural network compression","authors":"Jian-Hao Luo, Jianxin Wu, Weiyao Lin","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Michel et al., 2019] Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019.","title":"Are sixteen heads really better than one?","authors":"Paul Michel, Omer Levy, Graham Neubig","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Molchanov et al., 2016] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convo- lutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440, 2016.","title":"Pruning convolutional neural networks for resource efficient inference","authors":"Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, Jan Kautz","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Pan et al., 2021] Bowen Pan, Rameswar Panda, Yifan Jiang, Zhangyang Wang, Rogerio Feris, and Aude Oliva. Ia- red 2 : Interpretability-aware redundancy reduction for vi- sion transformers. In Advances in Neural Information Pro- cessing Systems (NeurIPS), 2021.","title":"Iared 2 : Interpretability-aware redundancy reduction for vision transformers","authors":"Rameswar Bowen Pan, Yifan Panda, Zhangyang Jiang, Rogerio Wang, Aude Feris, Oliva","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Rao et al., 2021] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsi- fication. In Advances in Neural Information Processing Systems (NeurIPS), 2021.","title":"Dynamicvit: Efficient vision transformers with dynamic token sparsification","authors":"Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, Cho-Jui Hsieh","year":"2021","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tan and Le, 2019] Mingxing Tan and Quoc Le. Efficient- net: Rethinking model scaling for convolutional neural networks. In ICML, pages 6105-6114. PMLR, 2019.","title":"Efficientnet: Rethinking model scaling for convolutional neural networks","authors":"Le ; Tan, Mingxing Tan, Quoc Le","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Tang et al., 2021] Yehui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao. Patch slimming for efficient vision transformers. arXiv preprint arXiv:2106.02852, 2021.","title":"Patch slimming for efficient vision transformers","authors":"Yehui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, Dacheng Tao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Touvron et al., 2021a] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transform- ers & distillation through attention. In International Con- ference on Machine Learning (ICML), pages 10347-10357, 2021.","title":"Training data-efficient image transformers & distillation through attention","authors":"Touvron","year":"2021","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Touvron et al., 2021b] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jé- gou. Going deeper with image transformers. arXiv preprint arXiv:2103.17239, 2021.","title":"Going deeper with image transformers","authors":"Touvron","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wang et al., 2021] Wenhai Wang, Enze Xie, Xiang Li, Deng- Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 568-578, 2021.","title":"Pyramid vision transformer: A versatile backbone for dense prediction without convolutions","authors":"Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wu et al., 2021] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 22-31, 2021.","title":"Cvt: Introducing convolutions to vision transformers","authors":"Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, Lei Zhang","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Xie et al., 2021] Enze Xie, Wenhai Wang, Zhiding Yu, An- ima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmenta- tion with transformers. arXiv preprint arXiv:2105.15203, 2021.","title":"Segformer: Simple and efficient design for semantic segmentation with transformers","authors":"Xie","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Xu et al., 2021] Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9981-9990, 2021.","title":"Co-scale conv-attentional image transformers","authors":"Weijian Xu, Yifan Xu, Tyler Chang, Zhuowen Tu","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Xu et al., 2022] Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Chang- sheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolu- tion for dynamic vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2022.","title":"Evo-vit: Slow-fast token evolution for dynamic vision transformer","authors":"Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, Xing Sun","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Yang et al., 2021] Huanrui Yang, Hongxu Yin, Pavlo Molchanov, Hai Li, and Jan Kautz. Nvit: Vision trans- former compression and parameter redistribution. arXiv preprint arXiv:2110.04869, 2021.","title":"Nvit: Vision transformer compression and parameter redistribution","authors":"Yang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Yin et al., 2021] Hongxu Yin, Arash Vahdat, Jose Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. Adavit: Adaptive tokens for efficient vision transformer. arXiv preprint arXiv:2112.07658, 2021.","title":"Adavit: Adaptive tokens for efficient vision transformer","authors":"Yin","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Yu et al., 2022] Shixing Yu, Tianlong Chen, Jiayi Shen, Huan Yuan, Jianchao Tan, Sen Yang, Ji Liu, and Zhangyang Wang. Unified visual transformer compression. arXiv preprint arXiv:2203.08243, 2022.","title":"Unified visual transformer compression","authors":"Shixing Yu, Tianlong Chen, Jiayi Shen, Huan Yuan, Jianchao Tan, Sen Yang, Ji Liu, Zhangyang Wang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Yuan et al., 2021] Li Yuan, Yunpeng Chen, Tao Wang, Wei- hao Yu, Yujun Shi, Zi-Hang Jiang, Francis E.H. Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vi- sion transformers from scratch on imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 558-567, October 2021.","title":"Tokens-to-token vit: Training vision transformers from scratch on imagenet","authors":"Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis E H Tay, Jiashi Feng, Shuicheng Yan","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Zheng et al., 2021] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H.S. Torr, and Li Zhang. Rethinking semantic segmentation from a sequence-to- sequence perspective with transformers. In CVPR, 2021.","title":"Rethinking semantic segmentation from a sequence-tosequence perspective with transformers","authors":"Zheng","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Zhu et al., 2020] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.","title":"Deformable detr: Deformable transformers for end-to-end object detection","authors":"Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, Jifeng Dai","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Zhu et al., 2021] Mingjian Zhu, Yehui Tang, and Kai Han. Vision transformer pruning. arXiv preprint arXiv:2104.08500, 2021.","title":"Vision transformer pruning","authors":"Zhu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}