{"type":"ScholarlyArticle","title":"Data Level Lottery Ticket Hypothesis for Vision Transformers","authors":[{"name":"Xuan Shen","slug":"xuan-shen","given":"Xuan","family":"Shen","affiliation":"Northeastern University"},{"name":"Zhenglun Kong","slug":"zhenglun-kong","given":"Zhenglun","family":"Kong","affiliation":"Northeastern University"},{"name":"Minghai Qin","slug":"minghai-qin","given":"Minghai","family":"Qin","affiliation":"Western Digital Research"},{"name":"Peiyan Dong","slug":"peiyan-dong","given":"Peiyan","family":"Dong","affiliation":"Northeastern University"},{"name":"Geng Yuan","slug":"geng-yuan","given":"Geng","family":"Yuan","affiliation":"Northeastern University"},{"name":"Xin Meng","slug":"xin-meng","given":"Xin","family":"Meng","affiliation":"Peking University"},{"name":"Hao Tang","slug":"hao-tang","given":"Hao","family":"Tang","affiliation":"ETH Zurich"},{"name":"Xiaolong Ma","slug":"xiaolong-ma","given":"Xiaolong","family":"Ma","affiliation":"Clemson University"},{"name":"Yanzhi Wang","slug":"yanzhi-wang","given":"Yanzhi","family":"Wang","affiliation":"Northeastern University"}],"abstract":"The conventional lottery ticket hypothesis (LTH) claims that there exists a sparse subnetwork within a dense neural network and a proper random initialization method, called the winning ticket, such that it can be trained from scratch to almost as good as the dense counterpart. Meanwhile, the research of LTH in vision transformers (ViTs) is scarcely evaluated. In this paper, we first show that the conventional winning ticket is hard to find at weight level of ViTs by existing methods. Then, we generalize the LTH for ViTs to input data consisting of image patches inspired by the input dependence of ViTs. That is, there exists a subset of input image patches such that a ViT can be trained from scratch by using only this subset of patches and achieve similar accuracy to the ViTs trained by using all image patches. We call this subset of input patches the winning tickets, which represent a significant amount of information in the input data. We use a ticket selector to generate the winning tickets based on the informativeness of patches for various types of ViT, including DeiT, LV-ViT, and Swin Transformers. The experiments show that there is a clear difference between the performance of models trained with winning tickets and randomly selected subsets, which verifies our proposed theory. We elaborate the analogical similarity between our proposed Data-LTH-ViTs and the conventional LTH for further verifying the integrity of our theory. The Source codes are available at https://github.com/shawnricecake/vit-lottery-ticket-input.","doi":"10.24963/ijcai.2023/153","url":"https://ijcai.org/proceedings/2023/153","pdf":"https://ijcai.org/proceedings/2023/0153.pdf","markdown":"https://ijcai.org/proceedings/2023/153.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/153","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":153,"pages":{"first":1378,"last":1386},"keywords":["Machine Learning: ML: Theory of deep learning","Computer Vision: CV: Machine learning for vision"],"videos":[],"references":[{"position":1,"raw":"[Bai et al., 2022] Yue Bai, Huan Wang, ZHIQIANG TAO, Kunpeng Li, and Yun Fu. Dual lottery ticket hypothesis. In International Conference on Learning Representations, 2022.","title":"Dual lottery ticket hypothesis","authors":"Yue Bai, Huan Wang, Zhiqiang Tao, Kunpeng Li, Yun Fu","year":"2022","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bao et al., 2021] Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021.","title":"Beit: Bert pre-training of image transformers","authors":"Bao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Carion et al., 2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.","title":"End-to-end object detection with transformers","authors":"Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko","year":"2020","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2020] Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre- trained bert networks. Advances in neural information pro- cessing systems, 33:15834-15846, 2020.","title":"The lottery ticket hypothesis for pretrained bert networks","authors":"Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, Michael Carbin","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2021a] Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision (ICCV), pages 12270-12280, 2021.","title":"Autoformer: Searching transformers for visual recognition","authors":"Chen","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2021b] Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing spar- sity in vision transformers: An end-to-end exploration. Advances in Neural Information Processing Systems, 34, 2021.","title":"Chasing sparsity in vision transformers: An end-to-end exploration","authors":"Chen","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2022] Tianlong Chen, Zhenyu Zhang, Yu Cheng, Ahmed Awadallah, and Zhangyang Wang. The principle of diversity: Training stronger vision transformers calls for reducing all levels of redundancy. arXiv preprint arXiv:2203.06345, 2022.","title":"The principle of diversity: Training stronger vision transformers calls for reducing all levels of redundancy","authors":"Tianlong Chen, Zhenyu Zhang, Yu Cheng, Ahmed Awadallah, Zhangyang Wang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Desai et al., 2019] Shrey Desai, Hongyuan Zhan, and Ahmed Aly. Evaluating lottery tickets under distributional shifts. arXiv preprint arXiv:1910.12708, 2019.","title":"Evaluating lottery tickets under distributional shifts","authors":"Shrey Desai, Hongyuan Zhan, Ahmed Aly","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale. ICLR, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[El-Nouby et al., 2021] Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Ar- mand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Syn- naeve, Jakob Verbeek, et al. Xcit: Cross-covariance image transformers. arXiv preprint arXiv:2106.09681, 2021.","title":"Alaaeldin El-Nouby","authors":"El-Nouby","year":"2021","venue":"Xcit: Cross-covariance image transformers","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Frankle and Carbin, 2018] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. ICLR, 2018.","title":"The lottery ticket hypothesis: Finding sparse, trainable neural networks","authors":"Carbin Frankle, Jonathan Frankle, Michael Carbin","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Frankle et al., 2020] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Inter- national Conference on Machine Learning, pages 3259- 3269. PMLR, 2020.","title":"Linear mode connectivity and the lottery ticket hypothesis","authors":"Frankle","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gale et al., 2019] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574, 2019.","title":"The state of sparsity in deep neural networks","authors":"Trevor Gale, Erich Elsen, Sara Hooker","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Girish et al., 2021] Sharath Girish, Shishira R Maiya, Ka- mal Gupta, Hao Chen, Larry Davis, and Abhinav Shrivas- tava. The lottery ticket hypothesis for object recognition. CVPR, 2021.","title":"The lottery ticket hypothesis for object recognition","authors":"Sharath Girish, Kamal Shishira R Maiya, Hao Gupta, Larry Chen, Abhinav Davis, Shrivastava","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hou and Kung, 2021] Zejiang Hou and Sun-Yuan Kung. Multi-dimensional model compression of vision trans- former. arXiv preprint arXiv:2201.00043, 2021.","title":"Multi-dimensional model compression of vision transformer","authors":"Kung Hou, Zejiang Hou, Sun-Yuan Kung","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Jiang et al., 2021] Zihang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Yujun Shi, Xiaojie Jin, Anran Wang, and Jiashi Feng. All tokens matter: Token labeling for training better vision transformers. arXiv preprint arXiv:2104.10858, 2021.","title":"All tokens matter: Token labeling for training better vision transformers","authors":"Jiang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kong et al., 2022a] Zhenglun Kong, Peiyan Dong, Xiao- long Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, et al. Spvit: Enabling faster vision transformers via latency-aware soft token pruning. In Computer Vision-ECCV 2022: 17th Euro- pean Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XI, pages 620-640. Springer, 2022.","title":"Spvit: Enabling faster vision transformers via latency-aware soft token pruning","authors":"Kong","year":"2022","venue":"Computer Vision-ECCV 2022: 17th European Conference","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Kong et al., 2022b] Zhenglun Kong, Haoyu Ma, Geng Yuan, Mengshu Sun, Yanyue Xie, Peiyan Dong, Xin Meng, Xuan Shen, Hao Tang, Minghai Qin, et al. Peel- ing the onion: Hierarchical reduction of data redundancy for efficient vision transformer training. arXiv preprint arXiv:2211.10801, 2022.","title":"Peeling the onion: Hierarchical reduction of data redundancy for efficient vision transformer training","authors":"Kong","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Krizhevsky, 2009] Alex Krizhevsky. Learning multiple lay- ers of features from tiny images. Technical report, Cite- seer, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liang et al., 2022] Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transform- ers via token reorganizations. In International Conference on Learning Representations, 2022.","title":"Not all patches are what you need: Expediting vision transformers via token reorganizations","authors":"Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, Pengtao Xie","year":"2022","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Liu et al., 2021a] Yahui Liu, Enver Sangineto, Wei Bi, Nicu Sebe, Bruno Lepri, and Marco De Nadai. Efficient train- ing of visual transformers with small-size datasets. arXiv preprint arXiv:2106.03746, 2021.","title":"Efficient training of visual transformers with small-size datasets","authors":"Liu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Liu et al., 2021b] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV), 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Liu","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ma et al., 2021] Xiaolong Ma, Geng Yuan, Xuan Shen, Tianlong Chen, Xuxi Chen, Xiaohan Chen, Ning Liu, Minghai Qin, Sijia Liu, Zhangyang Wang, et al. Sanity checks for lottery tickets: Does your winning ticket really win the jackpot? Advances in Neural Information Pro- cessing Systems, 34, 2021.","title":"Sanity checks for lottery tickets: Does your winning ticket really win the jackpot?","authors":"Xiaolong Ma, Geng Yuan, Xuan Shen, Tianlong Chen, Xuxi Chen, Xiaohan Chen, Ning Liu, Minghai Qin, Sijia Liu, Zhangyang Wang","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Mallya et al., 2018] Arun Mallya, Dillon Davis, and Svet- lana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In Proceedings of the European Conference on Computer Vision (ECCV), pages 67-82, 2018.","title":"Piggyback: Adapting a single network to multiple tasks by learning to mask weights","authors":"Arun Mallya, Dillon Davis, Svetlana Lazebnik","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Pan et al., 2021a] Bowen Pan, Rameswar Panda, Yifan Jiang, Zhangyang Wang, Rogerio Feris, and Aude Oliva. Ia-red: Interpretability-aware redundancy reduction for vi- sion transformers. Advances in Neural Information Pro- cessing Systems, 34, 2021.","title":"Ia-red: Interpretability-aware redundancy reduction for vision transformers","authors":"Pan","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Pan et al., 2021b] Zizheng Pan, Bohan Zhuang, Jing Liu, Haoyu He, and Jianfei Cai. Scalable vision transform- ers with hierarchical pooling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 377-386, 2021.","title":"Scalable vision transformers with hierarchical pooling","authors":"Pan","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Prasanna et al., 2020] Sai Prasanna, Anna Rogers, and Anna Rumshisky. When bert plays the lottery, all tickets are winning. EMNLP, 2020.","title":"When bert plays the lottery, all tickets are winning","authors":"Sai Prasanna, Anna Rogers, Anna Rumshisky","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Rao et al., 2021] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsi- fication. In Advances in Neural Information Processing Systems (NeurIPS), 2021.","title":"Dynamicvit: Efficient vision transformers with dynamic token sparsification","authors":"Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, Cho-Jui Hsieh","year":"2021","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Renda et al., 2020] Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. ICLR, 2020.","title":"Comparing rewinding and fine-tuning in neural network pruning","authors":"Alex Renda, Jonathan Frankle, Michael Carbin","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Steiner et al., 2021] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106.10270, 2021.","title":"How to train your vit? data, augmentation, and regularization in vision transformers","authors":"Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, Lucas Beyer","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Tang et al., 2022] Yehui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao. Patch slimming for efficient vision transformers. In CVPR, 2022.","title":"Patch slimming for efficient vision transformers","authors":"Tang","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Touvron et al., 2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablay- rolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, volume 139, pages 10347-10357, July 2021.","title":"Training data-efficient image transformers & distillation through attention","authors":"Touvron","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wang et al., 2021] Pichao Wang, Xue Wang, Fan Wang, Ming Lin, Shuning Chang, Wen Xie, Hao Li, and Rong Jin. Kvt: k-nn attention for boosting vision transformers. arXiv preprint arXiv:2106.00515, 2021.","title":"Kvt: k-nn attention for boosting vision transformers","authors":"Pichao Wang, Xue Wang, Fan Wang, Ming Lin, Shuning Chang, Wen Xie, Hao Li, Rong Jin","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wu et al., 2021] Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and im- proving relative position encoding for vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10033-10041, 2021.","title":"Rethinking and improving relative position encoding for vision transformer","authors":"Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, Hongyang Chao","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Xu et al., 2021] Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision transformer. arXiv preprint arXiv:2108.01390, 2021.","title":"Evo-vit: Slow-fast token evolution for dynamic vision transformer","authors":"Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, Xing Sun","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Xu et al., 2022] Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.","title":"Evo-vit: Slow-fast token evolution for dynamic vision transformer","authors":"Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, Xing Sun","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Yu et al., 2019] Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari S Morcos. Playing the lottery with rewards and multiple languages: lottery tickets in rl and nlp. arXiv preprint arXiv:1906.02768, 2019.","title":"Playing the lottery with rewards and multiple languages: lottery tickets in rl and nlp","authors":"Yu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Yu et al., 2022a] Shixing Yu, Tianlong Chen, Jiayi Shen, Huan Yuan, Jianchao Tan, Sen Yang, Ji Liu, and Zhangyang Wang. Unified visual transformer compres- sion. In International Conference on Learning Represen- tations, 2022.","title":"Unified visual transformer compression","authors":"Yu","year":"2022","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Yu et al., 2022b] Zhongzhi Yu, Yonggan Fu, Sicheng Li, Chaojian Li, and Yingyan Lin. Mia-former: Efficient and robust vision transformers via multi-grained input- adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.","title":"Mia-former: Efficient and robust vision transformers via multi-grained inputadaptation","authors":"Yu","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Zhou et al., 2019] Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. Advances in neural information processing systems, 32, 2019.","title":"Deconstructing lottery tickets: Zeros, signs, and the supermask","authors":"Hattie Zhou, Janice Lan, Rosanne Liu, Jason Yosinski","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Zhu and Gupta, 2017] Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878, 2017.","title":"To prune, or not to prune: exploring the efficacy of pruning for model compression","authors":"Michael Zhu, Suyog Gupta","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}