{"type":"ScholarlyArticle","title":"A Survey on Efficient Training of Transformers","authors":[{"name":"Bohan Zhuang","slug":"bohan-zhuang","given":"Bohan","family":"Zhuang","affiliation":"ZIP Lab , Monash University"},{"name":"Jing Liu","slug":"jing-liu","given":"Jing","family":"Liu","affiliation":"ZIP Lab , Monash University"},{"name":"Zizheng Pan","slug":"zizheng-pan","given":"Zizheng","family":"Pan","affiliation":"ZIP Lab , Monash University"},{"name":"Haoyu He","slug":"haoyu-he","given":"Haoyu","family":"He","affiliation":"ZIP Lab , Monash University"},{"name":"Yuetian Weng","slug":"yuetian-weng","given":"Yuetian","family":"Weng","affiliation":"ZIP Lab , Monash University"},{"name":"Chunhua Shen","slug":"chunhua-shen","given":"Chunhua","family":"Shen","affiliation":"Zhejiang University"}],"abstract":"Recent advances in Transformers have come with a huge requirement on computing resources, highlighting the importance of developing efficient training techniques to make Transformer training faster, at lower cost, and to higher accuracy by the efficient use of computation and memory resources. This survey provides the first systematic overview of the efficient training of Transformers, covering the recent progress in acceleration arithmetic and hardware, with a focus on the former. We analyze and compare methods that save computation and memory costs for intermediate tensors during training, together with techniques on hardware/algorithm co-design. We finally discuss challenges and promising areas for future research.","doi":"10.24963/ijcai.2023/764","url":"https://ijcai.org/proceedings/2023/764","pdf":"https://ijcai.org/proceedings/2023/0764.pdf","markdown":"https://ijcai.org/proceedings/2023/764.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/764","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Survey Track","track":"Survey Track","number":764,"pages":{"first":6823,"last":6831},"keywords":["Survey: Natural Language Processing","Survey: Machine Learning","Survey: Computer Vision","Survey: Multidisciplinary Topics and Applications"],"videos":[],"references":[{"position":1,"raw":"[Aghajanyan et al., 2021] Armen Aghajanyan, Sonal Gupta, et al. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In ACL, 2021.","title":"Intrinsic dimensionality explains the effectiveness of language model fine-tuning","authors":"Armen Aghajanyan, Sonal Gupta","year":"2021","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Allen-Zhu et al., 2019a] Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in over- parameterized neural networks, going beyond two layers. In NeurIPS, 2019.","title":"Learning and generalization in overparameterized neural networks, going beyond two layers","authors":"Allen-Zhu","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Allen-Zhu et al., 2019b] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. In ICML, 2019.","title":"A convergence theory for deep learning via over-parameterization","authors":"Allen-Zhu","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Arora et al., 2018] Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In ICML, 2018.","title":"On the optimization of deep networks: Implicit acceleration by overparameterization","authors":"Sanjeev Arora, Nadav Cohen, Elad Hazan","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bachlechner et al., 2021] Thomas Bachlechner, Bod- hisattwa Prasad Majumder, et al. Rezero is all you need: Fast convergence at large depth. In UAI, 2021.","title":"Rezero is all you need: Fast convergence at large depth","authors":"Thomas Bachlechner, Bodhisattwa Prasad Majumder","year":"2021","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bao et al., 2022] Hangbo Bao, Li Dong, et al. Beit: Bert pre-training of image transformers. In ICLR, 2022.","title":"Beit: Bert pre-training of image transformers","authors":"Bao","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Beaumont et al., 2021] Olivier Beaumont, Eyraud-Dubois, et al. Efficient combination of rematerialization and of- floading for training dnns. In NeurIPS, 2021.","title":"Efficient combination of rematerialization and offloading for training dnns","authors":"Beaumont","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bian et al., 2021] Zhengda Bian, Hongxin Liu, et al. Colossal-ai: A unified deep learning system for large-scale parallel training. arXiv preprint arXiv:2110.14883, 2021.","title":"Colossal-ai: A unified deep learning system for large-scale parallel training","authors":"Zhengda Bian, Hongxin Liu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Bottou et al., 2018] Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM Review, 2018.","title":"Optimization methods for large-scale machine learning","authors":"Bottou","year":"2018","venue":"SIAM Review","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, et al. Language models are few-shot learners. In NeurIPS, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Cambier et al., 2020] Léopold Cambier, Anahita Bhiwandi- walla, et al. Shifted and squeezed 8-bit floating point for- mat for low-precision training of deep neural networks. In ICLR, 2020.","title":"Shifted and squeezed 8-bit floating point format for low-precision training of deep neural networks","authors":"Léopold Cambier, Anahita Bhiwandiwalla","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Chen et al., 2016] Tianqi Chen, Bing Xu, et al. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016.","title":"Training deep nets with sublinear memory cost","authors":"Tianqi Chen, Bing Xu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Chen et al., 2021a] Jianfei Chen, Lianmin Zheng, et al. Actnn: Reducing training memory footprint via 2-bit acti- vation compressed training. In ICML, 2021.","title":"Actnn: Reducing training memory footprint via 2-bit activation compressed training","authors":"Chen","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Chen et al., 2021b] Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In ICCV, 2021.","title":"Autoformer: Searching transformers for visual recognition","authors":"Chen","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Chen et al., 2021c] Tianlong Chen, Yu Cheng, et al. Chas- ing sparsity in vision transformers: An end-to-end explo- ration. In NeurIPS, 2021.","title":"Chasing sparsity in vision transformers: An end-to-end exploration","authors":"Chen","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Chen et al., 2021d] Xiaohan Chen, Yu Cheng, et al. Early- bert: Efficient bert training via early-bird lottery tickets. In ACL, 2021.","title":"Earlybert: Efficient bert training via early-bird lottery tickets","authors":"Chen","year":"2021","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Chen et al., 2022] Xiangning Chen, Cho-Jui Hsieh, et al. When vision transformers outperform resnets without pre- training or strong data augmentations. In ICLR, 2022.","title":"When vision transformers outperform resnets without pretraining or strong data augmentations","authors":"Xiangning Chen, Cho-Jui Hsieh","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Chen et al., 2023] Xiangning Chen, Chen Liang, et al. Sym- bolic discovery of optimization algorithms. arXiv preprint arXiv:2302.06675, 2023.","title":"Symbolic discovery of optimization algorithms","authors":"Xiangning Chen, Chen Liang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Cordonnier et al., 2020] Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self-attention and convolutional layers. In ICLR, 2020.","title":"On the relationship between self-attention and convolutional layers","authors":"Cordonnier","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Dao et al., 2022] Tri Dao, Daniel Y. Fu, et al. FlashAtten- tion: Fast and memory-efficient exact attention with IO- awareness. In NeurIPS, 2022.","title":"FlashAttention: Fast and memory-efficient exact attention with IOawareness","authors":"Tri Dao, Daniel Y Fu","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Das et al., 2018] Dipankar Das, Naveen Mellempudi, et al. Mixed precision training of convolutional neural networks using integer operations. In ICLR, 2018.","title":"Mixed precision training of convolutional neural networks using integer operations","authors":"Dipankar Das, Naveen Mellempudi","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[De and Smith, 2020] Soham De and Sam Smith. Batch normalization biases residual blocks towards the identity function in deep networks. In NeurIPS, 2020.","title":"Soham De and Sam Smith. Batch normalization biases residual blocks towards the identity function in deep networks","authors":"Smith De","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Du et al., 2022a] Jiawei Du, Hanshu Yan, et al. Efficient sharpness-aware minimization for improved training of neural networks. In ICLR, 2022.","title":"Efficient sharpness-aware minimization for improved training of neural networks","authors":"Du","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Du et al., 2022b] Jiawei Du, Daquan Zhou, et al. Sharpness-aware training for free. In NeurIPS, 2022.","title":"Sharpness-aware training for free","authors":"Du","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"['Ascoli et al., 2021] Stéphane d'Ascoli, Hugo Touvron, et al. Convit: Improving vision transformers with soft con- volutional inductive biases. In ICML, 2021.","title":"Convit: Improving vision transformers with soft convolutional inductive biases","authors":"Ascoli","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Evci et al., 2020] Utku Evci, Trevor Gale, et al. Rigging the lottery: Making all tickets winners. In ICML, 2020.","title":"Rigging the lottery: Making all tickets winners","authors":"Utku Evci, Trevor Gale","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Fedus et al., 2022] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parame- ter models with simple and efficient sparsity. JMLR, 2022.","title":"Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity","authors":"William Fedus, Barret Zoph, Noam Shazeer","year":"2022","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Foret et al., 2021] Pierre Foret, Ariel Kleiner, et al. Sharpness-aware minimization for efficiently improving generalization. In ICLR, 2021.","title":"Sharpness-aware minimization for efficiently improving generalization","authors":"Foret","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Frankle and Carbin, 2019] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2019.","title":"The lottery ticket hypothesis: Finding sparse, trainable neural networks","authors":"Carbin Frankle, Jonathan Frankle, Michael Carbin","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Gong et al., 2019] Linyuan Gong, Di He, et al. Efficient training of bert by progressively stacking. In ICML, 2019.","title":"Efficient training of bert by progressively stacking","authors":"Gong","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Goyal et al., 2017] Priya Goyal, Piotr Dollár, et al. Accu- rate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.","title":"Accurate, large minibatch sgd: Training imagenet in 1 hour","authors":"Priya Goyal, Piotr Dollár","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Gu et al., 2021] Xiaotao Gu, Liyuan Liu, et al. On the trans- former growth for progressive bert training. In NAACL, 2021.","title":"On the transformer growth for progressive bert training","authors":"Gu","year":"2021","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Ham et al., 2020] Tae Jun Ham, Sung Jun Jung, et al. Aˆ 3: Accelerating attention mechanisms in neural networks with approximation. In HPCA, 2020.","title":"Aˆ 3: Accelerating attention mechanisms in neural networks with approximation","authors":"Ham","year":"2020","venue":"HPCA","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Ham et al., 2021] Tae Jun Ham, Yejin Lee, et al. Elsa: Hardware-software co-design for efficient, lightweight self-attention mechanism in neural networks. In ISCA, 2021.","title":"Elsa: Hardware-software co-design for efficient, lightweight self-attention mechanism in neural networks","authors":"Ham","year":"2021","venue":"ISCA","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[He et al., 2022] Kaiming He, Xinlei Chen, et al. Masked autoencoders are scalable vision learners. In CVPR, 2022.","title":"Masked autoencoders are scalable vision learners","authors":"He","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Herrmann et al., 2019] Julien Herrmann, Olivier Beau- mont, et al. Optimal checkpointing for heterogeneous chains: how to train deep neural networks with limited memory. arXiv preprint arXiv:1911.13214, 2019.","title":"Optimal checkpointing for heterogeneous chains: how to train deep neural networks with limited memory","authors":"Julien Herrmann, Olivier Beaumont","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Hoffmann et al., 2022] Jordan Hoffmann, Sebastian Borgeaud, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.","title":"Training compute-optimal large language models","authors":"Jordan Hoffmann, Sebastian Borgeaud","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Houlsby et al., 2019] Neil Houlsby, Andrei Giurgiu, et al. Parameter-efficient transfer learning for nlp. In ICML, 2019.","title":"Parameter-efficient transfer learning for nlp","authors":"Neil Houlsby, Andrei Giurgiu","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Hu et al., 2022] Edward J Hu, yelong shen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022.","title":"Lora: Low-rank adaptation of large language models","authors":"J Edward, Hu","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Huang et al., 2019] Yanping Huang, Youlong Cheng, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In NeurIPS, 2019.","title":"Gpipe: Efficient training of giant neural networks using pipeline parallelism","authors":"Yanping Huang, Youlong Cheng","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Huang et al., 2020] Xiao Shi Huang, Felipe Perez, et al. Im- proving transformer optimization through better initializa- tion. In ICML, 2020.","title":"Improving transformer optimization through better initialization","authors":"Xiao Shi Huang, Felipe Perez","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Hubara et al., 2017] Itay Hubara, Matthieu Courbariaux, et al. Quantized neural networks: Training neural net- works with low precision weights and activations. JMLR, 2017.","title":"Quantized neural networks: Training neural networks with low precision weights and activations","authors":"Itay Hubara, Matthieu Courbariaux","year":"2017","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Jia et al., 2022] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.","title":"Visual prompt tuning","authors":"Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, Ser-Nam Lim","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Johnson and Guestrin, 2018] Tyler B Johnson and Carlos Guestrin. Training deep models faster with robust, approx- imate importance sampling. In NeurIPS, 2018.","title":"Training deep models faster with robust, approximate importance sampling","authors":"B Tyler, Carlos Johnson, Guestrin","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Jouppi et al., 2017] Norman P Jouppi, Cliff Young, et al. In-datacenter performance analysis of a tensor processing unit. In ISCA, 2017.","title":"In-datacenter performance analysis of a tensor processing unit","authors":"Cliff Norman P Jouppi, Young","year":"2017","venue":"ISCA","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Kaddour, 2022] Jean Kaddour. Stop wasting my time! sav- ing days of imagenet and bert training with latest weight averaging. In NeurIPS workshop, 2022.","title":"Stop wasting my time! saving days of imagenet and bert training with latest weight averaging","authors":"Jean Kaddour, Kaddour","year":"2022","venue":"NeurIPS workshop","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Katharopoulos and Fleuret, 2018] Angelos Katharopoulos and Franc ¸ois Fleuret. Not all samples are created equal: Deep learning with importance sampling. In ICML, 2018.","title":"Angelos Katharopoulos and Franc ¸ois Fleuret. Not all samples are created equal: Deep learning with importance sampling","authors":"Fleuret Katharopoulos","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Kenton and Toutanova, 2019] Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton, Lee Kristina, Toutanova","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Kim et al., 2020] Chiheon Kim, Heungsub Lee, et al. torchgpipe: On-the-fly pipeline parallelism for training gi- ant models. arXiv preprint arXiv:2004.09910, 2020.","title":"torchgpipe: On-the-fly pipeline parallelism for training giant models","authors":"Kim","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Kingma and Ba, 2015] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Lan et al., 2020] Zhenzhong Lan, Mingda Chen, et al. Al- bert: A lite bert for self-supervised learning of language representations. In ICLR, 2020.","title":"Albert: A lite bert for self-supervised learning of language representations","authors":"Zhenzhong Lan, Mingda Chen","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Lee et al., 2019] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based on connection sensitivity. In ICLR, 2019.","title":"Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based on connection sensitivity","authors":"Lee","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Lester et al., 2021] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP, 2021.","title":"The power of scale for parameter-efficient prompt tuning","authors":"Brian Lester, Rami Al-Rfou, Noah Constant","year":"2021","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Li and Liang, 2018] Yuanzhi Li and Yingyu Liang. Learn- ing overparameterized neural networks via stochastic gra- dient descent on structured data. In NeurIPS, 2018.","title":"Learning overparameterized neural networks via stochastic gradient descent on structured data","authors":"Li, Liang ; ] Yuanzhi Li, Yingyu Liang","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":55,"raw":"[Li et al., 2017] Qunwei Li, Yi Zhou, et al. Convergence analysis of proximal gradient with momentum for noncon- vex optimization. In ICLR, 2017.","title":"Convergence analysis of proximal gradient with momentum for nonconvex optimization","authors":"Li","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":56,"raw":"[Li et al., 2020] Zhuohan Li, Eric Wallace, et al. Train big, then compress: Rethinking model size for efficient train- ing and inference of transformers. In ICML, 2020.","title":"Train big, then compress: Rethinking model size for efficient training and inference of transformers","authors":"Li","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":57,"raw":"[Li et al., 2022a] Changlin Li, Bohan Zhuang, et al. Auto- mated progressive learning for efficient training of vision transformers. In CVPR, 2022.","title":"Automated progressive learning for efficient training of vision transformers","authors":"Li","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":58,"raw":"[Li et al., 2022b] Yanghao Li, Haoqi Fan, et al. Scaling language-image pre-training via masking. arXiv preprint arXiv:2212.00794, 2022.","title":"Scaling language-image pre-training via masking","authors":"Li","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":59,"raw":"[Lin et al., 2020a] Ji Lin, Wei-Ming Chen, et al. Mcunet: Tiny deep learning on iot devices. In NeurIPS, 2020.","title":"Mcunet: Tiny deep learning on iot devices","authors":"Lin","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":60,"raw":"[Lin et al., 2020b] Zhouchen Lin, Huan Li, and Cong Fang. Accelerated optimization for machine learning. Nature Singapore: Springer, 2020.","title":"Accelerated optimization for machine learning","authors":"Lin","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":61,"raw":"[Liu et al., 2022] Xiaoxuan Liu, Lianmin Zheng, et al. GACT: activation compressed training for generic network architectures. In ICML, 2022.","title":"GACT: activation compressed training for generic network architectures","authors":"Liu","year":"2022","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":62,"raw":"[Loshchilov and Hutter, 2019] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.","title":"Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization","authors":"Hutter Loshchilov","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":63,"raw":"[Lu et al., 2021] Liqiang Lu, Yicheng Jin, et al. Sanger: A co-design framework for enabling sparse attention using reconfigurable architecture. In MICRO, 2021.","title":"Sanger: A co-design framework for enabling sparse attention using reconfigurable architecture","authors":"Liqiang Lu, Yicheng Jin","year":"2021","venue":"MICRO","doi":null,"url":null,"internal_url":null},{"position":64,"raw":"[Micikevicius et al., 2018] Paulius Micikevicius, Sharan Narang, et al. Mixed precision training. In ICLR, 2018.","title":"Mixed precision training","authors":"Paulius Micikevicius, Sharan Narang","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":65,"raw":"[Mishra et al., 2021] Asit Mishra, Jorge Albericio Latorre, et al. Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.08378, 2021.","title":"Accelerating sparse deep neural networks","authors":"Asit Mishra, Jorge Albericio Latorre","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":66,"raw":"[Narayanan et al., 2021] Deepak Narayanan, Mohammad Shoeybi, et al. Efficient large-scale language model train- ing on gpu clusters using megatron-lm. In SC, 2021.","title":"Efficient large-scale language model training on gpu clusters using megatron-lm","authors":"Deepak Narayanan, Mohammad Shoeybi","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":67,"raw":"[Nesterov, 1983] Yurii Nesterov. A method for uncon- strained convex minimization problem with the rate of convergence o (1/kˆ2). In Doklady ANSSSR, volume 269, pages 543-547, 1983.","title":"A method for unconstrained convex minimization problem with the rate of convergence o (1/kˆ2)","authors":"Yurii Nesterov","year":"1983","venue":"Doklady ANSSSR","doi":null,"url":null,"internal_url":null},{"position":68,"raw":"[Pal et al., 2018] Subhankar Pal, Jonathan Beaumont, et al. Outerspace: An outer product based sparse matrix multi- plication accelerator. In HPCA, 2018.","title":"Outerspace: An outer product based sparse matrix multiplication accelerator","authors":"Subhankar Pal, Jonathan Beaumont","year":"2018","venue":"HPCA","doi":null,"url":null,"internal_url":null},{"position":69,"raw":"[Pan et al., 2021] Zizheng Pan, Peng Chen, et al. Mesa: A memory-saving training framework for transformers. arXiv preprint arXiv:2111.11124, 2021.","title":"Mesa: A memory-saving training framework for transformers","authors":"Pan","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":70,"raw":"[Paul et al., 2021] Mansheej Paul, Surya Ganguli, et al. Deep learning on a data diet: Finding important examples early in training. In NeurIPS, 2021.","title":"Deep learning on a data diet: Finding important examples early in training","authors":"Paul","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":71,"raw":"[Qu et al., 2022] Zheng Qu, Liu Liu, et al. Dota: detect and omit weak attentions for scalable transformer acceleration. In ASPLOS, 2022.","title":"Dota: detect and omit weak attentions for scalable transformer acceleration","authors":"Zheng Qu, Liu Liu","year":"2022","venue":"ASPLOS","doi":null,"url":null,"internal_url":null},{"position":72,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":73,"raw":"[Rajbhandari et al., 2020] Samyam Rajbhandari, Jeff Rasley, et al. Zero: Memory optimizations toward training trillion parameter models. In SC, 2020.","title":"Zero: Memory optimizations toward training trillion parameter models","authors":"Rajbhandari","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":74,"raw":"[Rasley et al., 2020] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In KDD, 2020.","title":"Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters","authors":"Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, Yuxiong He","year":"2020","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":75,"raw":"[Ren et al., 2021] Jie Ren, Samyam Rajbhandari, et al. Zero- offload: Democratizing billion-scale model training. In USENIX ATC, 2021.","title":"Zerooffload: Democratizing billion-scale model training","authors":"Jie Ren, Samyam Rajbhandari","year":"2021","venue":"USENIX ATC","doi":null,"url":null,"internal_url":null},{"position":76,"raw":"[Schwartz et al., 2020] Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. Communications of the ACM, 2020.","title":"Green ai","authors":"Roy Schwartz, Jesse Dodge, Noah A Smith, Oren Etzioni","year":"2020","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":77,"raw":"[Shoeybi et al., 2019] Mohammad Shoeybi, Mostofa Pat- wary, et al. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.","title":"Megatron-lm: Training multi-billion parameter language models using model parallelism","authors":"Mohammad Shoeybi, Mostofa Patwary","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":78,"raw":"[Smith et al., 2022] Shaden Smith, Mostofa Patwary, et al. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11990, 2022.","title":"Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model","authors":"Smith","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":79,"raw":"[So et al., 2019] David So, Quoc Le, and Chen Liang. The evolved transformer. In ICML, pages 5877-5886, 2019.","title":"The evolved transformer","authors":"David So, Quoc Le, Chen Liang","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":80,"raw":"[Song et al., 2019] Kaitao Song, Xu Tan, et al. Mass: Masked sequence to sequence pre-training for language generation. In ICML, 2019.","title":"Mass: Masked sequence to sequence pre-training for language generation","authors":"Song","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":81,"raw":"[Sorscher et al., 2022] Ben Sorscher, Robert Geirhos, et al. Beyond neural scaling laws: beating power law scaling via data pruning. In NeurIPS, 2022.","title":"Beyond neural scaling laws: beating power law scaling via data pruning","authors":"Sorscher","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":82,"raw":"[Strubell et al., 2019] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. In ACM, 2019.","title":"Energy and policy considerations for deep learning in nlp","authors":"Emma Strubell, Ananya Ganesh, Andrew Mccallum","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":83,"raw":"[Touvron et al., 2021a] Hugo Touvron, Matthieu Cord, et al. Going deeper with image transformers. In ICCV, 2021.","title":"Going deeper with image transformers","authors":"Touvron","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":84,"raw":"[Touvron et al., 2021b] Hugo Touvron, Matthieu Cord, et al. Training data-efficient image transformers & distillation through attention. In ICML, 2021.","title":"Training data-efficient image transformers & distillation through attention","authors":"Touvron","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":85,"raw":"[Touvron et al., 2023] Hugo Touvron, Thibaut Lavril, Gau- tier Izacard, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.","title":"Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. Llama: Open and efficient foundation language models","authors":"Touvron","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":86,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, et al. Attention is all you need. In NeurIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":87,"raw":"[Wang et al., 2018] Naigang Wang, Jungwook Choi, et al. Training deep neural networks with 8-bit floating point numbers. In NeurIPS, 2018.","title":"Training deep neural networks with 8-bit floating point numbers","authors":"Wang","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":88,"raw":"[Wang et al., 2020] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In ICLR, 2020.","title":"Picking winning tickets before training by preserving gradient flow","authors":"Wang","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":89,"raw":"[You et al., 2017] Yang You, Igor Gitman, and Boris Gins- burg. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888, 2017.","title":"Large batch training of convolutional networks","authors":"Yang You, Igor Gitman, Boris Ginsburg","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":90,"raw":"[You et al., 2019] Yang You, Jing Li, et al. Large batch op- timization for deep learning: Training bert in 76 minutes. In ICLR, 2019.","title":"Large batch optimization for deep learning: Training bert in 76 minutes","authors":"Yang You, Jing Li","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":91,"raw":"[You et al., 2020] Haoran You, Chaojian Li, et al. Drawing early-bird tickets: Towards more efficient training of deep networks. In ICLR, 2020.","title":"Drawing early-bird tickets: Towards more efficient training of deep networks","authors":"Haoran You, Chaojian Li","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":92,"raw":"[You et al., 2023] Haoran You, Zhanyi Sun, et al. Vitcod: Vi- sion transformer acceleration via dedicated algorithm and accelerator co-design. In HPCA, 2023.","title":"Vitcod: Vision transformer acceleration via dedicated algorithm and accelerator co-design","authors":"Haoran You, Zhanyi Sun","year":"2023","venue":"HPCA","doi":null,"url":null,"internal_url":null},{"position":93,"raw":"[Zaken et al., 2022] Elad Ben Zaken, Yoav Goldberg, et al. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In ACL, 2022.","title":"Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models","authors":"Elad Ben Zaken, Yoav Goldberg","year":"2022","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":94,"raw":"[Zhang and He, 2020] Minjia Zhang and Yuxiong He. Ac- celerating training of transformer-based language models with progressive layer dropping. In NeurIPS, 2020.","title":"Minjia Zhang and Yuxiong He. Accelerating training of transformer-based language models with progressive layer dropping","authors":"He Zhang","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":95,"raw":"[Zhang et al., 2019a] Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning with- out normalization. In ICLR, 2019.","title":"Fixup initialization: Residual learning without normalization","authors":"Zhang","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":96,"raw":"[Zhang et al., 2019b] Jingzhao Zhang, Sai Praneeth Karim- ireddy, et al. Why adam beats sgd for attention models. In OpenReview, 2019.","title":"Why adam beats sgd for attention models","authors":"Zhang","year":"2019","venue":"OpenReview","doi":null,"url":null,"internal_url":null},{"position":97,"raw":"[Zhou et al., 2016] Shuchang Zhou, Yuxin Wu, et al. Dorefa- net: Training low bitwidth convolutional neural net- works with low bitwidth gradients. arXiv preprint arXiv:1606.06160, 2016.","title":"Dorefanet: Training low bitwidth convolutional neural networks with low bitwidth gradients","authors":"Zhou","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}