{"type":"ScholarlyArticle","title":"Survey on Efficient Training of Large Neural Networks","authors":[{"name":"Julia Gusak","slug":"julia-gusak","given":"Julia","family":"Gusak","affiliation":"Skolkovo Institute of Science and Technology , Russia"},{"name":"Daria Cherniuk","slug":"daria-cherniuk","given":"Daria","family":"Cherniuk","affiliation":"Skolkovo Institute of Science and Technology , Russia"},{"name":"Alena Shilova","slug":"alena-shilova","given":"Alena","family":"Shilova","affiliation":"Inria , University of Lille -CRIStAL , France"},{"name":"Alexandr Katrutsa","slug":"alexandr-katrutsa","given":"Alexandr","family":"Katrutsa","affiliation":"Skolkovo Institute of Science and Technology , Russia; AIRI , Moscow , Russia"},{"name":"Daniel Bershatsky","slug":"daniel-bershatsky","given":"Daniel","family":"Bershatsky","affiliation":"Skolkovo Institute of Science and Technology , Russia"},{"name":"Xunyi Zhao","slug":"xunyi-zhao","given":"Xunyi","family":"Zhao","affiliation":"Inria , University of Bordeaux , France"},{"name":"Lionel Eyraud-Dubois","slug":"lionel-eyraud-dubois","given":"Lionel","family":"Eyraud-Dubois","affiliation":"Inria , University of Bordeaux , France"},{"name":"Oleh Shliazhko","slug":"oleh-shliazhko","given":"Oleh","family":"Shliazhko","affiliation":"Sber , Russia"},{"name":"Denis Dimitrov","slug":"denis-dimitrov","given":"Denis","family":"Dimitrov","affiliation":"Sber , Russia"},{"name":"Ivan Oseledets","slug":"ivan-oseledets","given":"Ivan","family":"Oseledets","affiliation":"Skolkovo Institute of Science and Technology , Russia; AIRI , Moscow , Russia"},{"name":"Olivier Beaumont","slug":"olivier-beaumont","given":"Olivier","family":"Beaumont","affiliation":"Inria , University of Bordeaux , France"}],"abstract":"Modern Deep Neural Networks (DNNs) require significant memory to store weight, activations, and other intermediate tensors during training. Hence, many models don’t fit one GPU device or can be trained using only a small per-GPU batch size. This survey provides a systematic overview of the approaches that enable more efficient DNNs training. We analyze techniques that save memory and make good use of computation and communication resources on architectures with a single or several GPUs. We summarize the main categories of strategies and compare strategies within and across categories. Along with approaches proposed in the literature, we discuss available implementations.","doi":"10.24963/ijcai.2022/769","url":"https://ijcai.org/proceedings/2022/769","pdf":"https://ijcai.org/proceedings/2022/0769.pdf","markdown":"https://ijcai.org/proceedings/2022/769.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/769","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Survey Track","track":"Survey Track","number":769,"pages":{"first":5494,"last":5501},"keywords":["Survey Track: Natural Language Processing","Survey Track: Computer Vision","Survey Track: Machine Learning","Survey Track: -"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987312","length_sec":99,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985455","length_sec":700,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Alistarh et al., 2017] Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding. In NIPS, 2017.","title":"QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding","authors":"Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, Milan Vojnovic","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alistarh et al., 2019] Dan Alistarh, Torsten Hoefler, Mikael Johansson, Sarit Kririrat, Nikola Konstantinov, and Cedric Renggli. The Convergence of Sparsified Gradient Meth- ods. In NeurIPS, pages 5973-5983, 2019.","title":"The Convergence of Sparsified Gradient Methods","authors":"Dan Alistarh, Torsten Hoefler, Mikael Johansson, Sarit Kririrat, Nikola Konstantinov, Cedric Renggli","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Athlur et al., 2021] Sanjith Athlur, Nitika Saran, Muthian Sivathanu, Ramachandran Ramjee, and Nipun Kwatra. Varuna: Scalable, Low-cost Training of Massive Deep Learning Models. arXiv:2111.04007, 2021.","title":"Varuna: Scalable, Low-cost Training of Massive Deep Learning Models","authors":"Athlur","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Beaumont et al., 2019] Olivier Beaumont, Lionel Eyraud- Dubois, Julien Herrmann, Alexis Joly, and Alena Shilova. Optimal Checkpointing for Heterogeneous Chains: How to Train Deep Neural Networks with Limited Memory. Re- search Report RR-9302, INRIA, 2019.","title":"Optimal Checkpointing for Heterogeneous Chains: How to Train Deep Neural Networks with Limited Memory","authors":"Olivier Beaumont, Lionel Eyraud-Dubois, Julien Herrmann, Alexis Joly, Alena Shilova","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Beaumont et al., 2020] Olivier Beaumont, Lionel Eyraud- Dubois, and Alena Shilova. Optimal GPU-CPU Offload- ing Strategies for Deep Neural Network Training. In Euro- Par, 2020.","title":"Optimal GPU-CPU Offloading Strategies for Deep Neural Network Training","authors":"Beaumont","year":"2020","venue":"Euro-Par","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Beaumont et al., 2021a] Olivier Beaumont, Lionel Eyraud- Dubois, and Alena Shilova. Efficient Combination of Re- materialization and Offloading for Training DNNs. In NeurIPS, 2021.","title":"Efficient Combination of Rematerialization and Offloading for Training DNNs","authors":"Beaumont","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Beaumont et al., 2021b] Olivier Beaumont, Lionel Eyraud- Dubois, and Alena Shilova. Pipelined Model Parallelism: Complexity Results and Memory Considerations. In Euro- Par, pages 183-198, 2021.","title":"Pipelined Model Parallelism: Complexity Results and Memory Considerations","authors":"Beaumont","year":"2021","venue":"Euro-Par","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Bershatsky et al., 2022] Daniel Bershatsky, Aleksandr Mikhalev, Alexandr Katrutsa, Julia Gusak, Daniil Merkulov, and Ivan Oseledets. Memory-Efficient Backpropagation through Large Linear Layers. arXiv:2201.13195, 2022.","title":"Memory-Efficient Backpropagation through Large Linear Layers","authors":"Daniel Bershatsky, Aleksandr Mikhalev, Alexandr Katrutsa, Julia Gusak, Daniil Merkulov, Ivan Oseledets","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2016] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training Deep Nets with Sublinear Memory Cost. arXiv:1604.06174, 2016.","title":"Training Deep Nets with Sublinear Memory Cost","authors":"Tianqi Chen, Bing Xu, Chiyuan Zhang, Carlos Guestrin","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Cho et al., 2019] Minsik Cho, Vinod Muthusamy, Brad Ne- manich, and Ruchir Puri. GradZip: Gradient Compres- sion using Alternating Matrix Factorization for Large- scale Deep Learning. In Workshop on Systems for ML at NeurIPS'19, 2019.","title":"GradZip: Gradient Compression using Alternating Matrix Factorization for Largescale Deep Learning","authors":"Cho","year":"2019","venue":"Workshop on Systems for ML at NeurIPS'19","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dettmers et al., 2021] Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit Optimizers via Block-wise Quantization. CoRR, 2021.","title":"8-bit Optimizers via Block-wise Quantization","authors":"Tim Dettmers, Mike Lewis, Sam Shleifer, Luke Zettlemoyer","year":"2021","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Fan et al., 2021] Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guop- ing Long, Jun Yang, Lixue Xia, et al. DAPPLE: A Pipelined Data Parallel Approach for Training Large Mod- els. In PPOPP, 2021.","title":"DAPPLE: A Pipelined Data Parallel Approach for Training Large Models","authors":"Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia","year":"2021","venue":"PPOPP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Goyal et al., 2017] Priya Goyal, Piotr Dollár, Ross Gir- shick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Ky- rola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Ac- curate, Large Minibatch SGD. arXiv:1706.02677, 2017.","title":"Large Minibatch SGD","authors":"Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, Kaiming He","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Griewank and Walther, 2000] Andreas Griewank and An- drea Walther. Algorithm 799: Revolve: An Implemen- tation of Checkpointing for the Reverse or Adjoint Mode of Computational Differentiation. ACM TOMS, 2000.","title":"Andreas Griewank and Andrea Walther. Algorithm 799: Revolve: An Implementation of Checkpointing for the Reverse or Adjoint Mode of Computational Differentiation","authors":"Walther Griewank","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Grimm et al., 1996] José Grimm, Loïc Pottier, and Nicole Rostaing-Schmidt. Optimal Time and Minimum Space- Time Product for Reversing a Certain Class of Programs. Technical report, INRIA, 1996. [Gruslys et al., 2016] Audrunas Gruslys, Rémi Munos, Ivo Danihelka, Marc Lanctot, and Alex Graves. Memory- Efficient Backpropagation Through Time. In NIPS, 2016.","title":"Optimal Time and Minimum Space-Time Product for Reversing a Certain Class of Programs","authors":"Grimm","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hrinchuk et al., 2020] Oleksii Hrinchuk, Valentin Khrulkov, Leyla Mirvakhabova, Elena Orlova, and Ivan Oseledets. Tensorized embedding layers. In Find- ings of the Association for Computational Linguistics: EMNLP 2020, pages 4847-4860, Online, November 2020. Association for Computational Linguistics.","title":"Tensorized embedding layers","authors":"Oleksii Hrinchuk, Valentin Khrulkov, Leyla Mirvakhabova, Elena Orlova, Ivan Oseledets","year":"2020","venue":"Findings of the Association for Computational Linguistics: EMNLP 2020","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Huang et al., 2019] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, Hyouk Joong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism. In NeurIPS, 2019.","title":"GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism","authors":"Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, Joong Hyouk, Jiquan Lee, Quoc V Ngiam, Yonghui Le, Zhifeng Wu, Chen","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Huang et al., 2020] Chien-Chin Huang, Gu Jin, and Jinyang Li. SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping. In ASPLOS, 2020.","title":"SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping","authors":"Chien-Chin Huang, Gu Jin, Jinyang Li","year":"2020","venue":"ASPLOS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Jain et al., 2020a] Arpan Jain, Ammar Ahmad Awan, As- maa M. Aljuhani, Jahanzeb Maqbool Hashmi, Quentin G. Anthony, Hari Subramoni, Dhableswar K. Panda, Raghu Machiraju, and Anil Parwani. GEMS: GPU-Enabled Memory-Aware Model-Parallelism System for Distributed DNN Training. In SC, 2020.","title":"GEMS: GPU-Enabled Memory-Aware Model-Parallelism System for Distributed DNN Training","authors":"Jain","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Jain et al., 2020b] Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Joseph Gonza- lez, Kurt Keutzer, and Ion Stoica. Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization. In MLSys, 2020.","title":"Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization","authors":"Jain","year":"2020","venue":"MLSys","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Keskar et al., 2016] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. arXiv:1609.04836, 2016.","title":"On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima","authors":"Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak, Peter Tang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Kirisame et al., 2020] Marisa Kirisame, Steven Lyubomirsky, Altan Haan, Jennifer Brennan, Mike He, Jared Roesch, Tianqi Chen, and Zachary Tatlock. Dynamic Tensor Rematerialization. arXiv:2006.09616, 2020.","title":"Dynamic Tensor Rematerialization","authors":"Marisa Kirisame, Steven Lyubomirsky, Altan Haan, Jennifer Brennan, Mike He, Jared Roesch, Tianqi Chen, Zachary Tatlock","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Kumar et al., 2019] Ravi Kumar, Manish Purohit, Zoya Svitkina, Erik Vee, and Joshua Wang. Efficient Remate- rialization for Deep Networks. In NeurIPS, 2019.","title":"Efficient Rematerialization for Deep Networks","authors":"Ravi Kumar, Manish Purohit, Zoya Svitkina, Erik Vee, Joshua Wang","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Kusumoto et al., 2019] Mitsuru Kusumoto, Takuya Inoue, Gentaro Watanabe, Takuya Akiba, and Masanori Koyama. A Graph Theoretic Framework of Recomputation Al- gorithms for Memory-Efficient Backpropagation. In NeurIPS, 2019.","title":"A Graph Theoretic Framework of Recomputation Algorithms for Memory-Efficient Backpropagation","authors":"Mitsuru Kusumoto, Takuya Inoue, Gentaro Watanabe, Takuya Akiba, Masanori Koyama","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Le et al., 2018] Tung D. Le, Haruki Imai, Yasushi Negishi, and Kiyokuni Kawachiya. TFLMS: Large Model Support in TensorFlow by Graph Rewriting. arXiv:1807.02037, 2018.","title":"TFLMS: Large Model Support in TensorFlow by Graph Rewriting","authors":"D Tung, Haruki Le, Yasushi Imai, Kiyokuni Negishi, Kawachiya","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Li and Hoefler, 2021] Shigang Li and Torsten Hoefler. Chimera: Efficiently Training Large-Scale Neural Net- works with Bidirectional Pipelines. In SC, pages 1-14, 2021.","title":"Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines","authors":"Shigang Li, Torsten Hoefler","year":"2021","venue":"SC","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Li et al., 2021] Conglong Li, Ammar Ahmad Awan, Han- lin Tang, Samyam Rajbhandari, and Yuxiong He. 1- bit LAMB: Communication Efficient Large-Scale Large- Batch Training with LAMB's Convergence Speed. arXiv preprint arXiv:2104.06069, 2021.","title":"1bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed","authors":"Li","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Lin et al., 2021] Junyang Lin, An Yang, Jinze Bai, Chang Zhou, Le Jiang, Xianyan Jia, Ang Wang, Jie Zhang, Yong Li, Wei Lin, et al. M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretrain- ing. arXiv:2110.03888, 2021.","title":"M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining","authors":"Lin","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Narayanan et al., 2019] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Za- haria. Pipedream: Generalized Pipeline Parallelism for DNN Training. In SOSP, 2019.","title":"Pipedream: Generalized Pipeline Parallelism for DNN Training","authors":"Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, Matei Zaharia","year":"2019","venue":"SOSP","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Narayanan et al., 2021a] Deepak Narayanan, Amar Phan- ishayee, Kaiyu Shi, Xie Chen, and Matei Zaharia. Memory-Efficient Pipeline-Parallel DNN Training. In ICML, 2021.","title":"Memory-Efficient Pipeline-Parallel DNN Training","authors":"Narayanan","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Narayanan et al., 2021b] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Pat- wary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient Large- Scale Language Model Training on GPU Clusters Using Megatron-LM. In SC, pages 1-15, 2021.","title":"Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM","authors":"Narayanan","year":"2021","venue":"SC","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Novikov et al., 2022] Georgii Novikov, Daniel Bershatsky, Julia Gusak, Alex Shonenkov, Denis Dimitrov, and Ivan Oseledets. Few-Bit Backward: Quantized Gradients of Activation Functions for Memory Footprint Reduction. arXiv:2202.00441, 2022.","title":"Few-Bit Backward: Quantized Gradients of Activation Functions for Memory Footprint Reduction","authors":"Georgii Novikov, Daniel Bershatsky, Julia Gusak, Alex Shonenkov, Denis Dimitrov, Ivan Oseledets","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Park et al., 2020] Jay H. Park, Gyeongchan Yun, Chang M. Yi, Nguyen T. Nguyen, Seungmin Lee, Jaesik Choi, Sam H. Noh, and Young ri Choi. HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clus- ters through Integration of Pipelined Model Parallelism and Data Parallelism. In USENIX ATC, 2020.","title":"HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism","authors":"Jay H Park, Gyeongchan Yun, Chang M Yi, T Nguyen, Seungmin Nguyen, Jaesik Lee, Sam H Choi, Young Ri Noh, Choi","year":"2020","venue":"USENIX ATC","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Pudipeddi et al., 2020] Bharadwaj Pudipeddi, Maral Mes- makhosroshahi, Jinwen Xi, and Sujeeth Bharadwaj. Train- ing Large Neural Networks with Constant Memory using a New Execution Algorithm. arXiv:2002.05645, 2020.","title":"Training Large Neural Networks with Constant Memory using a New Execution Algorithm","authors":"Bharadwaj Pudipeddi, Maral Mesmakhosroshahi, Jinwen Xi, Sujeeth Bharadwaj","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Rajbhandari et al., 2020] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory Opti- mizations toward Training Trillion Parameter Models. In SC, pages 1-16, 2020.","title":"ZeRO: Memory Optimizations toward Training Trillion Parameter Models","authors":"Rajbhandari","year":"2020","venue":"SC","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Ren et al., 2021] Jie Ren, Samyam Rajbhandari, Reza Yaz- dani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. ZeRO- Offload: Democratizing Billion-Scale Model Training. arXiv:2101.06840, 1 2021.","title":"ZeRO-Offload: Democratizing Billion-Scale Model Training","authors":"Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, Yuxiong He","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Rhu et al., 2016] Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, and Stephen W. Keckler. vDNN: Virtualized Deep Neural Networks for Scalable, Memory-Efficient Neural Network Design. In MICRO, 2016.","title":"vDNN: Virtualized Deep Neural Networks for Scalable, Memory-Efficient Neural Network Design","authors":"Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, Stephen W Keckler","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Siskind and Pearlmutter, 2018] Jeffrey Mark Siskind and Barak A. Pearlmutter. Divide-and-Conquer Checkpoint- ing for Arbitrary Programs with no User Annotation. Op- timization Methods and Software, 2018.","title":"Divide-and-Conquer Checkpointing for Arbitrary Programs with no User Annotation","authors":"Pearlmutter ; Siskind, Mark Jeffrey, Barak A Siskind, Pearlmutter","year":"2018","venue":"Optimization Methods and Software","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Stich et al., 2018] Sebastian U. Stich, Jean-Baptiste Cor- donnier, and Martin Jaggi. Sparsified SGD with Memory. In NIPS, pages 4452-4463, 2018.","title":"Sparsified SGD with Memory","authors":"Sebastian U Stich, Jean-Baptiste Cordonnier, Martin Jaggi","year":"2018","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Sun et al., 2020] Xiao Sun, Naigang Wang, Chia-Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swa- gath Venkataramani, Kaoutar El Maghraoui, Vijayalak- shmi (Viji) Srinivasan, and Kailash Gopalakrishnan. Ultra- Low Precision 4-bit Training of Deep Neural Networks. In NeurIPS, volume 33, pages 1796-1807, 2020.","title":"Ultra-Low Precision 4-bit Training of Deep Neural Networks","authors":"Xiao Sun, Naigang Wang, Chia-Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swagath Venkataramani, Kaoutar El Maghraoui, ( Vijayalakshmi, Kailash Viji) Srinivasan, Gopalakrishnan","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Tang et al., 2021] Hanlin Tang, Shaoduo Gan, Ammar Ah- mad Awan, Samyam Rajbhandari, Conglong Li, Xian- gru Lian, Ji Liu, Ce Zhang, and Yuxiong He. 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed. In ICML, 2021.","title":"1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed","authors":"Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, Yuxiong He","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Tarnawski et al., 2021] Jakub M Tarnawski, Deepak Narayanan, and Amar Phanishayee. Piper: Multidimen- sional Planner for DNN Parallelization. In NeurIPS, volume 34, 2021.","title":"Piper: Multidimensional Planner for DNN Parallelization","authors":"Deepak Jakub M Tarnawski, Amar Narayanan, Phanishayee","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Vogels et al., 2019] Thijs Vogels, Sai Praneeth Karinireddy, and Martin Jaggi. PowerSGD: Practical low-rank gradi- ent compression for distributed optimization. In NeurIPS, 2019.","title":"PowerSGD: Practical low-rank gradient compression for distributed optimization","authors":"Thijs Vogels, Sai Praneeth Karinireddy, Martin Jaggi","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Walther and Griewank, 2008] Andrea Walther and Andreas Griewank. Evaluating Derivatives: Principles and Tech- niques of Algorithmic Differentiation. SIAM, 2008.","title":"Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation","authors":"Andrea Walther, Andreas Griewank","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Xu et al., 2021] Hang Xu, Chen-Yu Ho, Ahmed M Abdel- moniem, Aritra Dutta, El Houcine Bergou, Konstantinos Karatsenidis, Marco Canini, and Panos Kalnis. GRACE: A compressed communication framework for distributed machine learning. In ICDCS, pages 561-572, 2021.","title":"GRACE: A compressed communication framework for distributed machine learning","authors":"Hang Xu, Chen-Yu Ho, Ahmed M Abdelmoniem, Aritra Dutta, Houcine El, Konstantinos Bergou, Marco Karatsenidis, Panos Canini, Kalnis","year":"2021","venue":"ICDCS","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Yang et al., 2021] Bowen Yang, Jian Zhang, Jonathan Li, Christopher Re, Christopher Aberger, and Christopher De Sa. PipeMare: Asynchronous Pipeline Parallel DNN Training. In MLSys, 2021.","title":"PipeMare: Asynchronous Pipeline Parallel DNN Training","authors":"Bowen Yang, Jian Zhang, Jonathan Li, Christopher Re, Christopher Aberger, Christopher De Sa","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[You et al., 2017] Yang You, Igor Gitman, and Boris Gins- burg. Large Batch Training of Convolutional Networks. arXiv:1708.03888, 2017.","title":"Large Batch Training of Convolutional Networks","authors":"Yang You, Igor Gitman, Boris Ginsburg","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[You et al., 2019] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xi- aodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large Batch Optimization for Deep Learning. arXiv:1904.00962, 2019.","title":"Large Batch Optimization for Deep Learning","authors":"Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, Cho-Jui Hsieh","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Zhan and Zhang, 2019] Jun Zhan and Jinghui Zhang. Pipe- Torch: Pipeline-Based Distributed Deep Learning in a GPU Cluster with Heterogeneous Networking. In CBD, 2019.","title":"Pipe-Torch: Pipeline-Based Distributed Deep Learning in a GPU Cluster with Heterogeneous Networking","authors":"Zhang Zhan, Jun Zhan, Jinghui Zhang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Zhang et al., 2019] Junzhe Zhang, Sai Ho Yeung, Yao Shu, Bingsheng He, and Wei Wang. Efficient Mem- ory Management for GPU-based Deep Learning Systems. arXiv:1903.06631, 2019.","title":"Efficient Memory Management for GPU-based Deep Learning Systems","authors":"Zhang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}