{"type":"ScholarlyArticle","title":"OSDP: Optimal Sharded Data Parallel for Distributed Deep Learning","authors":[{"name":"Youhe Jiang","slug":"youhe-jiang","given":"Youhe","family":"Jiang","affiliation":"School of CS & Key Lab of High Confidence Software Technologies (MOE) , Peking University"},{"name":"Fangcheng Fu","slug":"fangcheng-fu","given":"Fangcheng","family":"Fu","affiliation":"School of CS & Key Lab of High Confidence Software Technologies (MOE) , Peking University"},{"name":"Xupeng Miao","slug":"xupeng-miao","given":"Xupeng","family":"Miao","affiliation":"Computer Science Department , Carnegie Mellon University"},{"name":"Xiaonan Nie","slug":"xiaonan-nie","given":"Xiaonan","family":"Nie","affiliation":"School of CS & Key Lab of High Confidence Software Technologies (MOE) , Peking University"},{"name":"Bin Cui","slug":"bin-cui","given":"Bin","family":"Cui","affiliation":"School of CS & Key Lab of High Confidence Software Technologies (MOE) , Peking University; Institute of Computational Social Science , Peking University (Qingdao"}],"abstract":"Large-scale deep learning models contribute to significant performance improvements on varieties of downstream tasks. Current data and model parallelism approaches utilize model replication and partition techniques to support the distributed training of ultra-large models. However, directly deploying these systems often leads to sub-optimal training efficiency due to the complex model architectures and the strict device memory constraints. In this paper, we propose Optimal Sharded Data Parallel (OSDP), an automated parallel training system that combines the advantages from both data and model parallelism. Given the model description and the device information, OSDP makes trade-offs between the memory consumption and the hardware utilization, thus automatically generates the distributed computation graph and maximizes the overall system throughput. In addition, OSDP introduces operator splitting to further alleviate peak memory footprints during training with negligible overheads, which enables the trainability of larger models as well as the higher throughput. Extensive experimental results of OSDP on multiple different kinds of large-scale models demonstrate that the proposed strategy outperforms the state-of-the-art in multiple regards.","doi":"10.24963/ijcai.2023/238","url":"https://ijcai.org/proceedings/2023/238","pdf":"https://ijcai.org/proceedings/2023/0238.pdf","markdown":"https://ijcai.org/proceedings/2023/238.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/238","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Data Mining","track":"Main Track","number":238,"pages":{"first":2142,"last":2150},"keywords":["Data Mining: DM: Big data and scalability","Data Mining: DM: Parallel, distributed and cloud-based high performance mining"],"videos":[],"references":[{"position":1,"raw":"[Brown et al., 2020] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.","title":"Language models are few-shot learners","authors":"Benjamin Tom B Brown, Nick Mann, Melanie Ryder, Jared Subbiah, Prafulla Kaplan, Arvind Dhariwal, Pranav Neelakantan, Girish Shyam, Amanda Sastry, Askell","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cai et al., 2021] Zixian Cai, Zhengyang Liu, Saeed Maleki, Madanlal Musuvathi, Todd Mytkowicz, Jacob Nelson, and Olli Saarikivi. Synthesizing optimal collective algorithms. Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, page 62-75, 2021.","title":"Synthesizing optimal collective algorithms","authors":"Cai","year":"2021","venue":"Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chan et al., 2007] Ernie Chan, Marcel Heimlich, Avi Purkayastha, and Robert Van De Geijn. Collective com- munication: theory, practice, and experience. Concurrency and Computation: Practice and Experience, 19(13):1749- 1783, 2007.","title":"Collective communication: theory, practice, and experience","authors":"Ernie Chan, Marcel Heimlich, Avi Purkayastha, Robert Van De Geijn","year":"2007","venue":"Concurrency and Computation: Practice and Experience","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[ChatGPT authors, 2022] ChatGPT authors. Chatgpt: Opti- mizing language models for dialogue. https://openai.com/ blog/chatgpt, 2022. Accessed: 2022-11-01.","title":"ChatGPT authors. Chatgpt: Optimizing language models for dialogue","authors":null,"year":"2022","venue":null,"doi":null,"url":"https://openai.com/blog/chatgpt,2022","internal_url":null},{"position":5,"raw":"[Chen et al., 2016] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016.","title":"Training deep nets with sublinear memory cost","authors":"Tianqi Chen, Bing Xu, Chiyuan Zhang, Carlos Guestrin","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dean et al., 2012] Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc' aurelio Ran- zato, Andrew Senior, Paul Tucker, Ke Yang, Quoc Le, and Andrew Ng. Large scale distributed deep networks. In Advances in Neural Information Processing Systems, vol- ume 25, 2012.","title":"Large scale distributed deep networks","authors":"Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc' Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, Quoc Le, Andrew Ng","year":"2012","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[FairScale authors, 2021] FairScale authors. Fairscale: A general purpose modular pytorch library for high per- formance and large scale training. https://github.com/ facebookresearch/fairscale, 2021. Accessed: 2022-11-01.","title":"FairScale authors. Fairscale: A general purpose modular pytorch library for high performance and large scale training","authors":null,"year":"2021","venue":null,"doi":null,"url":"https://github.com/facebookresearch/fairscale,2021","internal_url":null},{"position":9,"raw":"[Fu et al., 2020] Fangcheng Fu, Yuzheng Hu, Yihan He, Ji- awei Jiang, Yingxia Shao, Ce Zhang, and Bin Cui. Don't waste your bits! squeeze activations and gradients for deep neural networks via tinyscript. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, volume 119, pages 3304-3314. PMLR, 2020.","title":"Don't waste your bits! squeeze activations and gradients for deep neural networks via tinyscript","authors":"Fangcheng Fu, Yuzheng Hu, Yihan He, Jiawei Jiang, Yingxia Shao, Ce Zhang, Bin Cui","year":"2020","venue":"Proceedings of the 37th International Conference on Machine Learning, ICML 2020","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fu et al., 2022] Fangcheng Fu, Xupeng Miao, Jiawei Jiang, Huanran Xue, and Bin Cui. Towards communication- efficient vertical federated learning training via cache- enabled local update. Proceedings of the VLDB Endowment, 15(10):2111-2120, 2022.","title":"Towards communicationefficient vertical federated learning training via cacheenabled local update","authors":"Fangcheng Fu, Xupeng Miao, Jiawei Jiang, Huanran Xue, Bin Cui","year":"2022","venue":"Proceedings of the VLDB Endowment","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Harlap et al., 2018] Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons. Pipedream: Fast and ef- ficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377, 2018.","title":"Pipedream: Fast and efficient pipeline parallel dnn training","authors":"Harlap","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hockney, 1994] Roger W Hockney. The communication challenge for mpp: Intel paragon and meiko cs-2. Parallel computing, 20(3):389-398, 1994.","title":"The communication challenge for mpp: Intel paragon and meiko cs-2","authors":"W Roger, Hockney","year":"1994","venue":"Parallel computing","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huang et al., 2019] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32:103-112, 2019.","title":"Gpipe: Efficient training of giant neural networks using pipeline parallelism","authors":"Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, Hyoukjoong Lee, Jiquan Ngiam, V Quoc, Yonghui Le, Wu","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jain et al., 2020] Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Joseph Gonzalez, Kurt Keutzer, and Ion Stoica. Checkmate: Breaking the memory wall with optimal tensor rematerialization. Proceedings of Machine Learning and Systems, 2:497-511, 2020.","title":"Checkmate: Breaking the memory wall with optimal tensor rematerialization","authors":"Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Joseph Gonzalez, Kurt Keutzer, Ion Stoica","year":"2020","venue":"Proceedings of Machine Learning and Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kaplan et al., 2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.","title":"Scaling laws for neural language models","authors":"Jared Kaplan, Sam Mccandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2020] Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chin- tala. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006.15704, 2020.","title":"Pytorch distributed: Experiences on accelerating data parallel training","authors":"Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, Soumith Chintala","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2021] Junyang Lin, Rui Men, An Yang, Chang Zhou, Ming Ding, Yichang Zhang, Peng Wang, Ang Wang, Le Jiang, Xianyan Jia, Jie Zhang, Jianwei Zhang, Xu Zou, Zhikang Li, Xiaodong Deng, Jie Liu, Jinbao Xue, Huiling Zhou, Jianxin Ma, Jin Yu, Yong Li, Wei Lin, Jingren Zhou, Jie Tang, and Hongxia Yang. M6: A chinese multimodal pretrainer. arXiv preprint arXiv:2103.00823, 2021.","title":"M6: A chinese multimodal pretrainer","authors":"Lin","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yix- uan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Miao et al., 2022a] Xupeng Miao, Xiaonan Nie, Hailin Zhang, Tong Zhao, and Bin Cui. Hetu: A highly effi- cient automatic parallel distributed deep learning system. Sci. China Inf. Sci., 2022.","title":"Hetu: A highly efficient automatic parallel distributed deep learning system","authors":"Miao","year":"2022","venue":"Sci. China Inf. Sci","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Miao et al., 2022b] Xupeng Miao, Yujie Wang, Youhe Jiang, Chunan Shi, Xiaonan Nie, Hailin Zhang, and Bin Cui. Gal- vatron: Efficient transformer training over multiple gpus using automatic parallelism. Proceedings of the VLDB Endowment, 16:470-479, 2022.","title":"Galvatron: Efficient transformer training over multiple gpus using automatic parallelism","authors":"Miao","year":"2022","venue":"Proceedings of the VLDB Endowment","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Miao et al., 2022c] Xupeng Miao, Hailin Zhang, Yining Shi, Xiaonan Nie, Zhi Yang, Yangyu Tao, and Bin Cui. HET: scaling out huge embedding model training via cache- enabled distributed framework. Proc. VLDB Endow., 15(2):312-320, 2022.","title":"HET: scaling out huge embedding model training via cacheenabled distributed framework","authors":"Miao","year":"2022","venue":"Proc. VLDB Endow","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Miao et al., 2023] Xupeng Miao, Yining Shi, Zhi Yang, Bin Cui, and Zhihao Jia. Sdpipe: A semi-decentralized frame- work for heterogeneity-aware pipeline-parallel training. Proc. VLDB Endow., 16, 2023.","title":"Sdpipe: A semi-decentralized framework for heterogeneity-aware pipeline-parallel training","authors":"Miao","year":"2023","venue":"Proc. VLDB Endow","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Narayanan et al., 2021] Deepak Narayanan, Amar Phan- ishayee, Kaiyu Shi, Xie Chen, and Matei Zaharia. Memory- efficient pipeline-parallel dnn training. In International Conference on Machine Learning, pages 7937-7947.","title":"Memoryefficient pipeline-parallel dnn training","authors":"Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, Matei Zaharia","year":null,"venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"PMLR, 2021.","title":null,"authors":"Pmlr","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Nie et al., 2022] Xiaonan Nie, Xupeng Miao, Zhi Yang, and Bin Cui. Tsplit: Fine-grained gpu memory management for efficient dnn training via tensor splitting. In International Conference on Data Engineering, pages 2615-2628. IEEE, 2022.","title":"Tsplit: Fine-grained gpu memory management for efficient dnn training via tensor splitting","authors":"Xiaonan Nie, Xupeng Miao, Zhi Yang, Bin Cui","year":"2022","venue":"International Conference on Data Engineering","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Nie et al., 2023a] Xiaonan Nie, Yi Liu, Fangcheng Fu, Jin- bao Xue, Dian Jiao, Xupeng Miao, Yangyu Tao, and Bin Cui. Angel-ptm: A scalable and economical large-scale pre-training system in tencent. Proceedings of the VLDB Endowment, 2023.","title":"Angel-ptm: A scalable and economical large-scale pre-training system in tencent","authors":"Nie","year":"2023","venue":"Proceedings of the VLDB Endowment","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Nie et al., 2023b] Xiaonan Nie, Xupeng Miao, Zilong Wang, Zichao Yang, Jilong Xue, Lingxiao Ma, Gang Cao, and Bin Cui. Flexmoe: Scaling large-scale sparse pre-trained model training via dynamic device placement. In SIGMOD. ACM, 2023.","title":"Flexmoe: Scaling large-scale sparse pre-trained model training via dynamic device placement","authors":"Nie","year":"2023","venue":"SIGMOD. ACM","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Paszke et al., 2017] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS 2017 Workshop on Autodiff, 2017.","title":"Automatic differentiation in pytorch","authors":"Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Devito, Zeming Lin, Alban Desmaison, Luca Antiga, Adam Lerer","year":"2017","venue":"NIPS 2017 Workshop on Autodiff","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Peters et al., 2018] Matthew E. Peters, Mark Neumann, Mo- hit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word represen- tations. arXiv preprint arXiv:1802.05365, 2018.","title":"Deep contextualized word representations","authors":"Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. Ope- nAI blog, 1(8):9, 2019.","title":"Language models are unsupervised multitask learners","authors":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever","year":"2019","venue":"Ope-nAI blog","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Raffel et al., 2019] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.","title":"Exploring the limits of transfer learning with a unified text-to-text transformer","authors":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Rajbhandari et al., 2020] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory opti- mizations toward training trillion parameter models. arXiv preprint arXiv:1910.02054, 2020.","title":"Zero: Memory optimizations toward training trillion parameter models","authors":"Rajbhandari","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Rajbhandari et al., 2021] Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. Zero- infinity: Breaking the gpu memory wall for extreme scale deep learning. arXiv preprint arXiv:2104.07857, 2021.","title":"Zeroinfinity: Breaking the gpu memory wall for extreme scale deep learning","authors":"Rajbhandari","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Rasley et al., 2020] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505-3506, 2020.","title":"Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters","authors":"Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, Yuxiong He","year":"2020","venue":"Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Ren et al., 2021] Jie Ren, Samyam Rajbhandari, Reza Yaz- dani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Min- jia Zhang, Dong Li, and Yuxiong He. Zero-offload: De- mocratizing billion-scale model training. arXiv preprint arXiv:2101.06840, 2021.","title":"Zero-offload: Democratizing billion-scale model training","authors":"Ren","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Shallue et al., 2018] Christopher J Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E Dahl. Measuring the effects of data parallelism on neural network training. arXiv preprint arXiv:1811.03600, 2018.","title":"Measuring the effects of data parallelism on neural network training","authors":"Jaehoon Christopher J Shallue, Joseph Lee, Jascha Antognini, Roy Sohl-Dickstein, George E Frostig, Dahl","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Shoeybi et al., 2020] Mohammad Shoeybi, Mostofa Pat- wary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2020.","title":"Megatron-lm: Training multi-billion parameter language models using model parallelism","authors":"Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick Legresley, Jared Casper, Bryan Catanzaro","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Thakur et al., 2005] Rajeev Thakur, Rolf Rabenseifner, and William Gropp. Optimization of collective communication operations in mpich. The International Journal of High Performance Computing Applications, 19(1):49-66, 2005.","title":"Optimization of collective communication operations in mpich","authors":"Rajeev Thakur, Rolf Rabenseifner, William Gropp","year":"2005","venue":"The International Journal of High Performance Computing Applications","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Touvron et al., 2023] Hugo Touvron, Thibaut Lavril, Gau- tier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Ham- bro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.","title":"Hugo Touvron, Thibaut Lavril, Gautier Izacard","authors":"Touvron","year":"2023","venue":"Llama: Open and efficient foundation language models","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Yang et al., 2021] Bowen Yang, Jian Zhang, Jonathan Li, Christopher Re, Christopher Aberger, and Christopher De Sa. Pipemare: Asynchronous pipeline parallel dnn training. In Proceedings of Machine Learning and Systems, volume 3, pages 269-296, 2021.","title":"Pipemare: Asynchronous pipeline parallel dnn training","authors":"Bowen Yang, Jian Zhang, Jonathan Li, Christopher Re, Christopher Aberger, Christopher De Sa","year":"2021","venue":"Proceedings of Machine Learning and Systems","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Zinkevich et al., 2010] Martin Zinkevich, Markus Weimer, Alexander J Smola, and Lihong Li. Parallelized stochastic gradient descent. In NIPS, volume 4, page 4. Citeseer, 2010.","title":"Parallelized stochastic gradient descent","authors":"Martin Zinkevich, Markus Weimer, Alexander J Smola, Lihong Li","year":"2010","venue":"NIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}