{"type":"ScholarlyArticle","title":"FABA: An Algorithm for Fast Aggregation against Byzantine Attacks in Distributed Neural Networks","authors":[{"name":"Qi Xia","slug":"qi-xia","given":"Qi","family":"Xia","affiliation":"College of William and Mary , Williamsburg , VA 23185 , USA"},{"name":"Zeyi Tao","slug":"zeyi-tao","given":"Zeyi","family":"Tao","affiliation":"College of William and Mary , Williamsburg , VA 23185 , USA"},{"name":"Zijiang Hao","slug":"zijiang-hao","given":"Zijiang","family":"Hao","affiliation":"College of William and Mary , Williamsburg , VA 23185 , USA"},{"name":"Qun Li","slug":"qun-li","given":"Qun","family":"Li","affiliation":"College of William and Mary , Williamsburg , VA 23185 , USA"}],"abstract":"Many times, training a large scale deep learning neural network on a single machine becomes more and more difficult for a complex network model. Distributed training provides an efficient solution, but Byzantine attacks may occur on participating workers. They may be compromised or suffer from hardware failures. If they upload poisonous gradients, the training will become unstable or even converge to a saddle point. In this paper, we propose FABA, a Fast Aggregation algorithm against Byzantine Attacks, which removes the outliers in the uploaded gradients and obtains gradients that are close to the true gradients. We show the convergence of our algorithm. The experiments demonstrate that our algorithm can achieve similar performance to non-Byzantine case and higher efficiency as compared to previous algorithms.","doi":"10.24963/ijcai.2019/670","url":"https://ijcai.org/proceedings/2019/670","pdf":"https://ijcai.org/proceedings/2019/0670.pdf","markdown":"https://ijcai.org/proceedings/2019/670.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/670","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Multidisciplinary Topics and Applications","track":"Main track","number":670,"pages":{"first":4824,"last":4830},"keywords":["Machine Learning: Deep Learning","Multidisciplinary Topics and Applications: Security and Privacy"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2016] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vi- jay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: A system for large-scale machine learning. In Proceedings of OSDI, OSDI'16, pages 265-283, Berkeley, CA, USA, 2016. USENIX As- sociation.","title":"Tensorflow: A system for large-scale machine learning","authors":"Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, Xiaoqiang Zheng","year":"2016","venue":"Proceedings of OSDI, OSDI'16","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Alistarh et al., 2018] Dan Alistarh, Zeyuan Allen-Zhu, and Jerry Li. Byzantine stochastic gradient descent. CoRR, abs/1803.08917, 2018.","title":"Byzantine stochastic gradient descent","authors":"Dan Alistarh, Zeyuan Allen-Zhu, Jerry Li","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ba and Caurana, 2013] Lei Jimmy Ba and Rich Caurana. Do deep nets really need to be deep? CoRR, abs/1312.6184, 2013.","title":"Do deep nets really need to be deep?","authors":"Jimmy Lei, Rich Ba, Caurana","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Blanchard et al., 2017] Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 119-129.","title":"Machine learning with adversaries: Byzantine tolerant gradient descent","authors":"Peva Blanchard, Mahdi El El, Rachid Mhamdi, Julien Guerraoui, Stainer","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"Curran Associates, Inc., 2017.","title":null,"authors":"Curran Associates, Inc","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bottou, 1998] Léon Bottou. On-line learning in neural net- works. In David Saad, editor, On-line Learning and Stochastic Approximations, pages 9-42. Cambridge Uni- versity Press, New York, NY, USA, 1998.","title":"On-line learning in neural networks","authors":"Léon Bottou, Bottou","year":"1998","venue":"On-line Learning and Stochastic Approximations","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2017] Yudong Chen, Lili Su, and Jiaming Xu. Distributed statistical machine learning in adversarial set- tings: Byzantine gradient descent. Proc. ACM Meas. Anal. Comput. Syst., 1(2):44:1-44:25, December 2017.","title":"Distributed statistical machine learning in adversarial settings: Byzantine gradient descent","authors":"Yudong Chen, Lili Su, Jiaming Xu","year":"2017","venue":"Proc. ACM Meas. Anal. Comput. Syst","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Christian Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and An- drew Rabinovich. Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-9, June 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Damaskinos et al., 2018] Georgios Damaskinos, El Mahdi El Mhamdi, Rachid Guerraoui, Rhicheek Patra, and Mahsa Taziki. Asynchronous Byzantine machine learn- ing (the case of SGD). In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceed- ings of Machine Learning Research, pages 1145-1154, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.","title":"Asynchronous Byzantine machine learning (the case of SGD)","authors":"Georgios Damaskinos, Mahdi El El, Rachid Mhamdi, Rhicheek Guerraoui, Mahsa Patra, Taziki","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dean et al., 2012] Jeffrey Dean, Greg S. Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, Marc'Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, and Andrew Y. Ng. Large scale distributed deep networks. In Proceedings of the 25th International Confer- ence on Neural Information Processing Systems -Volume 1, NIPS'12, pages 1223-1231, USA, 2012.","title":"Large scale distributed deep networks","authors":"Jeffrey Dean, Greg S Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V Le, Mark Z Mao, Marc'aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, Andrew Y Ng","year":"2012","venue":"Proceedings of the 25th International Conference on Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Curran Asso- ciates Inc. Jia Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li-jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, June 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Curran Associates, Inc Jia Deng","year":"2009","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kaiming He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, June 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Keskar et al., 2016] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. CoRR, abs/1609.04836, 2016.","title":"On large-batch training for deep learning: Generalization gap and sharp minima","authors":"Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak, Peter Tang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krizhevsky, 2009] Alex Krizhevsky. Learning multiple lay- ers of features from tiny images. Technical report, Depart- ment of Computer Science, University of Toronto, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lamport et al., 1982] Leslie Lamport, Robert Shostak, and Marshall Pease. The byzantine generals problem. ACM Trans. Program. Lang. Syst., 4(3):382-401, July 1982.","title":"The byzantine generals problem","authors":"Leslie Lamport, Robert Shostak, Marshall Pease","year":"1982","venue":"ACM Trans. Program. Lang. Syst","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Métivier, 2011] Michel Métivier. Semimartingales: a course on stochastic processes, volume 2. Walter de Gruyter, 2011.","title":"Semimartingales: a course on stochastic processes","authors":"Michel Métivier","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Norman P. Jouppi et al., 2017] Norman P. Jouppi, Cliff Young, and Nishant Patil et al. In-datacenter performance analysis of a tensor processing unit. In 2017 ACM/IEEE 44th Annual International Symposium on Computer Archi- tecture (ISCA), pages 1-12, June 2017.","title":"In-datacenter performance analysis of a tensor processing unit","authors":"Norman P Jouppi, Cliff Young, Nishant Patil","year":"2017","venue":"ACM/IEEE 44th Annual International Symposium on Computer Architecture (ISCA)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Xie et al., 2018] Cong Xie, Oluwasanmi Koyejo, and In- dranil Gupta. Generalized byzantine-tolerant SGD. CoRR, abs/1802.10116, 2018.","title":"Generalized byzantine-tolerant SGD","authors":"Cong Xie, Oluwasanmi Koyejo","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yann Lecun et al., 1998] Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, Nov 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yin et al., 2018] Dong Yin, Yudong Chen, Kannan Ram- chandran, and Peter Bartlett. Byzantine-robust dis- tributed learning: Towards optimal statistical rates. CoRR, abs/1803.01498, 2018.","title":"Byzantine-robust distributed learning: Towards optimal statistical rates","authors":"Dong Yin, Yudong Chen, Kannan Ramchandran, Peter Bartlett","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}