{"type":"ScholarlyArticle","title":"Accelerated Asynchronous Greedy Coordinate Descent Algorithm for SVMs","authors":[{"name":"Bin Gu","slug":"bin-gu","given":"Bin","family":"Gu","affiliation":"* School of Computer & Software , Nanjing University of Information Science & Technology , P.R.China; † Department of Electrical & Computer Engineering , University of Pittsburgh , USA"},{"name":"Yingying Shan","slug":"yingying-shan","given":"Yingying","family":"Shan","affiliation":"* School of Computer & Software , Nanjing University of Information Science & Technology , P.R.China"},{"name":"Xiang Geng","slug":"xiang-geng","given":"Xiang","family":"Geng","affiliation":"* School of Computer & Software , Nanjing University of Information Science & Technology , P.R.China"},{"name":"Guansheng Zheng","slug":"guansheng-zheng","given":"Guansheng","family":"Zheng","affiliation":"* School of Computer & Software , Nanjing University of Information Science & Technology , P.R.China"}],"abstract":"Support vector machines play an important role in machine learning in the last two decades. Traditional SVM solvers (e.g. LIBSVM) are not scalable in the current big data era. Recently, a state of the art solver was proposed based on the asynchronous greedy coordinate descent (AsyGCD) algorithm. However, AsyGCD is still not scalable enough, and is limited to binary classification. To address these issues, in this paper we propose an asynchronous accelerated greedy coordinate descent algorithm (AsyAGCD) for  SVMs. Compared with AsyGCD, our AsyAGCD has the following two-fold advantages: 1) our AsyAGCD is an accelerated version of AsyGCD because active set strategy is used. Specifically, our AsyAGCD can converge much faster  than AsyGCD for the second half of iterations. 2) Our AsyAGCD can handle more SVM formulations (including binary classification and regression SVMs) than AsyGCD. We provide the comparison of computational complexity of AsyGCD and our AsyAGCD. Experiment results on a variety of datasets and learning applications confirm that our AsyAGCD is much faster than the existing SVM solvers (including AsyGCD).","doi":"10.24963/ijcai.2018/300","url":"https://ijcai.org/proceedings/2018/300","pdf":"https://ijcai.org/proceedings/2018/0300.pdf","markdown":"https://ijcai.org/proceedings/2018/300.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/300","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":300,"pages":{"first":2170,"last":2176},"keywords":["Machine Learning: Classification","Machine Learning: Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Bertsekas, 1999] Dimitri P Bertsekas. Nonlinear program- ming. Athena scientific Belmont, 1999.","title":"Nonlinear programming","authors":"Dimitri P Bertsekas","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Birgin and Martínez, 2002] Ernesto G Birgin and José Mario Martínez. Large-scale active-set box- constrained optimization method with spectral projected gradients. Computational Optimization and Applications, 23(1):101-125, 2002.","title":"Large-scale active-set boxconstrained optimization method with spectral projected gradients","authors":"G Ernesto, José Birgin, Martínez Mario","year":"2002","venue":"Computational Optimization and Applications","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Boyd and Vandenberghe, 2004] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge universi- ty press, 2004.","title":"Convex optimization","authors":"Vandenberghe Boyd, Stephen Boyd, Lieven Vandenberghe","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chandra, 2001] Rohit Chandra. Parallel programming in OpenMP. Morgan kaufmann, 2001.","title":"Parallel programming in OpenMP","authors":"Rohit Chandra","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chang and Lin, 2011] Chih-Chung Chang and Chih-Jen Lin. Libsvm: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3):27, 2011.","title":"Libsvm: a library for support vector machines","authors":"Chang, Lin ; ] Chih-Chung Chang, Chih-Jen Lin","year":"2011","venue":"ACM Transactions on Intelligent Systems and Technology (TIST)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chiang et al., 2016] Wei-Lin Chiang, Mu-Chu Lee, and Chih-Jen Lin. Parallel dual coordinate descent method for large-scale linear classification in multi-core environ- ments. In KDD, pages 1485-1494, 2016.","title":"Parallel dual coordinate descent method for large-scale linear classification in multi-core environments","authors":"Wei-Lin Chiang, Mu-Chu Lee, Chih-Jen Lin","year":"2016","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[De Santis et al., 2016] Marianna De Santis, Stefano Lucidi, and Francesco Rinaldi. A fast active set block coordinate descent algorithm for ℓ 1 -regularized least squares. SIAM Journal on Optimization, 26(1):781-809, 2016.","title":"A fast active set block coordinate descent algorithm for ℓ 1 -regularized least squares","authors":"Marianna De Santis, Stefano Lucidi, Francesco Rinaldi","year":"2016","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Duchi et al., 2015] John C Duchi, Sorathan Chaturapruek, and Christopher Ré. Asynchronous stochastic convex op- timization. arXiv preprint arXiv:1508.00882, 2015.","title":"Asynchronous stochastic convex optimization","authors":"Sorathan John C Duchi, Christopher Chaturapruek, Ré","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gu and Ling, 2015] Bin Gu and Charles Ling. A new gen- eralized error path algorithm for model selection. In Inter- national Conference on Machine Learning, pages 2549- 2558, 2015.","title":"A new generalized error path algorithm for model selection","authors":"Ling ; ] Bin Gu, Charles Gu, Ling","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gu et al., 2015a] Bin Gu, Victor S Sheng, Keng Yeow Tay, Walter Romano, and Shuo Li. Incremental support vec- tor learning for ordinal regression. IEEE Transactions on Neural networks and learning systems, 26(7):1403-1416, 2015.","title":"Incremental support vector learning for ordinal regression","authors":"Gu","year":"2015","venue":"IEEE Transactions on Neural networks and learning systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gu et al., 2015b] Bin Gu, Victor S Sheng, Zhijie Wang, Derek Ho, Said Osman, and Shuo Li. Incremental learning for ν-support vector regression. Neural Networks, 67:140- 150, 2015.","title":"Incremental learning for ν-support vector regression","authors":"Gu","year":"2015","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gu et al., 2017] Bin Gu, Victor S Sheng, Keng Yeow Tay, Walter Romano, and Shuo Li. Cross validation through two-dimensional solution surface for cost-sensitive svm. IEEE transactions on pattern analysis and machine intel- ligence, 39(6):1103-1121, 2017.","title":"Cross validation through two-dimensional solution surface for cost-sensitive svm","authors":"Bin Gu, S Victor, Keng Yeow Sheng, Walter Tay, Shuo Romano, Li","year":"2017","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gu et al., 2018a] Bin Gu, Zhouyuan Huo, and Heng Huang. Asynchronous doubly stochastic group regularized learn- ing. In International Conference on Artificial Intelligence and Statistics, pages 1791-1800, 2018.","title":"Asynchronous doubly stochastic group regularized learning","authors":"Gu","year":"2018","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gu et al., 2018b] Bin Gu, Xin Miao, Zhouyuan Huo, and Heng Huang. Asynchronous doubly stochastic sparse ker- nel learning. In AAAI, 2018.","title":"Asynchronous doubly stochastic sparse kernel learning","authors":"Gu","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Gu et al., 2018c] Bin Gu, De Wang, Zhouyuan Huo, and Heng Huang. Inexact proximal gradient methods for non- convex and non-smooth optimization. In AAAI, 2018.","title":"Inexact proximal gradient methods for nonconvex and non-smooth optimization","authors":"Gu","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hsieh et al., 2008] Cho-Jui Hsieh, Kai-Wei Chang, Chih- Jen Lin, S Sathiya Keerthi, and Sellamanickam Sundarara- jan. A dual coordinate descent method for large-scale lin- ear svm. In Proceedings of the 25th international confer- ence on Machine learning, pages 408-415. ACM, 2008.","title":"A dual coordinate descent method for large-scale linear svm","authors":"Cho-Jui Hsieh, Kai-Wei Chang, Chih-Jen Lin, S Sathiya Keerthi, Sellamanickam Sundararajan","year":"2008","venue":"Proceedings of the 25th international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Huo et al., 2018] Zhouyuan Huo, Bin Gu, Ji Liu, and Heng Huang. Accelerated method for stochastic composition optimization with nonsmooth regularization. In AAAI, 2018.","title":"Accelerated method for stochastic composition optimization with nonsmooth regularization","authors":"Huo","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2015] Ji Liu, Stephen J Wright, Christopher Ré, Victor Bittorf, and Srikrishna Sridhar. An asynchronous parallel stochastic coordinate descent algorithm. The Jour- nal of Machine Learning Research, 16(1):285-322, 2015.","title":"An asynchronous parallel stochastic coordinate descent algorithm","authors":"Ji Liu, Stephen J Wright, Christopher Ré, Victor Bittorf, Srikrishna Sridhar","year":"2015","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rahimi and Recht, 2008] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Ad- vances in neural information processing systems, pages 1177-1184, 2008.","title":"Random features for large-scale kernel machines","authors":"Recht Rahimi, Ali Rahimi, Benjamin Recht","year":"2008","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Steinwart et al., 2011] Ingo Steinwart, Don Hush, and Clint Scovel. Training svms without offset. Journal of Machine Learning Research, 12(Jan):141-202, 2011.","title":"Training svms without offset","authors":"Ingo Steinwart, Don Hush, Clint Scovel","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vapnik and Vapnik, 1998] Vladimir Naumovich Vapnik and Vlamimir Vapnik. Statistical learning theory, volume 1. Wiley New York, 1998.","title":"Vladimir Naumovich Vapnik and Vlamimir Vapnik","authors":"Vapnik Vapnik","year":"1998","venue":"Statistical learning theory","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Williams and Seeger, 2001] Christopher KI Williams and Matthias Seeger. Using the nyström method to speed up kernel machines. In Advances in neural information pro- cessing systems, pages 682-688, 2001.","title":"Using the nyström method to speed up kernel machines","authors":"K I Christopher, Matthias Williams, Seeger","year":"2001","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[You et al., 2015] Yang You, Haohuan Fu, Shuaiwen Leon Song, Amanda Randles, Darren Kerbyson, Andres Mar- quez, Guangwen Yang, and Adolfy Hoisie. Scaling sup- port vector machines on modern hpc platforms. Journal of Parallel and Distributed Computing, 76:16-31, 2015.","title":"Scaling support vector machines on modern hpc platforms","authors":"Yang You, Haohuan Fu, Shuaiwen, Leon Song, Amanda Randles, Darren Kerbyson, Andres Marquez, Guangwen Yang, Adolfy Hoisie","year":"2015","venue":"Journal of Parallel and Distributed Computing","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[You et al., 2016] Yang You, Xiangru Lian, Ji Liu, Hsiang- Fu Yu, Inderjit S Dhillon, James Demmel, and Cho-Jui Hsieh. Asynchronous parallel greedy coordinate descent. In Advances In Neural Information Processing Systems, pages 4682-4690, 2016.","title":"Asynchronous parallel greedy coordinate descent","authors":"Yang You, Xiangru Lian, Ji Liu, Hsiang-Fu Yu, Inderjit S Dhillon, James Demmel, Cho-Jui Hsieh","year":"2016","venue":"Advances In Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhao and Magoules, 2011] Hai-xiang Zhao and Frédéric Magoules. Parallel support vector machines on multi-core and multiprocessor systems. In 11th International Con- ference on Artificial Intelligence and Applications (AIA 2011). IASTED, 2011.","title":"Parallel support vector machines on multi-core and multiprocessor systems","authors":"Hai-Xiang Zhao, Frédéric Magoules","year":"2011","venue":"11th International Conference on Artificial Intelligence and Applications (AIA 2011)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}