{"type":"ScholarlyArticle","title":"Scalable Optimal Margin Distribution Machine","authors":[{"name":"Yilin Wang","slug":"yilin-wang","given":"Yilin","family":"Wang","affiliation":"National Engineering Research Center for Big Data Technology and System Services Computing Technology and System Lab , Cluster and Grid Computing Lab School of Computer Science and Technology , Huazhong University of Science and Technology , China"},{"name":"Nan Cao","slug":"nan-cao","given":"Nan","family":"Cao","affiliation":"National Engineering Research Center for Big Data Technology and System Services Computing Technology and System Lab , Cluster and Grid Computing Lab School of Computer Science and Technology , Huazhong University of Science and Technology , China"},{"name":"Teng Zhang","slug":"teng-zhang","given":"Teng","family":"Zhang","affiliation":"National Engineering Research Center for Big Data Technology and System Services Computing Technology and System Lab , Cluster and Grid Computing Lab School of Computer Science and Technology , Huazhong University of Science and Technology , China"},{"name":"Xuanhua Shi","slug":"xuanhua-shi","given":"Xuanhua","family":"Shi","affiliation":"National Engineering Research Center for Big Data Technology and System Services Computing Technology and System Lab , Cluster and Grid Computing Lab School of Computer Science and Technology , Huazhong University of Science and Technology , China"},{"name":"Hai Jin","slug":"hai-jin","given":"Hai","family":"Jin","affiliation":"National Engineering Research Center for Big Data Technology and System Services Computing Technology and System Lab , Cluster and Grid Computing Lab School of Computer Science and Technology , Huazhong University of Science and Technology , China"}],"abstract":"Optimal margin Distribution Machine (ODM) is a newly proposed statistical learning framework rooting in the novel margin theory, which demonstrates better generalization performance than the traditional large margin based counterparts. Nonetheless, it suffers from the ubiquitous scalability problem regarding both computation time and memory as other kernel methods. This paper proposes a scalable ODM, which can achieve nearly ten times speedup compared to the original ODM training method. For nonlinear kernels, we propose a novel distribution-aware partition method to make the local ODM trained on each partition be close and converge faster to the global one. When linear kernel is applied, we extend a communication efficient SVRG method to accelerate the training further. Extensive empirical studies validate that our proposed method is highly computational efficient and almost never worsen the generalization.","doi":"10.24963/ijcai.2023/485","url":"https://ijcai.org/proceedings/2023/485","pdf":"https://ijcai.org/proceedings/2023/0485.pdf","markdown":"https://ijcai.org/proceedings/2023/485.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/485","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":485,"pages":{"first":4362,"last":4370},"keywords":["Machine Learning: ML: Classification","Machine Learning: ML: Kernel methods","Data Mining: DM: Big data and scalability"],"videos":[],"references":[{"position":1,"raw":"[Boyd et al., 2010] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. Distributed Opti- mization and Statistical Learning via the Alternating Di- rection Method of Multipliers. Foundations and Trends in Machine Learning, 3(1):1-122, 2010.","title":"Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers","authors":"Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein","year":"2010","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cao et al., 2006] Lijuan Cao, Selvaraj Sathiya Keerthi, Chong Jin Ong, Jianqiu Zhang, Uvaraj Periyathamby, Xi- uju Fu, and Henry P. Lee. Parallel sequential minimal optimization for the training of support vector machines. IEEE Transactions on Neural Networks, 17(4):1039- 1049, 2006.","title":"Parallel sequential minimal optimization for the training of support vector machines","authors":"Lijuan Cao, Selvaraj Sathiya Keerthi, Chong, Jin Ong, Jianqiu Zhang, Uvaraj Periyathamby, Xiuju Fu, Henry P Lee","year":"2006","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cao et al., 2021] Nan Cao, Teng Zhang, and Hai Jin. Par- tial Multi-Label Optimal Margin Distribution Machine. In Proceedings of the 30th International Joint Conference on Artificial Intelligence, pages 2198-2204, Montreal- themed virtual reality, 2021.","title":"Partial Multi-Label Optimal Margin Distribution Machine","authors":"Nan Cao, Teng Zhang, Hai Jin","year":"2021","venue":"Proceedings of the 30th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2021/303"},{"position":4,"raw":"[Cao et al., 2022] Nan Cao, Teng Zhang, Xuanhua Shi, and Hai Jin. Posistive-Unlabeled Learning via Optimal Trans- port and Margin Distribution. In Proceedings of the 31st International Joint Conference on Artificial Intelligence, pages 2836-2842, Vienna, Austria, 2022.","title":"Posistive-Unlabeled Learning via Optimal Transport and Margin Distribution","authors":"Nan Cao, Teng Zhang, Xuanhua Shi, Hai Jin","year":"2022","venue":"Proceedings of the 31st International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2022/393"},{"position":5,"raw":"[Cheng et al., 2017] Fanyong Cheng, Jing Zhang, Cuihong Wen, Zhaohua Liu, and Zuoyong Li. Large cost-sensitive margin distribution machine for imbalanced data classifi- cation. Neurocomputing, 224:45-57, 2017.","title":"Large cost-sensitive margin distribution machine for imbalanced data classification","authors":"Fanyong Cheng, Jing Zhang, Cuihong Wen, Zhaohua Liu, Zuoyong Li","year":"2017","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Forero et al., 2010] Pedro A. Forero, Alfonso Cano, and Georgios B. Giannakis. Consensus-Based Distributed Support Vector Machines. Journal of Machine Learning Research, 11:1663-1701, 2010.","title":"Consensus-Based Distributed Support Vector Machines","authors":"Pedro A Forero, Alfonso Cano, Georgios B Giannakis","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gao and Zhou, 2013] Wei Gao and Zhi-Hua Zhou. On the doubt about margin explanation of boosting. Artificial In- telligence, 203:1-18, 2013.","title":"On the doubt about margin explanation of boosting","authors":"Wei Gao, Zhi-Hua Zhou","year":"2013","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Graf et al., 2004] Hans Peter Graf, Eric Cosatto, Leon Bot- tou, Igor Dourdanovic, and Vladimir Vapnik. Parallel Sup- port Vector Machines: The Cascade SVM. In Advances in Neural Information Processing Systems, pages 521-528, Vancouver, Canada, 2004.","title":"Parallel Support Vector Machines: The Cascade SVM","authors":"Graf","year":"2004","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Grønlund et al., 2019] Allan Grønlund, Lior Kamma, Kasper Green Larsen, Alexander Mathiasen, and Jelani Nelson. Margin-Based Generalization Lower Bounds for Boosted Classifiers. In Advances in Neural Information Processing Systems, pages 11963-11972, Vancouver, Canada, 2019.","title":"Margin-Based Generalization Lower Bounds for Boosted Classifiers","authors":"Allan Grønlund, Lior Kamma, Alexander Kasper Green Larsen, Jelani Mathiasen, Nelson","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hsieh et al., 2014] Cho-Jui Hsieh, Si Si, and Inderjit Singh Dhillon. A Divide-and-Conquer Solver for Kernel Sup- port Vector Machines. In Proceedings of the 31st Interna- tional Conference on Machine Learning, pages 566-574, Beijing, China, 2014.","title":"A Divide-and-Conquer Solver for Kernel Support Vector Machines","authors":"Cho-Jui Hsieh, Si Si, Inderjit Singh Dhillon","year":"2014","venue":"Proceedings of the 31st International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Johnson and Zhang, 2013] Rie Johnson and Tong Zhang. Accelerating Stochastic Gradient Descent using Predic- tive Variance Reduction. In Advances in Neural Informa- tion Processing Systems, pages 315-323, Lake Tahoe, NV, 2013.","title":"Accelerating Stochastic Gradient Descent using Predictive Variance Reduction","authors":"Johnson, Zhang, Rie Johnson, Tong Zhang","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lee et al., 2017] Jason D. Lee, Qihang Lin, Tengyu Ma, and Tianbao Yang. Distributed Stochastic Variance Reduced Gradient Methods by Sampling Extra Data with Replace- ment. Journal of Machine Learning Research, 18(122):1- 43, 2017.","title":"Distributed Stochastic Variance Reduced Gradient Methods by Sampling Extra Data with Replacement","authors":"Jason D Lee, Qihang Lin, Tengyu Ma, Tianbao Yang","year":"2017","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Loosli et al., 2007] Gaëlle Loosli, Stéphane Canu, and Léon Bottou. Training invariant support vector machines us- ing selective sampling. In Léon Bottou, Olivier Chapelle, Dennis DeCoste, and Jason Weston, editors, Large-Scale Kernel Machines, pages 301-320. MIT Press, Cambridge, MA, 2007.","title":"Training invariant support vector machines using selective sampling","authors":"Gaëlle Loosli, Stéphane Canu, Léon Bottou","year":"2007","venue":"Large-Scale Kernel Machines","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Luan et al., 2020] Tianxiang Luan, Tingjin Luo, Wenzhang Zhuge, and Chenping Hou. Optimal Representative Distri- bution Margin Machine for Multi-Instance Learning. IEEE Access, 8:74864-74874, 2020.","title":"Optimal Representative Distribution Margin Machine for Multi-Instance Learning","authors":"Tianxiang Luan, Tingjin Luo, Wenzhang Zhuge, Chenping Hou","year":"2020","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Navia-Vazquez et al., 2006] Angel Navia-Vazquez, D. Gutierrez-Gonzalez, Emilio Parrado-Hernandez, and J. J. Navarro-Abellan. Distributed Support Vector Machines. IEEE Transactions on Neural Networks, 17(4):1091-1097, 2006.","title":"Distributed Support Vector Machines","authors":"Navia-Vazquez","year":"2006","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rahimi and Recht, 2007] Ali Rahimi and Benjamin Recht. Random Features for Large-Scale Kernel Machines. In Advances in Neural Information Processing Systems, pages 1177-1184, Vancouver, Canada, 2007.","title":"Random Features for Large-Scale Kernel Machines","authors":"Recht Rahimi, Ali Rahimi, Benjamin Recht","year":"2007","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rastogi et al., 2020] Reshma Rastogi, Pritam Anand, and Suresh Chandra. Large-margin Distribution Machine- based regression. Neural Computing and Applications, 32:3633-3648, 2020.","title":"Large-margin Distribution Machinebased regression","authors":"Reshma Rastogi, Pritam Anand, Suresh Chandra","year":"2020","venue":"Neural Computing and Applications","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schölkopf and Smola, 2001] Bernhard Schölkopf and Alexander Johannes Smola. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT Press, Cambridge, MA, 2001.","title":"Learning with kernels: support vector machines, regularization, optimization, and beyond","authors":"Smola Schölkopf, Bernhard Schölkopf, Alexander Johannes, Smola","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Si et al., 2017] Si Si, Cho-Jui Hsieh, and Inderjit Singh Dhillon. Memory Efficient Kernel Approximation. Jour- nal of Machine Learning Research, 18:1-32, 2017.","title":"Memory Efficient Kernel Approximation","authors":"Si Si, Cho-Jui Hsieh, Inderjit Singh Dhillon","year":"2017","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Singh et al., 2017] Dinesh Singh, Debaditya Roy, and Chalavadi Krishna Mohan. DiP-SVM: Distribution Pre- serving Kernel Support Vector Machine for Big Data. IEEE Transactions on Big Data, 3(1):79-90, 2017.","title":"DiP-SVM: Distribution Preserving Kernel Support Vector Machine for Big Data","authors":"Dinesh Singh, Debaditya Roy, Chalavadi Krishna, Mohan","year":"2017","venue":"IEEE Transactions on Big Data","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tan et al., 2019] Zhi-Hao Tan, Teng Zhang, and Wei Wang. Coreset Stochastic Variance-Reduced Gradient with Ap- plication to Optimal Margin Distribution Machine. In Pro- ceedings of the 33rd AAAI Conference on Artificial Intelli- gence, pages 5083-5090, Honolulu, HI, 2019.","title":"Coreset Stochastic Variance-Reduced Gradient with Application to Optimal Margin Distribution Machine","authors":"Zhi-Hao Tan, Teng Zhang, Wei Wang","year":"2019","venue":"Proceedings of the 33rd AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tan et al., 2020] Zhi-Hao Tan, Peng Tan, Yuan Jiang, and Zhi-Hua Zhou. Multi-label Optimal Margin Distribution Machine. Machine Learning, 109(3):623-642, 2020.","title":"Multi-label Optimal Margin Distribution Machine","authors":"Zhi-Hao Tan, Peng Tan, Yuan Jiang, Zhi-Hua Zhou","year":"2020","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Williams and Seeger, 2001] Christopher Williams and Matthias Seeger. Using the Nyström Method to Speed Up Kernel Machines. In Advances in Neural Information Processing Systems, pages 682-688, Cambridge, MA, 2001.","title":"Using the Nyström Method to Speed Up Kernel Machines","authors":"Christopher Williams, Matthias Seeger","year":"2001","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yu et al., 2005] Hwanjo Yu, Jiong Yang, Jiawei Han, and Xiaolei Li. Making SVMs scalable to large data sets using hierarchical cluster indexing. Data Mining and Knowledge Discovery, 11(3):295-321, 2005.","title":"Making SVMs scalable to large data sets using hierarchical cluster indexing","authors":"Yu","year":"2005","venue":"Data Mining and Knowledge Discovery","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zaharia et al., 2012] Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauly, Michael Jay Franklin, Scott Shenker, and Ion Stoica. Re- silient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. In Proceedings of the 9th USENIX Symposium on Networked Systems Design and Implementation, pages 15-28, San Jose, CA, 2012.","title":"Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing","authors":"Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy Mccauly, Michael Jay Franklin, Scott Shenker, Ion Stoica","year":"2012","venue":"Proceedings of the 9th USENIX Symposium on Networked Systems Design and Implementation","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang and Jin, 2020] Teng Zhang and Hai Jin. Optimal Margin Distribution Machine for Multi-Instance Learning. In Proceedings of the 29th International Joint Conference on Artificial Intelligence, pages 2383-2389, 2020.","title":"Optimal Margin Distribution Machine for Multi-Instance Learning","authors":"Teng Zhang, Hai Jin","year":"2020","venue":"Proceedings of the 29th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2020/330"},{"position":27,"raw":"[Zhang and Zhou, 2018a] Teng Zhang and Zhi-Hua Zhou. Optimal margin distribution clustering. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, pages 4474-4481, New Orleans, LA, 2018.","title":"Optimal margin distribution clustering","authors":"Zhou Zhang, Teng Zhang, Zhi-Hua Zhou","year":"2018","venue":"Proceedings of the 32nd AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang and Zhou, 2018b] Teng Zhang and Zhi-Hua Zhou. Semi-supervised optimal margin distribution machines. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 3104-3110, Stockholm, Swe- den, 2018.","title":"Semi-supervised optimal margin distribution machines","authors":"Zhou Zhang, Teng Zhang, Zhi-Hua Zhou","year":"2018","venue":"Proceedings of the 27th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2018/431"},{"position":29,"raw":"[Zhang and Zhou, 2019] Teng Zhang and Zhi-Hua Zhou. Optimal Margin Distribution Machine. IEEE Transactions on Knowledge and Data Engineering, 32(6):1143-1156, 2019.","title":"Optimal Margin Distribution Machine","authors":"Teng Zhang, Zhi-Hua Zhou","year":"2019","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2020] Teng Zhang, Peng Zhao, and Hai Jin. Optimal Margin Distribution Learning in Dynamic Envi- ronments. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, pages 6821-6828, New York, NY, 2020.","title":"Optimal Margin Distribution Learning in Dynamic Environments","authors":"Teng Zhang, Peng Zhao, Hai Jin","year":"2020","venue":"Proceedings of the 34th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhou and Zhou, 2016] Yu-Hang Zhou and Zhi-Hua Zhou. Large Margin Distribution Learning with Cost Interval and Unlabeled data. IEEE Transactions on Knowledge and Data Engineering, 28(7):1749-1763, 2016.","title":"Large Margin Distribution Learning with Cost Interval and Unlabeled data","authors":"Yu-Hang Zhou, Zhi-Hua Zhou","year":"2016","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}