{"type":"ScholarlyArticle","title":"Improving the Generalization Performance of Multi-class SVM via Angular Regularization","authors":[{"name":"Jianxin Li","slug":"jianxin-li","given":"Jianxin","family":"Li","affiliation":"School of Computer Science and Engineering , Beihang University"},{"name":"Haoyi Zhou","slug":"haoyi-zhou","given":"Haoyi","family":"Zhou","affiliation":"School of Computer Science and Engineering , Beihang University"},{"name":"Pengtao Xie","slug":"pengtao-xie","given":"Pengtao","family":"Xie","affiliation":"Machine Learning Department , Carnegie Mellon University 3 Petuum Inc , USA"},{"name":"Yingchun Zhang","slug":"yingchun-zhang","given":"Yingchun","family":"Zhang","affiliation":"School of Computer Science and Engineering , Beihang University"}],"abstract":"In multi-class support vector machine (MSVM) for classification, one core issue is to regularize the coefficient vectors to reduce overfitting. Various regularizers have been proposed such as L2, L1, and trace norm. In this paper, we introduce a new type of regularization approach -- angular regularization, that encourages the coefficient vectors to have larger angles such that class regions can be widen to flexibly accommodate unseen samples. We propose a novel angular regularizer based on the singular values of the coefficient matrix, where the uniformity of singular values reduces the correlation among different classes and drives the angles between coefficient vectors to increase. In generalization error analysis, we show that decreasing this regularizer effectively reduces generalization error bound. On various datasets, we demonstrate the efficacy of the regularizer in reducing overfitting.","doi":"10.24963/ijcai.2017/296","url":"https://ijcai.org/proceedings/2017/296","pdf":"https://ijcai.org/proceedings/2017/0296.pdf","markdown":"https://ijcai.org/proceedings/2017/296.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/296","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":296,"pages":{"first":2131,"last":2137},"keywords":["Machine Learning: Machine Learning","Machine Learning: Learning Theory","Machine Learning: Classification"],"videos":[],"references":[{"position":1,"raw":"[Amit et al., 2007] Yonatan Amit, Michael Fink, Nathan Srebro, and Shimon Ullman. Uncovering shared structures in multiclass classification. In ICML, pages 17-24, 2007.","title":"Uncovering shared structures in multiclass classification","authors":"Yonatan Amit, Michael Fink, Nathan Srebro, Shimon Ullman","year":"2007","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bao et al., 2013] Yebo Bao, Hui Jiang, Lirong Dai, and Cong Liu. Incoherent training of deep neural networks to de-correlate bottleneck features for speech recognition. In ICASSP, pages 6980-6984. IEEE, 2013.","title":"Incoherent training of deep neural networks to de-correlate bottleneck features for speech recognition","authors":"Yebo Bao, Hui Jiang, Lirong Dai, Cong Liu","year":"2013","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bartlett and Mendelson, 2002] Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. JMLR, 3:463-482, 2002.","title":"Rademacher and gaussian complexities: Risk bounds and structural results","authors":"L Peter, Shahar Bartlett, Mendelson","year":"2002","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bradley and Mangasarian, 1998] Paul S Bradley and Olvi L Mangasarian. Feature selection via concave minimiza- tion and support vector machines. In ICML, pages 82-90, 1998.","title":"Feature selection via concave minimization and support vector machines","authors":"Mangasarian ; ] Bradley, S Paul, Bradley, Olvi L Mangasarian","year":"1998","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chang and Lin, 2011] Chih-Chung Chang and Chih-Jen Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1-27:27, 2011.","title":"LIBSVM: A library for support vector machines","authors":"Chang, Lin ; ] Chih-Chung Chang, Chih-Jen Lin","year":"2011","venue":"ACM Transactions on Intelligent Systems and Technology","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cortes et al., 2013] Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Multi-class classification with max- imum margin multiple kernel. In ICML, pages 46-54, 2013.","title":"Multi-class classification with maximum margin multiple kernel","authors":"Corinna Cortes, Mehryar Mohri, Afshin Rostamizadeh","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Crammer and Singer, 2001] Koby Crammer and Yoram Singer. On the algorithmic implementation of multiclass kernel-based vector machines. JMLR, 2:265-292, 2001.","title":"On the algorithmic implementation of multiclass kernel-based vector machines","authors":"Koby Crammer, Yoram Singer","year":"2001","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[FernáNdez et al., 2013] Alberto FernáNdez, Victoria LóPez, Mikel Galar, MaríA José Del Jesus, and Francisco Herrera. Analysing the classification of imbalanced data-sets with multiple classes: Binarization techniques and ad-hoc approaches. Knowledge-based systems, 42:97-110, 2013.","title":"Analysing the classification of imbalanced data-sets with multiple classes: Binarization techniques and ad-hoc approaches","authors":"Alberto Fernández, Victoria López, Mikel Galar, María José Del, Jesus, Francisco Herrera","year":"2013","venue":"Knowledge-based systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guermeur and Monfrini, 2011] Yann Guermeur and Em- manuel Monfrini. A quadratic loss multi-class svm for which a radius-margin bound applies. Informatica, 22(1):73-96, 2011.","title":"Yann Guermeur and Emmanuel Monfrini. A quadratic loss multi-class svm for which a radius-margin bound applies","authors":"Monfrini Guermeur","year":"2011","venue":"Informatica","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Guo, 2016] Xiaojie Guo. Exclusivity regularized machine. arXiv:1603.08318, 2016.","title":"Exclusivity regularized machine","authors":"Xiaojie Guo","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jolliffe, 2002] Ian Jolliffe. Principal component analysis. Wiley Online Library, 2002.","title":"Principal component analysis","authors":"Ian Jolliffe","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kulesza et al., 2012] Alex Kulesza, Ben Taskar, et al. De- terminantal point processes for machine learning. Founda- tions and Trends R in Machine Learning, 5(2-3):123-286, 2012.","title":"Determinantal point processes for machine learning","authors":"Alex Kulesza, Ben Taskar","year":"2012","venue":"Foundations and Trends R in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kumar et al., 2012] Abhishek Kumar, Alexandru Niculescu-Mizil, Koray Kavukcuoglu, and Hal Daume III. A binary classification framework for two-stage multiple kernel learning. In ICML, pages 1295-1302, 2012.","title":"A binary classification framework for two-stage multiple kernel learning","authors":"Abhishek Kumar, Alexandru Niculescu-Mizil, Koray Kavukcuoglu, Hal Daume, Iii","year":"2012","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lapin et al., 2015] Maksim Lapin, Matthias Hein, and Bernt Schiele. Top-k multiclass svm. In NIPS, pages 325- 333, 2015.","title":"Top-k multiclass svm","authors":"Maksim Lapin, Matthias Hein, Bernt Schiele","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lee et al., 2004] Yoonkyung Lee, Yi Lin, and Grace Wahba. Multicategory support vector machines: Theory and appli- cation to the classification of microarray data and satellite radiance data. JASA, 99(465):67-81, 2004.","title":"Multicategory support vector machines: Theory and application to the classification of microarray data and satellite radiance data","authors":"Lee","year":"2004","venue":"JASA","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Percy, 2015] Liang Percy. Lecture notes of statistical learn- ing theory. https:// web.stanford.edu/ class/ cs229t/ notes. pdf , 2015.","title":"Lecture notes of statistical learning theory","authors":"Percy ;, Liang Percy","year":"2015","venue":null,"doi":null,"url":"https://web.stanford.edu/class/cs229t/notes.pdf,2015","internal_url":null},{"position":17,"raw":"[Shalev-Shwartz and Tewari, 2011] Shai Shalev-Shwartz and Ambuj Tewari. Stochastic methods for l1-regularized loss minimization. JMLR, 12:1865-1892, 2011.","title":"Stochastic methods for l1-regularized loss minimization","authors":"Shai Shalev, -Shwartz, Ambuj Tewari","year":"2011","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shalev-Shwartz et al., 2011] Shai Shalev-Shwartz, Yoram Singer, Nathan Srebro, and Andrew Cotter. Pegasos: Pri- mal estimated sub-gradient solver for svm. Mathematical programming, 127(1):3-30, 2011.","title":"Pegasos: Primal estimated sub-gradient solver for svm","authors":"Shai Shalev-Shwartz, Yoram Singer, Nathan Srebro, Andrew Cotter","year":"2011","venue":"Mathematical programming","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tsochantaridis et al., 2004] Ioannis Tsochantaridis, Thomas Hofmann, Thorsten Joachims, and Yasemin Altun. Support vector machine learning for interdepen- dent and structured output spaces. In ICML, page 104, 2004.","title":"Support vector machine learning for interdependent and structured output spaces","authors":"Ioannis Tsochantaridis, Thomas Hofmann, Thorsten Joachims, Yasemin Altun","year":"2004","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vapnik and Vapnik, 1998] Vladimir Naumovich Vapnik and Vlamimir Vapnik. Statistical learning theory, volume 1. Wiley New York, 1998.","title":"Vladimir Naumovich Vapnik and Vlamimir Vapnik","authors":"Vapnik Vapnik","year":"1998","venue":"Statistical learning theory","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang and Shen, 2012] Lifeng Wang and Xiaotong Shen. On l1-norm multiclass support vector machines. JASA, 102(478):583-594, 2012.","title":"On l1-norm multiclass support vector machines","authors":"Wang, Shen ; ] Lifeng Wang, Xiaotong Shen","year":"2012","venue":"JASA","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang and Xue, 2014] Zhe Wang and Xiangyang Xue. Multi-class support vector machine. In Support Vector Ma- chines Applications, pages 23-48. Springer, 2014.","title":"Multi-class support vector machine","authors":"Zhe Wang, Xiangyang Xue","year":"2014","venue":"Support Vector Machines Applications","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang and Yao, 2012] Shuo Wang and Xin Yao. Multiclass imbalance problems: Analysis and potential solutions. TSMC, 42(4):1119-1130, 2012.","title":"Multiclass imbalance problems: Analysis and potential solutions","authors":"Shuo Wang, Xin Yao","year":"2012","venue":"TSMC","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2010] Zhuang Wang, Koby Crammer, and Slo- bodan Vucetic. Multi-class pegasos on a budget. In ICML, pages 1143-1150, 2010.","title":"Multi-class pegasos on a budget","authors":"Zhuang Wang, Koby Crammer, Slobodan Vucetic","year":"2010","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2011] Zhuang Wang, Nemanja Djuric, Koby Crammer, and Slobodan Vucetic. Trading representabil- ity for scalability: adaptive multi-hyperplane machine for nonlinear classification. In SIGKDD, pages 24-32. ACM, 2011.","title":"Trading representability for scalability: adaptive multi-hyperplane machine for nonlinear classification","authors":"Zhuang Wang, Nemanja Djuric, Koby Crammer, Slobodan Vucetic","year":"2011","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Weston and Watkins, 1998] Jason Weston and Chris Watkins. Multi-class support vector machines. Technical report, Citeseer, 1998.","title":"Multi-class support vector machines","authors":"Weston, Watkins, Jason Weston, Chris Watkins","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xie et al., 2015a] Pengtao Xie, Yuntian Deng, and Eric Xing. Diversifying restricted boltzmann machine for doc- ument modeling. In SIGKDD, pages 1315-1324, 2015.","title":"Diversifying restricted boltzmann machine for document modeling","authors":"Xie","year":"2015","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xie et al., 2015b] Pengtao Xie, Yuntian Deng, and Eric Xing. On the generalization error bounds of neural net- works under diversity-inducing mutual angular regulariza- tion. arXiv:1511.07110, 2015.","title":"On the generalization error bounds of neural networks under diversity-inducing mutual angular regularization","authors":"Xie","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xie, 2015] Pengtao Xie. Learning compact and effective distance metrics with diversity regularization. In ECML- KDD, pages 610-624. Springer, 2015.","title":"Pengtao Xie. Learning compact and effective distance metrics with diversity regularization","authors":"Xie","year":"2015","venue":"ECML-KDD","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yu et al., 2011] Yang Yu, Yu-Feng Li, and Zhi-Hua Zhou. Diversity regularized machine. In IJCAI, volume 22, pages 1603-1608. Citeseer, 2011.","title":"Diversity regularized machine","authors":"Yang Yu, Yu-Feng Li, Zhi-Hua Zhou","year":"2011","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2011/269"},{"position":31,"raw":"[Zeiler, 2012] Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv:1212.5701, 2012.","title":"Adadelta: an adaptive learning rate method","authors":"D Matthew, Zeiler","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}