{"type":"ScholarlyArticle","title":"Exclusivity Regularized Machine: A New Ensemble SVM Classifier","authors":[{"name":"Xiaojie Guo","slug":"xiaojie-guo","given":"Xiaojie","family":"Guo","affiliation":"† State Key Laboratory of Information Security , IIE , Chinese Academy of Sciences *"},{"name":"Xiaobo Wang","slug":"xiaobo-wang","given":"Xiaobo","family":"Wang","affiliation":"‡ University of Chinese Academy of Sciences National Laboratory of Pattern Recognition , IA ,"},{"name":"Haibin Ling","slug":"haibin-ling","given":"Haibin","family":"Ling","affiliation":"§ Chinese Academy of Sciences Department of Computer and Information Sciences ,; Temple University"}],"abstract":"The diversity of base learners is of utmost importance to a good ensemble. This paper defines a novel measurement of diversity, termed as exclusivity. With the designed exclusivity, we further propose an ensemble SVM classifier, namely Exclusivity Regularized Machine (ExRM), to jointly suppress the training error of ensemble and enhance the diversity between bases. Moreover, an Augmented Lagrange Multiplier based algorithm is customized to effectively and efficiently seek the optimal solution of ExRM. Theoretical analysis on convergence, global optimality and linear complexity of the proposed algorithm, as well as experiments are provided to reveal the efficacy of our method and show its superiority over state-of-the-arts in terms of accuracy and efficiency.","doi":"10.24963/ijcai.2017/241","url":"https://ijcai.org/proceedings/2017/241","pdf":"https://ijcai.org/proceedings/2017/0241.pdf","markdown":"https://ijcai.org/proceedings/2017/241.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/241","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":241,"pages":{"first":1739,"last":1745},"keywords":["Machine Learning: Ensemble Methods","Machine Learning: Classification"],"videos":[],"references":[{"position":1,"raw":"[Altman, 1992] N. Altman. An introduction to kernel and nearest-neighbor nonparametric regression. The American Statistician, 46(3):175-185, 1992.","title":"An introduction to kernel and nearest-neighbor nonparametric regression","authors":"N Altman","year":"1992","venue":"The American Statistician","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Breiman et al., 1984] L. Breiman, J. Friedman, R. Olshen, and C. Stone. Classification and Regression Trees. Wadsworth, Belmont, 1984.","title":"Classification and Regression Trees","authors":"L Breiman, J Friedman, R Olshen, C Stone","year":"1984","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Breiman, 1996] L. Breiman. Bagging predictors. Machine Learning, 24(2):123-140, 1996.","title":"Bagging predictors","authors":"L Breiman","year":"1996","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Breiman, 2001] L. Breiman. Random forests. Machine Learning, 45(1):5-32, 2001.","title":"Random forests","authors":"L Breiman","year":"2001","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chang et al., 2008] K. Chang, C. Hsieh, and C. Lin. Coor- dinate descent method for large-scale L 2 -loss linear svm. Journal of Machine Learning Research, 9:1369-1398, 2008.","title":"Coordinate descent method for large-scale L 2 -loss linear svm","authors":"K Chang, C Hsieh, C Lin","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dietterich, 2000] T. Dietterich. An experimental compari- son of three methods for constructing ensembles of deci- sion trees: Bagging, boosting, and randomization. Ma- chine Learning, 40(2):139-157, 2000.","title":"An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, boosting, and randomization","authors":"T Dietterich","year":"2000","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Domingos and Pazzani, 1997] P. Domingos and M. Paz- zani. On the optimality of the simple bayesian classifi- er under zero-one loss. Machine Learning, 29:103-130, 1997.","title":"On the optimality of the simple bayesian classifier under zero-one loss","authors":"P Domingos, M Pazzani","year":"1997","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Du and Ling, 2014] L. Du and H. Ling. Exploiting compe- tition relationship for robust visual recognition. In AAAI, pages 2746-2752, 2014.","title":"Exploiting competition relationship for robust visual recognition","authors":"L Du, H Ling","year":"2014","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fisher, 1936] R. Fisher. The use of multiple measurements in taxonomic problems. Annals of Eugenics, 7(2):179- 188, 1936.","title":"The use of multiple measurements in taxonomic problems","authors":"R Fisher","year":"1936","venue":"Annals of Eugenics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Freund and Schapire, 1997] Y. Freund and R. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and Sys- tem Sciences, 55(1):119-139, 1997.","title":"A decision-theoretic generalization of on-line learning and an application to boosting","authors":"Y Freund, R Schapire","year":"1997","venue":"Journal of Computer and System Sciences","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Giacinto and Roli, 2001] G. Giacinto and F. Roli. Design of effective neural network ensembles for image classi- fication purposes. Image and Vision Computing, 19(9- 10):699-707, 2001.","title":"Design of effective neural network ensembles for image classification purposes","authors":"G Giacinto, F Roli","year":"2001","venue":"Image and Vision Computing","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ho, 1998] T. Ho. The random subspace method for con- structing decision forests. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(8):832-844, 1998.","title":"The random subspace method for constructing decision forests","authors":"T Ho","year":"1998","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hsieh et al., 2008] C. Hsieh, K. Chang, S. Keerthi, S. Sun- dararajan, and C. Lin. A dual coordinate descent method for large-scale linear svm. In ICML, pages 408-415, 2008.","title":"A dual coordinate descent method for large-scale linear svm","authors":"C Hsieh, K Chang, S Keerthi, S Sundararajan, C Lin","year":"2008","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Joachims, 2006] T. Joachims. Training linear svms in linear time. In ACM SIGKDD, pages 217-226, 2006.","title":"Training linear svms in linear time","authors":"T Joachims","year":"2006","venue":"ACM SIGKDD","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Keerthi and DeCoste, 2005] S. Keerthi and D. DeCoste. A modified finite newton method for fast solution of large scale linear svms. Journal of Machine Learning Research, 6:627-650, 2005.","title":"A modified finite newton method for fast solution of large scale linear svms","authors":"S Keerthi, D Decoste","year":"2005","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kong et al., 2014] D. Kong, R. Fujimaki, J. Liu, F. Nie, and C. Ding. Exclusive feature learning on arbitrary structures via ℓ 1,2 -norm. In NIPS, pages 1655-1663, 2014.","title":"Exclusive feature learning on arbitrary structures via ℓ 1,2 -norm","authors":"D Kong, R Fujimaki, J Liu, F Nie, C Ding","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kowalski, 2009] M. Kowalski. Sparse regression using mixed norms. Applied and Computational Harmonic Analysis, 27(3):303-324, 2009.","title":"Sparse regression using mixed norms","authors":"M Kowalski","year":"2009","venue":"Applied and Computational Harmonic Analysis","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Krogh and Vedelsby, 1995] A. Krogh and J. Vedelsby. Neu- ral network ensembles, cross validation, and active learn- ing. In NIPS, pages 231-238, 1995.","title":"Neural network ensembles, cross validation, and active learning","authors":"A Krogh, J Vedelsby","year":"1995","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kuncheva et al., 2003] L. Kuncheva, C. Whitaker, C. Ship- p, and R. Duin. Limits on the majority vote accuracy in classification fusion. Pattern Analysis and Applications, 6(1):22-31, 2003.","title":"Limits on the majority vote accuracy in classification fusion","authors":"L Kuncheva, C Whitaker, C Shipp, R Duin","year":"2003","venue":"Pattern Analysis and Applications","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2012] N. Li, Y. Yu, and Z. Zhou. Diversity regular- ized ensemble pruning. In ECML PKDD, pages 330-345, 2012.","title":"Diversity regularized ensemble pruning","authors":"N Li, Y Yu, Z Zhou","year":"2012","venue":"ECML PKDD","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lin et al., 2008] C. Lin, R. Weng, and S. Keerthi. Trust region newton method for large-scale logistic regression. Journal of Machine Learning Research, 9:627-650, 2008.","title":"Trust region newton method for large-scale logistic regression","authors":"C Lin, R Weng, S Keerthi","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nie et al., 2014] F. Nie, Y. Huang, X. Wang, and H. Huang. New primal svm solver with linear computational cost for big data classifications. In ICML, Beijing, China, 2014.","title":"New primal svm solver with linear computational cost for big data classifications","authors":"F Nie, Y Huang, X Wang, H Huang","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Quinlan, 1993] J. Quinlan. C4.5: Programs for machine learning. Morgan Kaufmann Publisher, San Mateo, 1993.","title":"C4.5: Programs for machine learning","authors":"J Quinlan","year":"1993","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rodriguez et al., 2006] J. Rodriguez, L. Kubcheva, and C. Alonso. Rotation forest: A new classifier ensemble method. TPAMI, 28(10):1619-1630, 2006.","title":"Rotation forest: A new classifier ensemble method","authors":"J Rodriguez, L Kubcheva, C Alonso","year":"2006","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shalev-Shwartz et al., 2007] S. Shalev-Shwartz, Y. Singer, and N. Srebro. Pegasos: Primal estimated subgradient solver for svm. In ICML, pages 807-814, 2007.","title":"Pegasos: Primal estimated subgradient solver for svm","authors":"S Shalev-Shwartz, Y Singer, N Srebro","year":"2007","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Teo et al., 2010] C. Teo, S. Vishwanathan, A. Smola, and Q. Le. Bundle methods for regularized risk minimiza- tion. Journal of Machine Learning Research, 11:311-365, 2010.","title":"Bundle methods for regularized risk minimization","authors":"C Teo, S Vishwanathan, A Smola, Q Le","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ueda and Nakano, 1996] P. Ueda and R. Nakano. General- ization error of ensemble estimators. In Proceedings of In- ternational Conference on Neural Network (ICNN), pages 90-95, 1996.","title":"Generalization error of ensemble estimators","authors":"P Ueda, R Nakano","year":"1996","venue":"Proceedings of International Conference on Neural Network (ICNN)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Vapnik, 1995] V. Vapnik. The Nature of Statistical Learning Theory. Springe-Verlag, New York, NY, 1995.","title":"The Nature of Statistical Learning Theory","authors":"V Vapnik","year":"1995","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xie et al., 2015a] P. Xie, Y. Deng, and E. Xing. Latent vari- able modeling with diversity-inducing mutual angular reg- ularization. arXiv:1512.07336v1, 2015.","title":"Latent variable modeling with diversity-inducing mutual angular regularization","authors":"Xie","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xie et al., 2015b] P. Xie, Y. Deng, and E. Xing. On the generalization error bounds of neural networks under diversity-inducing mutual angular regularization. arX- iv:1511.07110v1, 2015.","title":"On the generalization error bounds of neural networks under diversity-inducing mutual angular regularization","authors":"Xie","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yu et al., 2011] Y. Yu, Y. Li, and Z. Zhou. Diversity regu- larized machine. In IJCAI, pages 1603-1608, 2011.","title":"Diversity regularized machine","authors":"Y Yu, Y Li, Z Zhou","year":"2011","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2011/269"},{"position":32,"raw":"[Zhang et al., 2010] Y. Zhang, R. Jin, and S. Hoi. Exclusive lasso for multi-task feature selection. In AISTATS, pages 988-995, 2010.","title":"Exclusive lasso for multi-task feature selection","authors":"Y Zhang, R Jin, S Hoi","year":"2010","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhou, 2012] Z. Zhou. Ensemble Methods: Foundations and Algorithms. Taylor & Francis Group, LLC, Boca Raton, FL, 2012.","title":"Ensemble Methods: Foundations and Algorithms","authors":"Z Zhou","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}