{"type":"ScholarlyArticle","title":"Robust Softmax Regression for Multi-class Classification with Self-Paced Learning","authors":[{"name":"Yazhou Ren","slug":"yazhou-ren","given":"Yazhou","family":"Ren","affiliation":"SMILE Lab & Big Data Research Center , School of Computer Science and Engineering , University of Electronic Science and Technology of China , Chengdu , 611731 , China."},{"name":"Peng Zhao","slug":"peng-zhao","given":"Peng","family":"Zhao","affiliation":"SMILE Lab & Big Data Research Center , School of Computer Science and Engineering , University of Electronic Science and Technology of China , Chengdu , 611731 , China."},{"name":"Yongpan Sheng","slug":"yongpan-sheng","given":"Yongpan","family":"Sheng","affiliation":"SMILE Lab & Big Data Research Center , School of Computer Science and Engineering , University of Electronic Science and Technology of China , Chengdu , 611731 , China."},{"name":"Dezhong Yao","slug":"dezhong-yao","given":"Dezhong","family":"Yao","affiliation":"Key Laboratory for NeuroInformation of Ministry of Education , School of Life Science and Technology , University of Electronic Science and Technology of China , Chengdu , 611731 , China."},{"name":"Zenglin Xu","slug":"zenglin-xu","given":"Zenglin","family":"Xu","affiliation":"SMILE Lab & Big Data Research Center , School of Computer Science and Engineering , University of Electronic Science and Technology of China , Chengdu , 611731 , China."}],"abstract":"Softmax regression, a generalization of Logistic regression (LR) in the setting of multi-class classification, has been widely used in many machine learning applications. However, the performance of softmax regression is extremely sensitive to the presence of noisy data and outliers. To address this issue, we propose a model of robust softmax regression (RoSR) originated from the self-paced learning (SPL) paradigm for multi-class classification. Concretely, RoSR equipped with the soft weighting scheme is able to evaluate the importance of each data instance. Then, data instances participate in the classification problem according to their weights. In this way, the influence of noisy data and outliers (which are typically with small weights) can be significantly reduced. However, standard SPL may suffer from the imbalanced class influence problem, where some classes may have little influence in the training process if their instances are not sensitive to the loss. To alleviate this problem, we design two novel soft weighting schemes that assign weights and select instances locally for each class. Experimental results demonstrate the effectiveness of the proposed methods.","doi":"10.24963/ijcai.2017/368","url":"https://ijcai.org/proceedings/2017/368","pdf":"https://ijcai.org/proceedings/2017/0368.pdf","markdown":"https://ijcai.org/proceedings/2017/368.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/368","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":368,"pages":{"first":2641,"last":2647},"keywords":["Machine Learning: Machine Learning","Machine Learning: Classification","Machine Learning: Data Mining"],"videos":[],"references":[{"position":1,"raw":"[Amari et al., 1996] Shun Ichi Amari, Andrzej Cichocki, and Howard Hua Yang. A new learning algorithm for blind signal separation. In Advances in Neural Information Pro- cessing Systems, pages 757-763, 1996.","title":"A new learning algorithm for blind signal separation","authors":"Amari","year":"1996","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anand et al., 1993] Rangachari Anand, Kishan Mehrotra, Chilukuri K. Mohan, and Sanjay Ranka. An improved algorithm for neural network classification of imbalanced training sets. IEEE Transactions on Neural Networks, 4(6):962-969, 1993.","title":"An improved algorithm for neural network classification of imbalanced training sets","authors":"Rangachari Anand, Kishan Mehrotra, Chilukuri K Mohan, Sanjay Ranka","year":"1993","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bengio et al., 2009] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learn- ing. In Proceedings of the 26th International Conference on Machine Learning, pages 41-48, 2009.","title":"Curriculum learning","authors":"Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston","year":"2009","venue":"Proceedings of the 26th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bishop, 2006] Christopher M. Bishop. Pattern Recognition and Machine Learning, pages 206-210. Springer, 2006.","title":"Pattern Recognition and Machine Learning","authors":"Christopher M Bishop","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bhning, 1992] Dankmar Bhning. Multinomial logistic re- gression algorithm. Annals of the Institute of Statistical Mathematics, 44(1):197-200, 1992.","title":"Dankmar Bhning. Multinomial logistic regression algorithm","authors":"Bhning","year":"1992","venue":"Annals of the Institute of Statistical Mathematics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cox, 1958] David R. Cox. The regression analysis of bi- nary sequences. Journal of the Royal Statistical Society, 20(2):215-242, 1958.","title":"The regression analysis of binary sequences","authors":"R David, Cox","year":"1958","venue":"Journal of the Royal Statistical Society","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Farid et al., 2014] Dewan Md. Farid, Li Zhang, Chowd- hury Mofizur Rahman, M. A. Hossain, and Rebecca Stra- chan. Hybrid decision tree and naive bayes classifiers for multi-class classification tasks. Expert Systems with Ap- plications, 41(4):1937-1946, 2014.","title":"Hybrid decision tree and naive bayes classifiers for multi-class classification tasks","authors":"Dewan Md, Li Farid, Chowdhury Zhang, M A Mofizur Rahman, Rebecca Hossain, Strachan","year":"2014","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Feng et al., 2014] Jiashi Feng, Huan Xu, Shie Mannor, and Shuicheng Yan. Robust logistic regression and classifica- tion. In Advances in Neural Information Processing Sys- tems, pages 253-261, 2014.","title":"Robust logistic regression and classification","authors":"Feng","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Feng et al., 2016] Yunlong Feng, Yuning Yang, Xiaolin Huang, Siamak Mehrkanoon, and Johan A. K Suykens. Robust support vector machines for classification with nonconvex and smooth losses. Neural Computation, 28(6):1217-1247, 2016.","title":"Robust support vector machines for classification with nonconvex and smooth losses","authors":"Yunlong Feng, Yuning Yang, Xiaolin Huang, Siamak Mehrkanoon, Johan A K Suykens","year":"2016","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Freedman, 2009] David A. Freedman. Statistical models: theory and practice. Cambridge university press, 2009.","title":"Statistical models: theory and practice","authors":"David A Freedman","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hsu and Lin, 2002] Chih-Wei Hsu and Chih-Jen Lin. A comparison of methods for multiclass support vector machines. IEEE Transactions on Neural Networks, 13(4):1026-1027, 2002.","title":"A comparison of methods for multiclass support vector machines","authors":"Lin ; ] Chih-Wei Hsu, Chih-Jen Hsu, Lin","year":"2002","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jiang et al., 2014] Lu Jiang, Deyu Meng, Teruko Mitamura, and Alexander G. Hauptmann. Easy samples first: Self- paced reranking for zero-example multimedia search. In Proceedings of the 22nd ACM International Conference on Multimedia, pages 547-556. ACM, 2014.","title":"Easy samples first: Selfpaced reranking for zero-example multimedia search","authors":"Lu Jiang, Deyu Meng, Teruko Mitamura, Alexander G Hauptmann","year":"2014","venue":"Proceedings of the 22nd ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jiang et al., 2015] Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander G. Hauptmann. Self-paced curriculum learning. In Proceedings of the 29th AAAI Con- ference on Artificial Intelligence, pages 2694-2900, 2015.","title":"Self-paced curriculum learning","authors":"Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, Alexander G Hauptmann","year":"2015","venue":"Proceedings of the 29th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krawczyk et al., 2014] Bartosz Krawczyk, Micha Woniak, and Gerald Schaefer. Cost-sensitive decision tree ensem- bles for effective imbalanced classification. Applied Soft Computing, 14(1):554-562, 2014.","title":"Cost-sensitive decision tree ensembles for effective imbalanced classification","authors":"Bartosz Krawczyk, Micha Woniak, Gerald Schaefer","year":"2014","venue":"Applied Soft Computing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kumar et al., 2010] M. Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In Advances in Neural Information Processing Systems, pages 1189-1197, 2010.","title":"Self-paced learning for latent variable models","authors":"M Pawan Kumar, Benjamin Packer, Daphne Koller","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kumar et al., 2011] M. Pawan Kumar, Haithem Turki, Dan Preston, and Daphne Koller. Learning specific-class seg- mentation from diverse data. In Proceedings of IEEE In- ternational Conference on Computer Vision, pages 1800- 1807, 2011.","title":"Learning specific-class segmentation from diverse data","authors":"M Pawan Kumar, Haithem Turki, Dan Preston, Daphne Koller","year":"2011","venue":"Proceedings of IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lee and Grauman, 2011] Yong Jae Lee and Kristen Grau- man. Learning the easy things first: Self-paced visual cat- egory discovery. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 1721- 1728, 2011.","title":"Learning the easy things first: Self-paced visual category discovery","authors":"Jae Yong, Kristen Lee, Grauman","year":"2011","venue":"Proceedings of IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pi et al., 2016] Te Pi, Xi Li, Zhongfei Zhang, Deyu Meng, Fei Wu, Jun Xiao, and Yueting Zhuang. Self-paced boost learning for classification. In Proceedings of the 25th International Joint Conference on Artificial Intelligence, pages 1932-1938, 2016.","title":"Self-paced boost learning for classification","authors":"Te Pi, Xi Li, Zhongfei Zhang, Deyu Meng, Fei Wu, Jun Xiao, Yueting Zhuang","year":"2016","venue":"Proceedings of the 25th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2016/276"},{"position":19,"raw":"[Ren et al., 2017] Yazhou Ren, Peng Zhao, Zenglin Xu, and Dezhong Yao. Balanced self-paced learning with feature corruption. In Proceedings of the International Joint Con- ference on Neural Networks, pages 2064-2071, 2017.","title":"Balanced self-paced learning with feature corruption","authors":"Yazhou Ren, Peng Zhao, Zenglin Xu, Dezhong Yao","year":"2017","venue":"Proceedings of the International Joint Conference on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shafieezadeh-Abadeh et al., 2015] Soroosh Shafieezadeh- Abadeh, Peyman Mohajerin Esfahani, and Daniel Kuhn. Distributionally robust logistic regression. In Advances in Neural Information Processing Systems, pages 1576- 1584, 2015.","title":"Distributionally robust logistic regression","authors":"Soroosh Shafieezadeh-Abadeh, Peyman Mohajerin Esfahani, Daniel Kuhn","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Supan Či Č and Ramanan, 2013] James Steven Supan Či Č and Deva Ramanan. Self-paced learning for long-term tracking. In Proceedings of IEEE Conference on Com- puter Vision and Pattern Recognition, pages 2379-2386, 2013.","title":"Self-paced learning for long-term tracking","authors":"Supan Či, Č, Ramanan, James Steven, Supan Či, Č, Deva Ramanan","year":"2013","venue":"Proceedings of IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tang et al., 2012] Kevin Tang, Vignesh Ramanathan, Fei- Fei Li, and Daphne Koller. Shifting weights: Adapting object detectors from image to video. In Advances in Neu- ral Information Processing Systems, pages 647-655, 2012.","title":"Shifting weights: Adapting object detectors from image to video","authors":"Kevin Tang, Vignesh Ramanathan, Fei-Fei Li, Daphne Koller","year":"2012","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tibshirani and Manning, 2014] Julie Tibshirani and Christopher D. Manning. Robust logistic regression using shift parameters. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 124-129, 2014.","title":"Robust logistic regression using shift parameters","authors":"Julie Tibshirani, Christopher D Manning","year":"2014","venue":"Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Walker and Duncan, 1967] Strother H. Walker and David B. Duncan. Estimation of the probability of an event as a function of several independent variables. Biometrika, 54(1-2):167-179, 1967.","title":"Estimation of the probability of an event as a function of several independent variables","authors":"H Strother, David B Walker, Duncan","year":"1967","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhao et al., 2015] Qian Zhao, Deyu Meng, Lu Jiang, Qi Xie, Zongben Xu, and Alexander G Hauptmann. Self- paced learning for matrix factorization. In Proceedings of AAAI Conference on Artificial Intelligence, pages 3196- 3202, 2015.","title":"Selfpaced learning for matrix factorization","authors":"Qian Zhao, Deyu Meng, Lu Jiang, Qi Xie, Zongben Xu, Alexander G Hauptmann","year":"2015","venue":"Proceedings of AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}