{"type":"ScholarlyArticle","title":"Multi-Class Learning using Unlabeled Samples: Theory and Algorithm","authors":[{"name":"Jian Li","slug":"jian-li","given":"Jian","family":"Li","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences; School of Cyber Security , University of Chinese Academy of Sciences"},{"name":"Yong Liu","slug":"yong-liu","given":"Yong","family":"Liu","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences"},{"name":"Rong Yin","slug":"rong-yin","given":"Rong","family":"Yin","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences; School of Cyber Security , University of Chinese Academy of Sciences"},{"name":"Weiping Wang","slug":"weiping-wang","given":"Weiping","family":"Wang","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences"}],"abstract":"In this paper, we investigate the generalization performance of multi-class classification, for which we obtain a shaper error bound by using the notion of local Rademacher complexity and additional unlabeled samples, substantially improving the state-of-the-art bounds in existing multi-class learning methods. The statistical learning motivates us to devise an efficient multi-class learning framework with the local Rademacher complexity and Laplacian regularization. Coinciding with the theoretical analysis, experimental results demonstrate that the stated approach achieves better performance.","doi":"10.24963/ijcai.2019/399","url":"https://ijcai.org/proceedings/2019/399","pdf":"https://ijcai.org/proceedings/2019/0399.pdf","markdown":"https://ijcai.org/proceedings/2019/399.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/399","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":399,"pages":{"first":2880,"last":2886},"keywords":["Machine Learning: Classification","Machine Learning: Semi-Supervised Learning","Machine Learning: Learning Theory"],"videos":[],"references":[{"position":1,"raw":"[Allwein et al., 2000] Erin L Allwein, Robert E Schapire, and Yoram Singer. Reducing multiclass to binary: A uni- fying approach for margin classifiers. Journal of Machine Learning Research, 1:113-141, 2000.","title":"Reducing multiclass to binary: A unifying approach for margin classifiers","authors":"Robert E Erin L Allwein, Yoram Schapire, Singer","year":"2000","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Balcan and Blum, 2010] Maria-Florina Balcan and Avrim Blum. A discriminative model for semi-supervised learn- ing. Journal of the ACM (JACM), 57(3):19, 2010.","title":"A discriminative model for semi-supervised learning","authors":"Maria-Florina Balcan, Avrim Blum","year":"2010","venue":"Journal of the ACM (JACM)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bartlett et al., 2005] Peter L Bartlett, Olivier Bousquet, Shahar Mendelson, et al. Local rademacher complexities. The Annals of Statistics, 33(4):1497-1537, 2005.","title":"Local rademacher complexities","authors":"Olivier Peter L Bartlett, Bousquet, Shahar Mendelson","year":"2005","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Camoriano et al., 2016] Raffaello Camoriano, Tomás An- gles, Alessandro Rudi, and Lorenzo Rosasco. Nytro: When subsampling meets early stopping. In Artificial In- telligence and Statistics, pages 1403-1411, 2016.","title":"Nytro: When subsampling meets early stopping","authors":"Raffaello Camoriano, Tomás Angles, Alessandro Rudi, Lorenzo Rosasco","year":"2016","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cortes et al., 2013] Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Multi-class classification with max- imum margin multiple kernel. In Proceedings of the 30th International Conference on Machine Learning (ICML), pages 46-54, 2013.","title":"Multi-class classification with maximum margin multiple kernel","authors":"Corinna Cortes, Mehryar Mohri, Afshin Rostamizadeh","year":"2013","venue":"Proceedings of the 30th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Daniely et al., 2015] Amit Daniely, Sivan Sabato, Shai Ben- David, and Shai Shalev-Shwartz. Multiclass learnability and the erm principle. Journal of Machine Learning Re- search, 16(1):2377-2404, 2015.","title":"Multiclass learnability and the erm principle","authors":"Amit Daniely, Sivan Sabato, Shai Ben-David, Shai Shalev-Shwartz","year":"2015","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Koltchinskii and Panchenko, 2002] Vladimir Koltchinskii and Dmitry Panchenko. Empirical margin distributions and bounding the generalization error of combined classifiers. The Annals of Statistics, 30:1-50, 2002.","title":"Vladimir Koltchinskii and Dmitry Panchenko. Empirical margin distributions and bounding the generalization error of combined classifiers","authors":"Panchenko Koltchinskii","year":"2002","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Koltchinskii et al., 2001] Vladimir Koltchinskii, Dmitriy Panchenko, and Fernando Lozano. Some new bounds on the generalization error of combined classifiers. In Advances in Neural Information Processing Systems 14 (NIPS), pages 245-251, 2001.","title":"Some new bounds on the generalization error of combined classifiers","authors":"Vladimir Koltchinskii, Dmitriy Panchenko, Fernando Lozano","year":"2001","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Koltchinskii, 2011] Vladimir Koltchinskii. Oracle Inequal- ities in Empirical Risk Minimization and Sparse Recovery Problems. Springer, 2011.","title":"Oracle Inequalities in Empirical Risk Minimization and Sparse Recovery Problems","authors":"Vladimir Koltchinskii","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lei et al., 2015] Yunwen Lei, Urun Dogan, Alexander Binder, and Marius Kloft. Multi-class SVMs: From tighter data-dependent generalization bounds to novel algorithms. In Advances in Neural Information Processing Systems 28 (NIPS), pages 2035-2043, 2015.","title":"Multi-class SVMs: From tighter data-dependent generalization bounds to novel algorithms","authors":"Lei","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li and Guo, 2015] Xin Li and Yuhong Guo. Max-margin zero-shot learning for multi-class classification. In Artifi- cial Intelligence and Statistics, pages 626-634, 2015.","title":"Max-margin zero-shot learning for multi-class classification","authors":"Guo ; Li, Xin Li, Yuhong Guo","year":"2015","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2015] Xin Li, Yuhong Guo, and Dale Schuurmans. Semi-supervised zero-shot classification with label rep- resentation learning. In Proceedings of the IEEE Inter- national Conference on Computer Vision (ICCV), pages 4211-4219, 2015.","title":"Semi-supervised zero-shot classification with label representation learning","authors":"Xin Li, Yuhong Guo, Dale Schuurmans","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2017a] Jian Li, Yong Liu, Hailun Lin, Yinliang Yue, and Weiping Wang. Efficient kernel selection via spectral analysis. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI), pages 2124-2130, 2017.","title":"Efficient kernel selection via spectral analysis","authors":"Li","year":"2017","venue":"Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2017/295"},{"position":14,"raw":"[Li et al., 2017b] Xiao Li, Min Fang, and Jinqiao Wu. Zero- shot classification by transferring knowledge and preserv- ing data structure. Neurocomputing, 238:76-83, 2017.","title":"Zeroshot classification by transferring knowledge and preserving data structure","authors":"Li","year":"2017","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2018] Jian Li, Yong Liu, Rong Yin, Hua Zhang, Lizhong Ding, and Weiping Wang. Multi-class learning: From theory to algorithm. In Advances in Neural Informa- tion Processing Systems 31 (NeurIPS), pages 1591-1600, 2018.","title":"Multi-class learning: From theory to algorithm","authors":"Jian Li, Yong Liu, Rong Yin, Hua Zhang, Lizhong Ding, Weiping Wang","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu and Chang, 2009] Wei Liu and Shih-Fu Chang. Robust multi-class transductive learning with graphs. In Proceed- ings of the 22nd IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 381-388. IEEE, 2009.","title":"Robust multi-class transductive learning with graphs","authors":"Wei Liu, Shih-Fu Chang","year":"2009","venue":"Proceedings of the 22nd IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2017] Yong Liu, Shizhong Liao, Hailun Lin, Yin- liang Yue, and Weiping Wang. Infinite kernel learn- ing: generalization bounds and algorithms. In Proceed- ings of the 31st AAAI Conference on Artificial Intelligence (AAAI), pages 2280-2286, 2017.","title":"Infinite kernel learning: generalization bounds and algorithms","authors":"Yong Liu, Shizhong Liao, Hailun Lin, Yinliang Yue, Weiping Wang","year":"2017","venue":"Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2019] Yong Liu, Shizhong Liao, Shali Jiang, Lizhong Ding, Hailun Lin, and Weiping Wang. Fast cross- validation for kernel-based algorithms. IEEE transactions on pattern analysis and machine intelligence, 2019.","title":"Fast crossvalidation for kernel-based algorithms","authors":"Yong Liu, Shizhong Liao, Shali Jiang, Lizhong Ding, Hailun Lin, Weiping Wang","year":"2019","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lu et al., 2015] Canyi Lu, Changbo Zhu, Chunyan Xu, Shuicheng Yan, and Zhouchen Lin. Generalized singu- lar value thresholding. In Proceedings of the 29th AAAI Conference on Artificial Intelligence (AAAI), pages 1805- 1811, 2015.","title":"Generalized singular value thresholding","authors":"Lu","year":"2015","venue":"Proceedings of the 29th AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Maximov and Reshetova, 2016] Yu Maximov and Daria Reshetova. Tight risk bounds for multi-class margin classi- fiers. Pattern Recognition and Image Analysis, 26(4):673- 680, 2016.","title":"Tight risk bounds for multi-class margin classifiers","authors":"Yu Maximov, Daria Reshetova","year":"2016","venue":"Pattern Recognition and Image Analysis","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Maximov et al., 2018] Yury Maximov, Massih-Reza Amini, and Zaid Harchaoui. Rademacher complexity bounds for a penalized multi-class semi-supervised algorithm. Journal of Artificial Intelligence Research, 61:761-786, 2018.","title":"Rademacher complexity bounds for a penalized multi-class semi-supervised algorithm","authors":"Yury Maximov, Massih-Reza Amini, Zaid Harchaoui","year":"2018","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Oneto et al., 2011] Luca Oneto, Davide Anguita, Alessan- dro Ghio, and Sandro Ridella. The impact of unlabeled patterns in rademacher complexity theory for kernel clas- sifiers. In Advances in Neural Information Processing Sys- tems 24 (NIPS), pages 585-593, 2011.","title":"The impact of unlabeled patterns in rademacher complexity theory for kernel classifiers","authors":"Luca Oneto, Davide Anguita, Alessandro Ghio, Sandro Ridella","year":"2011","venue":"Advances in Neural Information Processing Systems 24 (NIPS)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Oneto et al., 2015] Luca Oneto, Alessandro Ghio, Sandro Ridella, and Davide Anguita. Local rademacher complex- ity: Sharper risk bounds with and without unlabeled sam- ples. Neural Networks, 65:115-125, 2015.","title":"Local rademacher complexity: Sharper risk bounds with and without unlabeled samples","authors":"Luca Oneto, Alessandro Ghio, Sandro Ridella, Davide Anguita","year":"2015","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shalev-Shwartz et al., 2011] Shai Shalev-Shwartz, Yoram Singer, Nathan Srebro, and Andrew Cotter. Pegasos: Pri- mal estimated sub-gradient solver for svm. Mathematical programming, 127(1):3-30, 2011.","title":"Pegasos: Primal estimated sub-gradient solver for svm","authors":"Shai Shalev-Shwartz, Yoram Singer, Nathan Srebro, Andrew Cotter","year":"2011","venue":"Mathematical programming","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2016] Chang Xu, Tongliang Liu, Dacheng Tao, and Chao Xu. Local rademacher complexity for multi- label learning. IEEE Transactions on Image Processing, 25(3):1495-1507, 2016.","title":"Local rademacher complexity for multilabel learning","authors":"Chang Xu, Tongliang Liu, Dacheng Tao, Chao Xu","year":"2016","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}