{"type":"ScholarlyArticle","title":"Rademacher Complexity Bounds for a Penalized Multi-class Semi-supervised Algorithm (Extended Abstract)","authors":[{"name":"Yury Maximov","slug":"yury-maximov","given":"Yury","family":"Maximov","affiliation":"Skolkovo Institute of Science and Technology , Center for Energy Systems; Los Alamos National Laboratory , Theoretical Division T-4 and CNLS"},{"name":"Massih-Reza Amini","slug":"massih-reza-amini","given":"Massih-Reza","family":"Amini","affiliation":"University of Washington"},{"name":"Zaid Harchaoui","slug":"zaid-harchaoui","given":"Zaid","family":"Harchaoui","affiliation":"Université Grenoble Alpes"}],"abstract":"We propose Rademacher complexity bounds for multi-class classifiers trained with a two-step semi-supervised model. In the first step, the algorithm partitions the partially labeled data and then identifies dense clusters containing k predominant classes using the labeled training examples such that the proportion of their non-predominant classes is below a fixed threshold stands for clustering consistency. In the second step,  a classifier is trained by minimizing a margin empirical loss over the labeled training set and a penalization term measuring the disability of the learner to predict the k predominant classes of the identified clusters. The resulting data-dependent generalization error bound involves the margin distribution of the classifier, the stability of the clustering technique used in the first step and Rademacher complexity terms corresponding to partially labeled training data. Our theoretical result exhibit convergence rates extending those proposed in the literature for the binary case, and experimental results on different multi-class classification problems show empirical evidence that supports the theory.","doi":"10.24963/ijcai.2018/800","url":"https://ijcai.org/proceedings/2018/800","pdf":"https://ijcai.org/proceedings/2018/0800.pdf","markdown":"https://ijcai.org/proceedings/2018/800.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/800","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Journal track","track":"Journal track","number":800,"pages":{"first":5637,"last":5641},"keywords":["Machine Learning: Semi-Supervised Learning","Machine Learning: Machine Learning","Machine Learning: Clustering","Machine Learning: Classification"],"videos":[],"references":[{"position":1,"raw":"[Amini and Usunier, 2015] Massih-Reza Amini and Nicolas Usunier. Learning with Partially Labeled and Interdepen- dent Data. Springer, 2015.","title":"Learning with Partially Labeled and Interdependent Data","authors":"Massih-Reza Amini, Nicolas Usunier","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amini et al., 2008] Massih-Reza Amini, Franc ¸ois Lavio- lette, and Nicolas Usunier. A transductive bound for the voted classifier with an application to semi-supervised learning. In Advances in Neural Information Processing Systems (NIPS 22), pages 65-72, 2008.","title":"A transductive bound for the voted classifier with an application to semi-supervised learning","authors":"Massih-Reza Amini, Franc ¸ois Laviolette, Nicolas Usunier","year":"2008","venue":"Advances in Neural Information Processing Systems (NIPS 22)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Balcan and Blum, 2010] Maria-Florina Balcan and Avrim Blum. A discriminative model for semi-supervised learn- ing. J. ACM, 57(3), 2010.","title":"A discriminative model for semi-supervised learning","authors":"Maria-Florina Balcan, Avrim Blum","year":"2010","venue":"J. ACM","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ben-David et al., 2008] Shai Ben-David, Tyler Lu, and Dávid Pál. Does unlabeled data provably help? worst- case analysis of the sample complexity of semi-supervised learning. In 21st Annual Conference on Learning Theory (COLT), pages 33-44, 2008.","title":"Does unlabeled data provably help? worstcase analysis of the sample complexity of semi-supervised learning","authors":"Shai Ben-David, Tyler Lu, Dávid Pál","year":"2008","venue":"21st Annual Conference on Learning Theory (COLT)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Blum and Mitchell, 1998] Avrim Blum and Tom M. Mitchell. Combining labeled and unlabeled sata with co-training. In 11th Annual Conference on Computational Learning Theory (COLT), pages 92-100, 1998.","title":"Combining labeled and unlabeled sata with co-training","authors":"Avrim Blum, Tom M Mitchell","year":"1998","venue":"11th Annual Conference on Computational Learning Theory (COLT)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Castelli and Cover, 1995] Vittorio Castelli and Thomas M. Cover. On the exponential value of labeled samples. Pat- tern Recognition Letters, 16(1):105-111, 1995.","title":"On the exponential value of labeled samples","authors":"Vittorio Castelli, Thomas M Cover","year":"1995","venue":"Pattern Recognition Letters","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chapelle et al., 2006] Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. Semi-supervised learning. MIT press, 2006.","title":"Semi-supervised learning","authors":"Olivier Chapelle, Bernhard Schölkopf, Alexander Zien","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[El-Yaniv and Pechyony, 2009] Ran El-Yaniv and Dmitry Pechyony. Transductive rademacher complexity and its applications. Journal of Artificial Intelligence Research (JAIR), 35:193-234, 2009.","title":"Ran El-Yaniv and Dmitry Pechyony. Transductive rademacher complexity and its applications","authors":"Pechyony El-Yaniv","year":"2009","venue":"Journal of Artificial Intelligence Research (JAIR)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kääriäinen, 2005] Matti Kääriäinen. Generalization error bounds using unlabeled data. In Learning Theory, 18th An- nual Conference on Learning Theory (COLT), pages 127- 142, 2005.","title":"Generalization error bounds using unlabeled data","authors":"Matti Kääriäinen","year":"2005","venue":"Learning Theory, 18th Annual Conference on Learning Theory (COLT)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Knuth, 1976] Donald E. Knuth. Big omicron and big omega and big theta. SIGACT News, 8(2):18-24, 1976.","title":"Big omicron and big omega and big theta","authors":"Donald E Knuth","year":"1976","venue":"SIGACT News","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Koltchinskii and Panchenko, 2002] Vladimir Koltchinskii and Dmitry Panchenko. Empirical margin distributions and bounding the generalization error of combined classifiers. Annals of Statistics, pages 1-50, 2002.","title":"Vladimir Koltchinskii and Dmitry Panchenko. Empirical margin distributions and bounding the generalization error of combined classifiers","authors":"Panchenko Koltchinskii","year":"2002","venue":"Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kuznetsov et al., 2014] Vitaly Kuznetsov, Mehryar Mohri, and Umar Syed. Multi-class deep boosting. In Advances in Neural Information Processing Systems, pages 2501- 2509, 2014.","title":"Multi-class deep boosting","authors":"Vitaly Kuznetsov, Mehryar Mohri, Umar Syed","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lei et al., 2015] Yunwen Lei, Urun Dogan, Alexander Binder, and Marius Kloft. Multi-class svms: From tighter data-dependent generalization bounds to novel algorithms. In Advances in Neural Information Processing Systems, pages 2035-2043, 2015.","title":"Multi-class svms: From tighter data-dependent generalization bounds to novel algorithms","authors":"Lei","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Leskes, 2005] Boaz Leskes. The value of agreement, a new boosting algorithm. In 18th Annual Conference on Learn- ing Theory (COLT), pages 95-110, 2005.","title":"Boaz Leskes. The value of agreement, a new boosting algorithm","authors":"Leskes","year":"2005","venue":"18th Annual Conference on Learning Theory (COLT)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Luxburg et al., 2004] Ulrike Von Luxburg, Olivier Bous- quet, and Mikhail Belkin. On the convergence of spec- tral clustering on random samples: The normalized case. In 17th Annual Conference on Learning Theory (COLT), pages 457-471, 2004.","title":"On the convergence of spectral clustering on random samples: The normalized case","authors":"Luxburg","year":"2004","venue":"17th Annual Conference on Learning Theory (COLT)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Luxburg, 2010] Ulrike Von Luxburg. Clustering stability: An overview. Journal Foundations and Trends in Machine Learning, 2(3):235-274, 2010.","title":"Clustering stability: An overview","authors":"Ulrike Von, Luxburg","year":"2010","venue":"Journal Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Maximov and Reshetova, 2016] Yury Maximov and Daria Reshetova. Tight risk bounds for multi-class margin classi- fiers. Pattern Recognition and Image Analysis, 26(4):673- 680, 2016.","title":"Yury Maximov and Daria Reshetova. Tight risk bounds for multi-class margin classifiers","authors":"Reshetova Maximov","year":"2016","venue":"Pattern Recognition and Image Analysis","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Maximov et al., 2018] Yury Maximov, Massih-Reza Amini, and Zaid Harchaoui. Rademacher complexity bounds for a penalized multi-class semi-supervised algo- rithm. Journal of Artificial Intelligence Research (JAIR), 61:761-786, 2018.","title":"Rademacher complexity bounds for a penalized multi-class semi-supervised algorithm","authors":"Yury Maximov, Massih-Reza Amini, Zaid Harchaoui","year":"2018","venue":"Journal of Artificial Intelligence Research (JAIR)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rakhlin and Caponnetto, 2006] Alexander Rakhlin and An- drea Caponnetto. Stability of k-means clustering. In Ad- vances in Neural Information Processing Systems (NIPS 19), pages 1121-1128, 2006.","title":"Alexander Rakhlin and Andrea Caponnetto. Stability of k-means clustering","authors":"Caponnetto Rakhlin","year":"2006","venue":"Advances in Neural Information Processing Systems (NIPS 19)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Seeger, 2000] Matthias Seeger. Learning with labeled and unlabeled data. Technical report, Institute for Adaptive and Neural Computation, University of Edinburgh, 2000.","title":"Learning with labeled and unlabeled data","authors":"Matthias Seeger","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shamir and Tishby, 2007] Ohad Shamir and Naftali Tishby. Cluster stability for finite samples. In Advances in Neural Information Processing Systems (NIPS 20), pages 1297- 1304, 2007.","title":"Cluster stability for finite samples","authors":"Tishby Shamir, Ohad Shamir, Naftali Tishby","year":"2007","venue":"Advances in Neural Information Processing Systems (NIPS 20)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Thiagarajan et al., 2011] Jayaraman J. Thiagarajan, Karthikeyan Natesan Ramamurthy, and Andreas Spanias. Optimality and stability of the k-hyperline clustering algorithm. Pattern Recognition Letters, 32(9):1299-1304, 2011.","title":"Optimality and stability of the k-hyperline clustering algorithm","authors":"J Jayaraman, Karthikeyan Thiagarajan, Andreas Natesan Ramamurthy, Spanias","year":"2011","venue":"Pattern Recognition Letters","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Urner et al., 2011] Ruth Urner, Shai Shalev-Shwartz, and Shai Ben-David. Access to unlabeled data can speed up prediction time. In 28th International Conference on Ma- chine Learning (ICML), pages 641-648, 2011.","title":"Access to unlabeled data can speed up prediction time","authors":"Ruth Urner, Shai Shalev-Shwartz, Shai Ben-David","year":"2011","venue":"28th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vapnik, 2000] Vladimir Vapnik. The nature of statistical learning theory. Springer Science & Business Media, 2000.","title":"The nature of statistical learning theory","authors":"Vladimir Vapnik","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou et al., 2003] Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. In Advances in Neural Information Processing Systems (NIPS 16), pages 321-328, 2003.","title":"Learning with local and global consistency","authors":"Zhou","year":"2003","venue":"Advances in Neural Information Processing Systems (NIPS 16)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhu, 2002] Xiaojin Zhu. Learning from labeled and unla- beled data with label propagation. Technical Report CMU- CALD-02-107, Carnegie Mellon University, 2002.","title":"Learning from labeled and unlabeled data with label propagation","authors":"Xiaojin Zhu","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhu, 2005] Xiaojin Zhu. Semi-supervised learning litera- ture survey. technical report 1530. Technical report, De- partment of Computer Sciences, University of Wisconsin, 2005.","title":"Semi-supervised learning literature survey","authors":"Xiaojin Zhu","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}