{"type":"ScholarlyArticle","title":"Teaching Semi-Supervised Classifier via Generalized Distillation","authors":[{"name":"Chen Gong","slug":"chen-gong","given":"Chen","family":"Gong","affiliation":"School of Computer Science and Engineering , Nanjing University of Science and Technology; Jiangsu Key Laboratory of Image and Video Understanding for Social Security"},{"name":"Xiaojun Chang","slug":"xiaojun-chang","given":"Xiaojun","family":"Chang","affiliation":"Language Technologies Institute , Carnegie Mellon University"},{"name":"Meng Fang","slug":"meng-fang","given":"Meng","family":"Fang","affiliation":"Tencent AI Lab"},{"name":"Jian Yang","slug":"jian-yang","given":"Jian","family":"Yang","affiliation":"School of Computer Science and Engineering , Nanjing University of Science and Technology; Jiangsu Key Laboratory of Image and Video Understanding for Social Security"}],"abstract":"Semi-Supervised Learning (SSL) is able to build reliable classifier with very scarce labeled examples by properly utilizing the abundant unlabeled examples. However, existing SSL algorithms often yield unsatisfactory performance due to the lack of supervision information. To address this issue, this paper formulates SSL as a Generalized Distillation (GD) problem, which treats existing SSL algorithm as a learner and introduces a teacher to guide the learner?s training process. Specifically, the intelligent teacher holds the privileged knowledge that ?explains? the training data but remains unknown to the learner, and the teacher should convey its rich knowledge to the imperfect learner through a specific teaching function. After that, the learner gains knowledge by ?imitating? the output of the teaching function under an optimization framework. Therefore, the learner in our algorithm learns from both the teacher and the training data, so its output can be substantially distilled and enhanced. By deriving the Rademacher complexity and error bounds of the proposed algorithm, the usefulness of the introduced teacher is theoretically demonstrated. The superiority of our algorithm to the related state-of-the-art methods has also been empirically demonstrated by the experiments on different datasets with various sources of privileged knowledge.","doi":"10.24963/ijcai.2018/298","url":"https://ijcai.org/proceedings/2018/298","pdf":"https://ijcai.org/proceedings/2018/0298.pdf","markdown":"https://ijcai.org/proceedings/2018/298.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/298","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":298,"pages":{"first":2156,"last":2162},"keywords":["Machine Learning: Classification","Machine Learning: Semi-Supervised Learning","Machine Learning: Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Ao et al., 2017] S. Ao, X. Li, and C. Ling. Fast generalized dis- tillation for semi-supervised domain adaptation. In AAAI, pages 1719-1725, 2017.","title":"Fast generalized distillation for semi-supervised domain adaptation","authors":"S Ao, X Li, C Ling","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bartlett and Mendelson, 2002] P. Bartlett and S. Mendelson. Rademacher and Gaussian complexities: risk bounds and struc- tural results. JMLR, 3:463-482, 2002.","title":"Rademacher and Gaussian complexities: risk bounds and structural results","authors":"P Bartlett, S Mendelson","year":"2002","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Belkin et al., 2006] M. Belkin, P. Niyogi, and V. Sindhwani. Man- ifold regularization: A geometric framework for learning from labeled and unlabeled examples. JMLR, 7(11):2399-2434, 2006.","title":"Manifold regularization: A geometric framework for learning from labeled and unlabeled examples","authors":"M Belkin, P Niyogi, V Sindhwani","year":"2006","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cai et al., 2006] D. Cai, X. He, J. Han, and H. Zhang. Orthogo- nal laplacianfaces for face recognition. TIP, 15(11):3608-3614, 2006.","title":"Orthogonal laplacianfaces for face recognition","authors":"D Cai, X He, J Han, H Zhang","year":"2006","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Celik et al., 2016] Z. Celik, D. Lopez-Paz, and P. McDaniel. Patient-driven privacy control through generalized distillation. arXiv:1611.08648, 2016.","title":"Patient-driven privacy control through generalized distillation","authors":"Z Celik, D Lopez-Paz, P Mcdaniel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chapelle et al., 2006] O. Chapelle, B. Schölkopf, and A. Zien. Semi-Supervised Learning. MIT Press, Cambridge, MA, 2006.","title":"Semi-Supervised Learning","authors":"O Chapelle, B Schölkopf, A Zien","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[El-Yaniv and Pechyony, 2009] R. El-Yaniv and D. Pechyony. Transductive rademacher complexity and its applications. JAIR, 35(1):193, 2009.","title":"Transductive rademacher complexity and its applications","authors":"R El-Yaniv, D Pechyony","year":"2009","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fang et al., 2014] Y. Fang, K. Chang, and H. Lauw. Graph-based semi-supervised learning: realizing pointwise smoothness prob- abilistically. In ICML, pages 406-414, 2014.","title":"Graph-based semi-supervised learning: realizing pointwise smoothness probabilistically","authors":"Y Fang, K Chang, H Lauw","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gammerman et al., 1998] A. Gammerman, V. Vovk, and V. Vap- nik. Learning by transduction. In UAI, pages 148-155, 1998.","title":"Learning by transduction","authors":"A Gammerman, V Vovk, V Vapnik","year":"1998","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gong et al., 2014] C. Gong, D. Tao, K. Fu, and J. Yang. ReLISH: Reliable label inference via smoothness hypothesis. In AAAI, pages 1840-1846, 2014.","title":"ReLISH: Reliable label inference via smoothness hypothesis","authors":"C Gong, D Tao, K Fu, J Yang","year":"2014","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gong et al., 2015a] C. Gong, T. Liu, D. Tao, K. Fu, E. Tu, and J. Yang. Deformed graph Laplacian for semisupervised learning. TNNLS, 26(10):2261-2274, Oct. 2015.","title":"Deformed graph Laplacian for semisupervised learning","authors":"Gong","year":"2015","venue":"TNNLS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gong et al., 2015b] C. Gong, D. Tao, K. Fu, and J. Yang. Fick's law assisted propagation for semisupervised learning. TNNLS, 26(9):2148-2162, 2015.","title":"Fick's law assisted propagation for semisupervised learning","authors":"Gong","year":"2015","venue":"TNNLS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gong et al., 2017a] C. Gong, D. Tao, W. Liu, L. Liu, and J. Yang. Label propagation via teaching-to-learn and learning-to-teach. TNNLS, 28(6):1452-1465, 2017.","title":"Label propagation via teaching-to-learn and learning-to-teach","authors":"Gong","year":"2017","venue":"TNNLS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Gong et al., 2017b] C. Gong, H. Zhang, J. Yang, and D. Tao. Learning with inadequate and incorrect supervision. In ICDM, pages 889-894. IEEE, 2017.","title":"Learning with inadequate and incorrect supervision","authors":"Gong","year":"2017","venue":"ICDM","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Grandvalet and Bengio, 2004] Y. Grandvalet and Y. Bengio. Semi- supervised learning by entropy minimization. In NIPS, vol- ume 17, pages 529-536, 2004.","title":"Semisupervised learning by entropy minimization","authors":"Y Grandvalet, Y Bengio","year":"2004","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hinton et al., 2015] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"G Hinton, O Vinyals, J Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Hu et al., 2016] Z. Hu, X. Ma, Z. Liu, E. Hovy, and E. Xing. Har- nessing deep neural networks with logic rules. In ACL, pages 2410-2420, 2016.","title":"Harnessing deep neural networks with logic rules","authors":"Z Hu, X Ma, Z Liu, E Hovy, E Xing","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Joachims, 1999] T. Joachims. Transductive inference for text clas- sification using support vector machines. In ICML, volume 99, pages 200-209, 1999.","title":"Transductive inference for text classification using support vector machines","authors":"T Joachims","year":"1999","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Krizhevsky and Hinton, 2009] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. 2009.","title":"Learning multiple layers of features from tiny images","authors":"A Krizhevsky, G Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Krizhevsky et al., 2012] A. Krizhevsky, I. Sutskever, and G. Hin- ton. ImageNet classification with deep convolutional neural net- works. In NIPS, pages 1097-1105, 2012.","title":"ImageNet classification with deep convolutional neural networks","authors":"A Krizhevsky, I Sutskever, G Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li and Zhou, 2015] Y. Li and Z. Zhou. Towards making unlabeled data never hurt. TPAMI, 37(1):175-188, 2015.","title":"Towards making unlabeled data never hurt","authors":"Y Li, Z Zhou","year":"2015","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Li et al., 2009] Y. Li, J. Kwok, and Z. Zhou. Semi-supervised learning using label mean. In ICML, pages 633-640. ACM, 2009.","title":"Semi-supervised learning using label mean","authors":"Y Li, J Kwok, Z Zhou","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Li et al., 2016a] W. Li, D. Dai, M. Tan, D. Xu, and L. Gool. Fast algorithms for linear and kernel svm+. In CVPR, pages 2258- 2266, 2016.","title":"Fast algorithms for linear and kernel svm+","authors":"Li","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Li et al., 2016b] Y. Li, J. Kwok, and Z. Zhou. Towards safe semi- supervised learning for multivariate performance measures. In AAAI, pages 1816-1822, 2016.","title":"Towards safe semisupervised learning for multivariate performance measures","authors":"Li","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Liu et al., 2017] Y. Liu, Y. Guo, H. Wang, F. Nie, and H. Huang. Semi-supervised classifications via elastic and robust embedding. In AAAI, pages 2294-2300, 2017.","title":"Semi-supervised classifications via elastic and robust embedding","authors":"Y Liu, Y Guo, H Wang, F Nie, H Huang","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Lopez-Paz et al., 2016] D. Lopez-Paz, B. Schölkopf, L. Bottou, and V. Vapnik. Unifying distillation and privileged information. In ICLR, 2016.","title":"Unifying distillation and privileged information","authors":"D Lopez-Paz, B Schölkopf, L Bottou, V Vapnik","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Pereira et al., 2014] J. Pereira, E. Coviello, G. Doyle, N. Rasiwa- sia, G. Lanckriet, R. Levy, and N. Vasconcelos. On the role of correlation and abstraction in cross-modal multimedia retrieval. TPAMI, 36(3):521-535, 2014.","title":"On the role of correlation and abstraction in cross-modal multimedia retrieval","authors":"J Pereira, E Coviello, G Doyle, N Rasiwasia, G Lanckriet, R Levy, N Vasconcelos","year":"2014","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Simonyan and Zisserman, 2014] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv: 1409.1556v6, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"K Simonyan, A Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Vapnik and Izmailov, 2015] V. Vapnik and R. Izmailov. Learning using privileged information: similarity control and knowledge transfer. JMLR, 16:2023-2049, 2015.","title":"Learning using privileged information: similarity control and knowledge transfer","authors":"V Vapnik, R Izmailov","year":"2015","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2009] J. Wang, F. Wang, and C. Zhang. Linear neigh- borhood propagation and its applications. TPAMI, 31(9):1600- 1615, 2009.","title":"Linear neighborhood propagation and its applications","authors":"J Wang, F Wang, C Zhang","year":"2009","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yan et al., 2016] Y. Yan, Z. Xu, I. Tsang, G. Long, and Y. Yang. Robust semi-supervised learning through label aggregation. In AAAI, pages 2244-2250, 2016.","title":"Robust semi-supervised learning through label aggregation","authors":"Y Yan, Z Xu, I Tsang, G Long, Y Yang","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang et al., 2016] X. Yang, M. Wang, L. Zhang, and D. Tao. Em- pirical risk minimization for metric learning using privileged in- formation. In IJCAI, pages 2266-2272, 2016.","title":"Empirical risk minimization for metric learning using privileged information","authors":"Yang","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/323"},{"position":33,"raw":"[You et al., 2017] S. You, C. Xu, Y. Wang, C. Xu, and D. Tao. Priv- ileged multi-label learning. arXiv:1701.07194, 2017.","title":"Privileged multi-label learning","authors":"S You, C Xu, Y Wang, C Xu, D Tao","year":"2017","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2017/466"},{"position":34,"raw":"[Zhou et al., 2004] D. Zhou, O. Bousquet, T. Lal, J. Weston, and B. Schölkopf. Learning with local and global consistency. In NIPS, pages 321-328, 2004.","title":"Learning with local and global consistency","authors":"D Zhou, O Bousquet, T Lal, J Weston, B Schölkopf","year":"2004","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhou et al., 2016] J. Zhou, X. Xu, S. Pan, I. Tsang, Z. Qin, and R. Goh. Transfer hashing with privileged information. arX- iv:1605.04034, 2016.","title":"Transfer hashing with privileged information","authors":"J Zhou, X Xu, S Pan, I Tsang, Z Qin, R Goh","year":"2016","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2016/344"},{"position":36,"raw":"[Zhu and Goldberg, 2009] X. Zhu and B. Goldberg. Introduction to Semi-Supervised Learning. Morgan & Claypool Publishers, 2009.","title":"Introduction to Semi-Supervised Learning","authors":"Goldberg Zhu, X Zhu, B Goldberg","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhu et al., 2003] X. Zhu, X. Ghahramani, and J. Lafferty. Semi- supervised learning using Gaussian fields and harmonic function- s. In ICML, pages 912-919, 2003.","title":"Semisupervised learning using Gaussian fields and harmonic functions","authors":"Zhu","year":"2003","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}