{"type":"ScholarlyArticle","title":"Tri-net for Semi-Supervised Deep Learning","authors":[{"name":"Dong-Dong Chen","slug":"dong-dong-chen","given":"Dong-Dong","family":"Chen","affiliation":"National Key Laboratory for Novel Software Technology Nanjing University , Nanjing 210023 , China"},{"name":"Wei Wang","slug":"wei-wang","given":"Wei","family":"Wang","affiliation":"National Key Laboratory for Novel Software Technology Nanjing University , Nanjing 210023 , China"},{"name":"Wei Gao","slug":"wei-gao","given":"Wei","family":"Gao","affiliation":"National Key Laboratory for Novel Software Technology Nanjing University , Nanjing 210023 , China"},{"name":"Zhi-Hua Zhou","slug":"zhi-hua-zhou","given":"Zhi-Hua","family":"Zhou","affiliation":"National Key Laboratory for Novel Software Technology Nanjing University , Nanjing 210023 , China"}],"abstract":"Deep neural networks have witnessed great successes in various real applications, but it requires a large number of labeled data for training. In this paper, we propose tri-net, a  deep neural network which is able to use  massive unlabeled data to help  learning with limited labeled data. We consider model initialization, diversity augmentation and pseudo-label editing simultaneously. In our work, we utilize output smearing to  initialize modules, use fine-tuning on labeled data to  augment diversity and eliminate unstable pseudo-labels to alleviate the influence of suspicious pseudo-labeled data.\r\nExperiments show that our method achieves the best performance  in comparison with state-of-the-art semi-supervised deep learning methods. In particular, it achieves 8.30% error rate on CIFAR-10 by using only 4000 labeled examples.","doi":"10.24963/ijcai.2018/278","url":"https://ijcai.org/proceedings/2018/278","pdf":"https://ijcai.org/proceedings/2018/0278.pdf","markdown":"https://ijcai.org/proceedings/2018/278.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/278","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":278,"pages":{"first":2014,"last":2020},"keywords":["Machine Learning: Deep Learning","Machine Learning: Semi-Supervised Learning"],"videos":[],"references":[{"position":1,"raw":"[Abbasnejad et al., 2017] Ehsan Abbasnejad, Anthony R. Dick, and Anton van den Hengel. Infinite variational au- toencoder for semi-supervised learning. In CVPR, pages 781-790, 2017.","title":"Infinite variational au-toencoder for semi-supervised learning","authors":"Ehsan Abbasnejad, Anthony R Dick, Anton Van Den, Hengel","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ardehaly and Culotta, 2017] Ehsan Mohammady Ardehaly and Aron Culotta. Co-training for demographic classifica- tion using deep learning from label proportions. In ICDM Workshop, pages 1017-1024, 2017.","title":"Co-training for demographic classification using deep learning from label proportions","authors":"Ehsan Mohammady, Ardehaly, Aron Culotta","year":"2017","venue":"ICDM Workshop","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bachman et al., 2014] Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. In NIPS, pages 3365-3373, 2014.","title":"Learning with pseudo-ensembles","authors":"Philip Bachman, Ouais Alsharif, Doina Precup","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Balcan and Blum, 2010] Maria-Florina Balcan and Avrim Blum. A discriminative model for semi-supervised learn- ing. Journal of the ACM, 57(3):19:1-19:46, 2010.","title":"A discriminative model for semi-supervised learning","authors":"Maria-Florina Balcan, Avrim Blum","year":"2010","venue":"Journal of the ACM","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Balcan et al., 2004] Maria-Florina Balcan, Avrim Blum, and Ke Yang. Co-training and expansion: Towards bridg- ing theory and practice. In NIPS, pages 89-96, 2004.","title":"Co-training and expansion: Towards bridging theory and practice","authors":"Maria-Florina Balcan, Avrim Blum, Ke Yang","year":"2004","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Blum and Mitchell, 1998] Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In COLT, pages 92-100, 1998.","title":"Combining labeled and unlabeled data with co-training","authors":"Avrim Blum, Tom Mitchell","year":"1998","venue":"COLT","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Breiman, 2000] Leo Breiman. Randomizing outputs to in- crease prediction accuracy. Machine Learning, 40(3):229- 242, 2000.","title":"Randomizing outputs to increase prediction accuracy","authors":"Leo Breiman, Breiman","year":"2000","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chapelle et al., 2006] Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. Semi-supervised learning. MIT Press, 2006.","title":"Semi-supervised learning","authors":"Olivier Chapelle, Bernhard Schölkopf, Alexander Zien","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cheng et al., 2016] Yanhua Cheng, Xin Zhao, Rui Cai, Zhi- wei Li, Kaiqi Huang, and Yong Rui. Semi-supervised mul- timodal deep learning for RGB-D object recognition. In IJCAI, pages 3345-3351, 2016.","title":"Semi-supervised multimodal deep learning for RGB-D object recognition","authors":"Yanhua Cheng, Xin Zhao, Rui Cai, Zhiwei Li, Kaiqi Huang, Yong Rui","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/473"},{"position":10,"raw":"[Dai et al., 2017] Zihang Dai, Zhilin Yang, Fan Yang, William W. Cohen, and Ruslan Salakhutdinov. Good semi- supervised learning that requires a bad GAN. In NIPS, pages 6513-6523, 2017.","title":"Good semisupervised learning that requires a bad GAN","authors":"Dai","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Girshick et al., 2014] Ross B. Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierar- chies for accurate object detection and semantic segmen- tation. In CVPR, pages 580-587, 2014.","title":"Rich feature hierarchies for accurate object detection and semantic segmentation","authors":"Ross B Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Goldman and Zhou, 2000] Sally A. Goldman and Yan Zhou. Enhancing supervised learning with unlabeled data. In ICML, pages 327-334, 2000.","title":"Enhancing supervised learning with unlabeled data","authors":"Sally A Goldman, Yan Zhou","year":"2000","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Goodfellow et al., 2014] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. CoRR, abs/1412.6572, 2014.","title":"Explaining and harnessing adversarial examples","authors":"Ian J Goodfellow, Jonathon Shlens, Christian Szegedy","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In ICML, pages 448-456, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kingma et al., 2014] Diederik P. Kingma, Shakir Mo- hamed, Danilo Jimenez Rezende, and Max Welling. Semi- supervised learning with deep generative models. In NIPS, pages 3581-3589, 2014.","title":"Semisupervised learning with deep generative models","authors":"P Diederik, Shakir Kingma, Danilo Mohamed, Max Jimenez Rezende, Welling","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1106- 1114, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Laine and Aila, 2016] Samuli Laine and Timo Aila. Tem- poral ensembling for semi-supervised learning. CoRR, abs/1610.02242, 2016.","title":"Temporal ensembling for semi-supervised learning","authors":"Laine, Aila ;, Samuli Laine, Timo Aila","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Luo et al., 2017] Yucen Luo, Jun Zhu, Mengxi Li, Yong Ren, and Bo Zhang. Smooth neighbors on teacher graphs for semi-supervised learning. CoRR, abs/1711.00258, 2017.","title":"Smooth neighbors on teacher graphs for semi-supervised learning","authors":"Luo","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Maaløe et al., 2016] Lars Maaløe, Casper Kaae Sønderby, Søren Kaae Sønderby, and Ole Winther. Auxiliary deep generative models. In ICML, pages 1445-1453, 2016.","title":"Auxiliary deep generative models","authors":"Maaløe","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Miyato et al., 2017] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. CoRR, abs/1704.03976, 2017.","title":"Virtual adversarial training: a regularization method for supervised and semi-supervised learning","authors":"Miyato","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Park et al., 2018] Sungrae Park, Jun-Keon Park, Su-Jin Shin, and Il-Chul Moon. Adversarial dropout for super- vised and semi-supervised learning. In AAAI, 2018.","title":"Adversarial dropout for supervised and semi-supervised learning","authors":"Park","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rasmus et al., 2015] Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi- supervised learning with ladder networks. In NIPS, pages 3546-3554, 2015.","title":"Semisupervised learning with ladder networks","authors":"Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, Tapani Raiko","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Saito et al., 2017] Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada. Asymmetric tri-training for unsupervised domain adaptation. In ICML, pages 2988-2997, 2017.","title":"Asymmetric tri-training for unsupervised domain adaptation","authors":"Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sajjadi et al., 2016] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic trans- formations and perturbations for deep semi-supervised learning. In NIPS, pages 1163-1171, 2016.","title":"Regularization with stochastic transformations and perturbations for deep semi-supervised learning","authors":"Sajjadi","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Salimans et al., 2016] Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In NIPS, pages 2226-2234, 2016.","title":"Improved techniques for training GANs","authors":"Tim Salimans, Ian J Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shelhamer et al., 2017] Evan Shelhamer, Jonathan Long, and Trevor Darrell. Fully convolutional networks for se- mantic segmentation. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 39(4):640-651, 2017.","title":"Fully convolutional networks for semantic segmentation","authors":"Evan Shelhamer, Jonathan Long, Trevor Darrell","year":"2017","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey E. Hin- ton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1958, 2014.","title":"Dropout: A simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Szegedy et al., 2016] Christian Szegedy, Vincent Van- houcke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, pages 2818-2826, 2016.","title":"Rethinking the inception architecture for computer vision","authors":"Szegedy","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tarvainen and Valpola, 2017] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- averaged consistency targets improve semi-supervised deep learning results. In NIPS, pages 1195-1204, 2017.","title":"Mean teachers are better role models: Weightaveraged consistency targets improve semi-supervised deep learning results","authors":"Valpola Tarvainen, Antti Tarvainen, Harri Valpola","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang and Zhou, 2010] Wei Wang and Zhi-Hua Zhou. A new analysis of co-training. In ICML, pages 1135-1142, 2010.","title":"A new analysis of co-training","authors":"Wei Wang, Zhi-Hua Zhou","year":"2010","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang and Zhou, 2017] Wei Wang and Zhi-Hua Zhou. The- oretical foundation of co-training and disagreement-based algorithms. CoRR, abs/1708.04403, 2017.","title":"Theoretical foundation of co-training and disagreement-based algorithms","authors":"Wei Wang, Zhi-Hua Zhou","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Weston et al., 2012] Jason Weston, Frédéric Ratle, Hossein Mobahi, and Ronan Collobert. Deep learning via semi- supervised embedding. In Neural Networks: Tricks of the Trade -Second Edition, pages 639-655. 2012.","title":"Deep learning via semisupervised embedding","authors":"Jason Weston, Frédéric Ratle, Hossein Mobahi, Ronan Collobert","year":"2012","venue":"Neural Networks: Tricks of the Trade -Second Edition","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang and Zhou, 2011] Min-Ling Zhang and Zhi-Hua Zhou. CoTrade: Confident co-training with data editing. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 41(6):1612-1626, 2011.","title":"CoTrade: Confident co-training with data editing","authors":"Zhou ; ] Min-Ling Zhang, Zhi-Hua Zhang, Zhou","year":"2011","venue":"IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhou and Li, 2005a] Zhi-Hua Zhou and Ming Li. Semi- supervised regression with co-training. In IJCAI, pages 908-916, 2005.","title":"Semisupervised regression with co-training","authors":"Li ; A] Zhi-Hua Zhou, Ming Zhou, Li","year":"2005","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhou and Li, 2005b] Zhi-Hua Zhou and Ming Li. Tri- training: Exploiting unlabeled data using three classifiers. IEEE Transactions on Knowledge and Data Engineering, 17(11):1529-1541, 2005.","title":"Tritraining: Exploiting unlabeled data using three classifiers","authors":"Li ; B] Zhi-Hua Zhou, Ming Zhou, Li","year":"2005","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhou and Li, 2010] Zhi-Hua Zhou and Ming Li. Semi- supervised learning by disagreement. Knowledge and In- formation Systems, 24(3):415-439, 2010.","title":"Semisupervised learning by disagreement","authors":"Li ; ] Zhi-Hua Zhou, Ming Zhou, Li","year":"2010","venue":"Knowledge and Information Systems","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhou, 2012] Zhi-Hua Zhou. Ensemble Methods: Founda- tions and Algorithms. Chapman & Hall/CRC, 2012.","title":"Ensemble Methods: Foundations and Algorithms","authors":"Zhi-Hua Zhou","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhu, 2007] Xiaojin Zhu. Semi-supervised learning lit- erature survey. Technical Report 1530, University of Wisconsin-Madison, 2007.","title":"Semi-supervised learning literature survey","authors":"Xiaojin Zhu","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}