{"type":"ScholarlyArticle","title":"Interpolation Consistency Training for Semi-supervised Learning","authors":[{"name":"Vikas Verma","slug":"vikas-verma","given":"Vikas","family":"Verma","affiliation":"Aalto University , Finland; Montreal Institute for Learning Algorithms (MILA)"},{"name":"Alex Lamb","slug":"alex-lamb","given":"Alex","family":"Lamb","affiliation":"Montreal Institute for Learning Algorithms (MILA)"},{"name":"Juho Kannala","slug":"juho-kannala","given":"Juho","family":"Kannala","affiliation":"Aalto University , Finland"},{"name":"Yoshua Bengio","slug":"yoshua-bengio","given":"Yoshua","family":"Bengio","affiliation":"Montreal Institute for Learning Algorithms (MILA)"},{"name":"David Lopez-Paz","slug":"david-lopez-paz","given":"David","family":"Lopez-Paz","affiliation":"Facebook Artificial Intelligence Research (FAIR)"}],"abstract":"We introduce Interpolation Consistency Training (ICT), a simple and computation efficient algorithm for  training  Deep  Neural  Networks  in  the  semi-supervised learning paradigm. ICT encourages the prediction at an interpolation of unlabeled points to be consistent with the interpolation of the predictions at those points. In classification problems, ICT moves the decision boundary to low-density regions of the data distribution. Our experiments show that ICT achieves state-of-the-art performance when applied to standard neural network architectures on the CIFAR-10 and SVHN benchmark dataset.","doi":"10.24963/ijcai.2019/504","url":"https://ijcai.org/proceedings/2019/504","pdf":"https://ijcai.org/proceedings/2019/0504.pdf","markdown":"https://ijcai.org/proceedings/2019/504.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/504","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":504,"pages":{"first":3635,"last":3641},"keywords":["Machine Learning: Deep Learning","Machine Learning: Semi-Supervised Learning"],"videos":[],"references":[{"position":1,"raw":"[Athiwaratkun et al., 2019] Ben Athiwaratkun, Marc Finzi, Pavel Izmailov, and Andrew Gordon Wilson. There are many consistent explanations of unlabeled data: Why you should average. In International Conference on Learning Representations, 2019.","title":"There are many consistent explanations of unlabeled data: Why you should average","authors":"Ben Athiwaratkun, Marc Finzi, Pavel Izmailov, Andrew Gordon, Wilson","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Berthelot et al., 2019] David Berthelot, Colin Raffel, Aurko Roy, and Ian Goodfellow. Understanding and improving interpolation in autoencoders via an adversarial regularizer. In International Conference on Learning Representations, 2019.","title":"Understanding and improving interpolation in autoencoders via an adversarial regularizer","authors":"David Berthelot, Colin Raffel, Aurko Roy, Ian Goodfellow","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chapelle et al., 2010] Olivier Chapelle, Bernhard Schlkopf, and Alexander Zien. Semi-Supervised Learning. The MIT Press, 1st edition, 2010.","title":"Semi-Supervised Learning","authors":"Chapelle","year":"2010","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Clanuwat et al., 2018] Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:1812.01718, 2018.","title":"Deep learning for classical japanese literature","authors":"Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, David Ha","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dumoulin et al., 2016] Vincent Dumoulin, Ishmael Belg- hazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, and Aaron Courville. Adversarially learned in- ference. arXiv preprint arXiv:1606.00704, 2016.","title":"Adversarially learned inference","authors":"Ishmael Vincent Dumoulin, Ben Belghazi, Olivier Poole, Alex Mastropietro, Martin Lamb, Aaron Arjovsky, Courville","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sher- jil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information pro- cessing systems, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Grandvalet and Bengio, 2005] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In L. K. Saul, Y. Weiss, and L. Bottou, editors, Advances in Neural Information Processing Systems 17, pages 529-536.","title":"Semi-supervised learning by entropy minimization","authors":"Bengio Grandvalet, Yves Grandvalet, Yoshua Bengio","year":"2005","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Laine and Aila, 2016] Samuli Laine and Timo Aila. Tem- poral ensembling for semi-supervised learning. CoRR, abs/1610.02242, 2016.","title":"Temporal ensembling for semi-supervised learning","authors":"Laine, Aila ;, Samuli Laine, Timo Aila","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lecouat et al., 2018] Bruno Lecouat, Chuan-Sheng Foo, Houssam Zenati, and Vijay Chandrasekhar. Manifold reg- ularization with gans for semi-supervised learning. arXiv preprint arXiv:1807.04307, 2018.","title":"Manifold regularization with gans for semi-supervised learning","authors":"Bruno Lecouat, Chuan-Sheng Foo, Houssam Zenati, Vijay Chandrasekhar","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[LeCun et al., 2015] Yann LeCun, Yoshua Bengio, and Geof- frey Hinton. Deep learning. nature, 521(7553):436, 2015.","title":"Deep learning","authors":"Yann Lecun, Yoshua Bengio, Geoffrey Hinton","year":"2015","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Loshchilov and Hutter, 2016] Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with restarts. CoRR, abs/1608.03983, 2016.","title":"Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with restarts","authors":"Hutter Loshchilov","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Luo et al., 2018] Yucen Luo, Jun Zhu, Mengxi Li, Yong Ren, and Bo Zhang. Smooth neighbors on teacher graphs for semi-supervised learning. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 8896-8905, 2018.","title":"Smooth neighbors on teacher graphs for semi-supervised learning","authors":"Luo","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Miyato et al., 2018] Takeru Miyato, Shin ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi- supervised learning. IEEE transactions on pattern analysis and machine intelligence, 2018. [Nakkiran, 2019] Preetum Nakkiran. Adversarial robust- ness may be at odds with simplicity. arXiv preprint arXiv:1901.00532, 2019.","title":"Virtual adversarial training: a regularization method for supervised and semisupervised learning","authors":"Miyato","year":"2018","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Oliver et al., 2018] A. Oliver, A. Odena, C. Raffel, E. D. Cubuk, and I. J. Goodfellow. Realistic Evaluation of Deep Semi-Supervised Learning Algorithms. In Neural Informa- tion Processing Systems (NIPS), 2018.","title":"Realistic Evaluation of Deep Semi-Supervised Learning Algorithms","authors":"A Oliver, A Odena, C Raffel, E D Cubuk, I J Goodfellow","year":"2018","venue":"Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Park et al., 2018] Sungrae Park, JunKeon Park, Su-Jin Shin, and Il-Chul Moon. Adversarial dropout for supervised and semi-supervised learning. AAAI, 2018.","title":"Il-Chul Moon. Adversarial dropout for supervised and semi-supervised learning","authors":"Park","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Radford et al., 2015] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.","title":"Unsupervised representation learning with deep convolutional generative adversarial networks","authors":"Radford","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sajjadi et al., 2016] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transforma- tions and perturbations for deep semi-supervised learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, pages 1171-1179, USA, 2016.","title":"Regularization with stochastic transformations and perturbations for deep semi-supervised learning","authors":"Mehdi Sajjadi, Mehran Javanmardi, Tolga Tasdizen","year":"2016","venue":"Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Curran Associates Inc. [Shawe-Taylor et al., 1996] John Shawe-Taylor, Peter Bartlett, Robert C. Williamson, and Martin Anthony. A framework for structural risk minimisation. pages 68-76, 01 1996.","title":"A framework for structural risk minimisation","authors":"Curran Associates, Inc [ Shawe-Taylor","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tarvainen and Valpola, 2017] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Pro- cessing Systems 30, pages 1195-1204, 2017.","title":"Mean teachers are better role models: Weightaveraged consistency targets improve semi-supervised deep learning results","authors":"Valpola Tarvainen, Antti Tarvainen, Harri Valpola","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tokozume et al., 2018] Yuji Tokozume, Yoshitaka Ushiku, and Tatsuya Harada. Between-class learning for image classification. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.","title":"Between-class learning for image classification","authors":"Yuji Tokozume, Yoshitaka Ushiku, Tatsuya Harada","year":"2018","venue":"The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tsipras et al., 2018] Dimitris Tsipras, Shibani Santurkar, Lo- gan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. stat, 1050:11, 2018.","title":"Robustness may be at odds with accuracy","authors":"Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, Aleksander Madry","year":"2018","venue":"stat","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Verma et al., 2018] Vikas Verma, Alex Lamb, Christo- pher Beckham, Amir Najafi, Ioannis Mitliagkas, Aaron Courville, David Lopez-Paz, and Yoshua Bengio. Manifold Mixup: Better Representations by Interpolating Hidden States. arXiv e-prints, page arXiv:1806.05236, Jun 2018.","title":"Manifold Mixup: Better Representations by Interpolating Hidden States","authors":"Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, Aaron Courville, David Lopez-Paz, Yoshua Bengio","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zagoruyko and Komodakis, 2016] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith, editors, Proceedings of the British Machine Vision Confer- ence (BMVC), pages 87.1-87.12","title":"Wide residual networks","authors":"Sergey Zagoruyko, Nikos Komodakis","year":null,"venue":"Proceedings of the British Machine Vision Conference (BMVC)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2018] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.","title":"mixup: Beyond empirical risk minimization","authors":"Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, David Lopez-Paz","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}