{"type":"ScholarlyArticle","title":"Pseudo-spherical Knowledge Distillation","authors":[{"name":"Kyungmin Lee","slug":"kyungmin-lee","given":"Kyungmin","family":"Lee","affiliation":"Agency for Defense Development"},{"name":"Hyeongkeun Lee","slug":"hyeongkeun-lee","given":"Hyeongkeun","family":"Lee","affiliation":"Agency for Defense Development"}],"abstract":"Knowledge distillation aims to transfer the information by minimizing the cross-entropy between the probabilistic outputs of the teacher and student network.\r\nIn this work, we propose an alternative distillation objective by maximizing the scoring rule, which quantitatively measures the agreement of a distribution to the reference distribution. \r\nWe demonstrate that the proper and homogeneous scoring rule exhibits more preferable properties for distillation than the original cross entropy based approach. \r\nTo that end, we present an efficient implementation of the distillation objective based on a pseudo-spherical scoring rule, which is a family of proper and homogeneous scoring rules. We refer to it as pseudo-spherical knowledge distillation. \r\nThrough experiments on various model compression tasks, we validate the effectiveness of our method by showing its superiority over the original knowledge distillation. \r\nMoreover, together with structural distillation methods such as contrastive representation distillation, we achieve state of the art results in CIFAR100 benchmarks.","doi":"10.24963/ijcai.2022/441","url":"https://ijcai.org/proceedings/2022/441","pdf":"https://ijcai.org/proceedings/2022/0441.pdf","markdown":"https://ijcai.org/proceedings/2022/441.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/441","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":441,"pages":{"first":3178,"last":3184},"keywords":["Machine Learning: Probabilistic Machine Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38984960","length_sec":465,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38986817","length_sec":465,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Buciluǎ et al., 2006] Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Pro- ceedings of the 12th ACM SIGKDD international confer- ence on Knowledge discovery and data mining, pages 535- 541, 2006.","title":"Model compression","authors":"Cristian Buciluǎ, Rich Caruana, Alexandru Niculescu-Mizil","year":"2006","venue":"Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2021] Liqun Chen, Dong Wang, Zhe Gan, Jingjing Liu, Ricardo Henao, and Lawrence Carin. Wasser- stein contrastive representation distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 16296-16305, 2021.","title":"Wasserstein contrastive representation distillation","authors":"Liqun Chen, Dong Wang, Zhe Gan, Jingjing Liu, Ricardo Henao, Lawrence Carin","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dawid, 1998] A Philip Dawid. Coherent measures of dis- crepancy, uncertainty and dependence, with applications to bayesian predictive experimental design. Department of Statistical Science, University College London. http://www. ucl. ac. uk/Stats/research/abs94. html, Tech. Rep, 139, 1998.","title":"Coherent measures of discrepancy, uncertainty and dependence, with applications to bayesian predictive experimental design","authors":"Dawid Philip","year":"1998","venue":null,"doi":null,"url":"http://www.ucl.ac.uk/Stats/research/abs94.html,Tech.Rep,139","internal_url":null},{"position":4,"raw":"[Friedman, 1983] Daniel Friedman. Effective scoring rules for probabilistic forecasts. Management Science, 29(4):447- 454, 1983.","title":"Effective scoring rules for probabilistic forecasts","authors":"Daniel Friedman","year":"1983","venue":"Management Science","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fujisawa and Eguchi, 2008] Hironori Fujisawa and Shinto Eguchi. Robust parameter estimation with a small bias against heavy contamination. Journal of Multivariate Anal- ysis, 99(9):2053-2081, 2008.","title":"Robust parameter estimation with a small bias against heavy contamination","authors":"Fujisawa, Eguchi ;, Shinto Eguchi","year":"2008","venue":"Journal of Multivariate Analysis","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gneiting and Raftery, 2007] Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and es- timation. Journal of the American statistical Association, 102(477):359-378, 2007.","title":"Strictly proper scoring rules, prediction, and estimation","authors":"Tilmann Gneiting, Adrian E Raftery","year":"2007","venue":"Journal of the American statistical Association","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kornblith et al., 2019] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519-3529.","title":"Similarity of neural network representations revisited","authors":"Simon Kornblith, Mohammad Norouzi, Honglak Lee, Geoffrey Hinton","year":null,"venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"PMLR, 2019.","title":null,"authors":"Pmlr","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ma et al., 2018] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference on computer vision (ECCV), pages 116-131, 2018.","title":"Shufflenet v2: Practical guidelines for efficient cnn architecture design","authors":"Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, Jian Sun","year":"2018","venue":"Proceedings of the European conference on computer vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Parry et al., 2012] Matthew Parry, A Philip Dawid, and Stef- fen Lauritzen. Proper local scoring rules. The Annals of Statistics, 40(1):561-592, 2012.","title":"Proper local scoring rules","authors":"Matthew Parry, , A Philip Dawid, Steffen Lauritzen","year":"2012","venue":"The Annals of Statistics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Romero et al., 2014] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.","title":"Fitnets: Hints for thin deep nets","authors":"Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Takenouchi and Kanamori, 2017] Takashi Takenouchi and Takafumi Kanamori. Statistical inference with unnor- malized discrete models and localized homogeneous di- vergences. The Journal of Machine Learning Research, 18(1):1804-1829, 2017.","title":"Statistical inference with unnormalized discrete models and localized homogeneous divergences","authors":"Kanamori Takenouchi, Takashi Takenouchi, Takafumi Kanamori","year":"2017","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tian et al., 2019] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv:1910.10699, 2019.","title":"Contrastive representation distillation","authors":"Yonglong Tian, Dilip Krishnan, Phillip Isola","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Xu et al., 2020] Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self- supervision. In European Conference on Computer Vision, pages 588-604. Springer, 2020.","title":"Knowledge distillation meets selfsupervision","authors":"Guodong Xu, Ziwei Liu, Xiaoxiao Li, Chen Change Loy","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Yu et al., 2021] Lantao Yu, Jiaming Song, Yang Song, and Stefano Ermon. Pseudo-spherical contrastive divergence. Advances in Neural Information Processing Systems, 34, 2021.","title":"Pseudo-spherical contrastive divergence","authors":"Lantao Yu, Jiaming Song, Yang Song, Stefano Ermon","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zagoruyko and Komodakis, 2016a] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Im- proving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928, 2016.","title":"Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer","authors":"Komodakis Zagoruyko, Sergey Zagoruyko, Nikos Komodakis","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zagoruyko and Komodakis, 2016b] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.","title":"Wide residual networks","authors":"Komodakis Zagoruyko, Sergey Zagoruyko, Nikos Komodakis","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zeng and Martinez, 2000] Xinchuan Zeng and Tony R. Mar- tinez. Using a neural network to approximate an ensemble of classifiers. Neural Processing Letters, 12(3):225-237, 2000.","title":"Martinez. Using a neural network to approximate an ensemble of classifiers","authors":"Martinez Zeng, Xinchuan Zeng, Tony R","year":"2000","venue":"Neural Processing Letters","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2018] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convo- lutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6848-6856, 2018.","title":"Shufflenet: An extremely efficient convolutional neural network for mobile devices","authors":"Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, Jian Sun","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}