{"type":"ScholarlyArticle","title":"Rethinking Centered Kernel Alignment in Knowledge Distillation","authors":[{"name":"Zikai Zhou","slug":"zikai-zhou","given":"Zikai","family":"Zhou","affiliation":"School of Computer Science and Technology , East China Normal University , Shanghai , China"},{"name":"Yunhang Shen","slug":"yunhang-shen","given":"Yunhang","family":"Shen","affiliation":"Youtu Lab , Tencent , Shanghai , China"},{"name":"Shitong Shao","slug":"shitong-shao","given":"Shitong","family":"Shao","affiliation":"School of Computer Science and Technology , East China Normal University , Shanghai , China"},{"name":"Linrui Gong","slug":"linrui-gong","given":"Linrui","family":"Gong","affiliation":"Hunan University , China"},{"name":"Shaohui Lin","slug":"shaohui-lin","given":"Shaohui","family":"Lin","affiliation":"School of Computer Science and Technology , East China Normal University , Shanghai , China; Key Laboratory of Advanced Theory and Application in Statistics and Data Science , MOE , China"}],"abstract":"Knowledge distillation has emerged as a highly effective method for bridging the representation discrepancy between large-scale models and lightweight models. Prevalent approaches involve leveraging appropriate metrics to minimize the divergence or distance between the knowledge extracted from the teacher model and the knowledge learned by the student model. Centered Kernel Alignment (CKA) is widely used to measure representation similarity and has been applied in several knowledge distillation methods. However, these methods are complex and fail to uncover the essence of CKA, thus not answering the question of how to use CKA to achieve simple and effective distillation properly. This paper first provides a theoretical perspective to illustrate the effectiveness of CKA, which decouples CKA to the upper bound of Maximum Mean Discrepancy (MMD) and a constant term. Drawing from this, we propose a novel Relation-Centered Kernel Alignment (RCKA) framework, which practically establishes a connection between CKA and MMD. Furthermore, we dynamically customize the application of CKA based on the characteristics of each task, with less computational source yet comparable performance than the previous methods. The extensive experiments on the CIFAR-100, ImageNet-1k, and MS-COCO demonstrate that our method achieves state-of-the-art performance on almost all teacher-student pairs for image classification and object detection, validating the effectiveness of our approaches. Our code is available in https://github.com/Klayand/PCKA.","doi":"10.24963/ijcai.2024/628","url":"https://ijcai.org/proceedings/2024/628","pdf":"https://ijcai.org/proceedings/2024/0628.pdf","markdown":"https://ijcai.org/proceedings/2024/628.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/628","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":628,"pages":{"first":5680,"last":5688},"keywords":["Machine Learning: ML: Representation learning","Machine Learning: ML: Deep learning architectures","Computer Vision: CV: Representation learning","Computer Vision: CV: Recognition (object detection, categorization)"],"videos":[],"references":[{"position":1,"raw":"[Ahn et al., 2019] Sungsoo Ahn, Shell Xu Hu, et al. Varia- tional information distillation for knowledge transfer. In CVPR, pages 9163-9171, Long Beach, CA, USA, Jun. 2019. IEEE.","title":"Variational information distillation for knowledge transfer","authors":"Ahn","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Boudiaf et al., 2020] Malik Boudiaf, Jérôme Rony, Im- tiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and Ismail Ben Ayed. A unifying mutual in- formation view of metric learning: Cross-entropy vs. pair- wise losses. In ECCV, 2020.","title":"A unifying mutual information view of metric learning: Cross-entropy vs. pairwise losses","authors":"Malik Boudiaf, Jérôme Rony, Masud Imtiaz, Eric Ziko, Marco Granger, Pablo Pedersoli, Ismail Piantanida, Ayed Ben","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cai and Vasconcelos, 2017] Zhaowei Cai and Nuno Vas- concelos. Cascade r-cnn: Delving into high quality object detection. CVPR, pages 6154-6162, 2017.","title":"Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection","authors":"Vasconcelos Cai","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2021] Pengguang Chen, Shu Liu, et al. Dis- tilling knowledge via knowledge review. In CVPR, pages 5008-5017, Virtual Event, Jun. 2021. IEEE.","title":"Distilling knowledge via knowledge review","authors":"Chen","year":"2021","venue":"Virtual Event","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2023] Huanran Chen, Yinpeng Dong, Zhengyi Wang, Xiao Yang, Chengqi Duan, Hang Su, and Jun Zhu. Robust classification via a single diffusion model. arXiv preprint arXiv:2305.15241, 2023.","title":"Robust classification via a single diffusion model","authors":"Huanran Chen, Yinpeng Dong, Zhengyi Wang, Xiao Yang, Chengqi Duan, Hang Su, Jun Zhu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chong et al., 2022] Zhiyu Chong, Xinzhu Ma, Hong Zhang, Yuxin Yue, Haojie Li, Zhihui Wang, and Wanli Ouyang. Monodistill: Learning spatial features for monocular 3d object detection. ArXiv, abs/2201.10830, 2022.","title":"Monodistill: Learning spatial features for monocular 3d object detection","authors":"Zhiyu Chong, Xinzhu Ma, Hong Zhang, Yuxin Yue, Haojie Li, Zhihui Wang, Wanli Ouyang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cui et al., 2023] Jiequan Cui, Zhuotao Tian, Zhisheng Zhong, Xiaojuan Qi, Bei Yu, and Hanwang Zhang. Decoupled kullback-leibler divergence loss. ArXiv, abs/2305.13948, 2023.","title":"Decoupled kullback-leibler divergence loss","authors":"Cui","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dai et al., 2021] Xing Dai, Zeren Jiang, et al. General in- stance distillation for object detection. In CVPR, pages 7842-7851, Virtual Event, Jun. 2021.","title":"General instance distillation for object detection","authors":"Xing Dai, Zeren Jiang","year":"2021","venue":"Virtual Event","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[de Rijk et al., 2022] Philip de Rijk, Lukas Schneider, Mar- ius Cordts, and Dariu M. Gavrila. Structural knowledge distillation for object detection. ArXiv, abs/2211.13133, 2022.","title":"Structural knowledge distillation for object detection","authors":"De Rijk","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Frankle and Carbin, 2019] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, New Orleans, LA, USA, May 2019.","title":"The lottery ticket hypothesis: Finding sparse, trainable neural networks","authors":"Carbin Frankle, Jonathan Frankle, Michael Carbin","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gong et al., 2023] Linrui Gong, Shaohui Lin, Baochang Zhang, Yunhang Shen, Ke Li, Ruizhi Qiao, Bohan Ren, Muqing Li, Zhou Yu, and Lizhuang Ma. Adaptive hierarchy-branch fusion for online knowledge distillation. In AAAI, 2023.","title":"Adaptive hierarchy-branch fusion for online knowledge distillation","authors":"Gong","year":"2023","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Heo et al., 2019] Byeongho Heo, Jeesoo Kim, and Sangdoo et al. A comprehensive overhaul of feature distillation. In ICCV, pages 1921-1930, Seoul, Korea (South), Oct.-Nov. 2019. IEEE.","title":"A comprehensive overhaul of feature distillation","authors":"Heo","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2(7), 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Huang et al., 2022] Tao Huang, Shan You, et al. Knowl- edge distillation from a stronger teacher. In NeurIPS, New Orleans, LA, USA, Nov.-Dec. 2022. NIPS.","title":"Knowledge distillation from a stronger teacher","authors":"Tao Huang, Shan You","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kornblith et al., 2019] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. Simi- larity of neural network representations revisited. ArXiv, abs/1905.00414, 2019.","title":"Similarity of neural network representations revisited","authors":"Simon Kornblith, Mohammad Norouzi, Honglak Lee, Geoffrey E Hinton","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof- frey Hinton. Learning multiple layers of features from tiny images. Handbook of Sys temic Autoimmune Diseases, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":"Handbook of Sys temic Autoimmune Diseases","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2020] Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection. ArXiv, abs/2006.04388, 2020.","title":"Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection","authors":"Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, Jian Yang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, et al. Mi- crosoft coco: Common objects in context. In ECCV, pages 740-755, Zurich, Switzerland, Sept. 2014. Springer.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lin et al., 2017] Tsung-Yi Lin, Priya Goyal, Ross B. Gir- shick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. ICCV, pages 2999-3007, 2017.","title":"Focal loss for dense object detection","authors":"Tsung-Yi Lin, Priya Goyal, Ross B Girshick, Kaiming He, Piotr Dollár","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu et al., 2021a] Li Liu, Qingle Huang, Sihao Lin, Hong- wei Xie, Bing Wang, Xiaojun Chang, and Xiao-Xue Liang. Exploring inter-channel correlation for diversity- preserved knowledge distillation. ICCV, pages 8251- 8260, 2021.","title":"Exploring inter-channel correlation for diversitypreserved knowledge distillation","authors":"Liu","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2021b] Songhua Liu, Tianwei Lin, Dongliang He, Fu Li, Ruifeng Deng, Xin Li, Errui Ding, and Hao Wang. Paint transformer: Feed forward neural painting with stroke prediction. In ICCV, pages 6598-6607, Octo- ber 2021.","title":"Paint transformer: Feed forward neural painting with stroke prediction","authors":"Liu","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Marcel and Rodriguez, 2010] Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In ACM MM, pages 1485-1488, Firenze, Italy, Oct. 2010. ACM.","title":"Torchvision the machine-vision package of torch","authors":"Marcel, Rodriguez, Sébastien Marcel, Yann Rodriguez","year":"2010","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Nguyen et al., 2020] Thao Nguyen, Maithra Raghu, and Si- mon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. ArXiv, abs/2010.15327, 2020.","title":"Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth","authors":"Thao Nguyen, Maithra Raghu, Simon Kornblith","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Orhan and Pitkow, 2017] Emin Orhan and Xaq Pitkow. Skip connections eliminate singularities. In ICLR, 2017.","title":"Skip connections eliminate singularities","authors":"Emin Orhan, Xaq Pitkow","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Park et al., 2019] Wonpyo Park, Dongju Kim, et al. Rela- tional knowledge distillation. In CVPR, pages 3967-3976, Long Beach, CA, USA, Jun. 2019. IEEE.","title":"Wonpyo Park, Dongju Kim, et al. Relational knowledge distillation","authors":"Park","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Peng et al., 2019] Baoyun Peng, Xiao Jin, et al. Correlation congruence for knowledge distillation. In ICCV, pages 5007-5016, 2019.","title":"Correlation congruence for knowledge distillation","authors":"Peng","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Qiu et al., 2022] Zengyu Qiu, Xinzhu Ma, Kunlin Yang, Chunya Liu, Jun Hou, Shuai Yi, and Wanli Ouyang. Bet- ter teacher better student: Dynamic prior knowledge for knowledge distillation. ArXiv, abs/2206.06067, 2022.","title":"Better teacher better student: Dynamic prior knowledge for knowledge distillation","authors":"Qiu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross B. Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time ob- ject detection with region proposal networks. IEEE PAMI, 39:1137-1149, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"IEEE PAMI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Romero et al., 2014] Adriana Romero, Nicolas Ballas, et al. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.","title":"Fitnets: Hints for thin deep nets","authors":"Adriana Romero, Nicolas Ballas","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, et al. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Saha et al., 2022] Aninda Saha, Alina Bialkowski, and Sara Khalifa. Distilling representational similarity using cen- tered kernel alignment (cka). In BMVC, 2022.","title":"Distilling representational similarity using centered kernel alignment (cka)","authors":"Aninda Saha, Alina Bialkowski, Sara Khalifa","year":"2022","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Shao et al., 2023a] Shitong Shao, Huanran Chen, Zhen Huang, Linrui Gong, Shuai Wang, and Xinxiao Wu. Teaching what you should teach: a data-based distillation method. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 1351- 1359, 2023.","title":"Teaching what you should teach: a data-based distillation method","authors":"Shao","year":"2023","venue":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2023/150"},{"position":33,"raw":"[Shao et al., 2023b] Shitong Shao, Xu Dai, Shouyi Yin, Lu- jun Li, Huanran Chen, and Yang Hu. Catch-up distilla- tion: You only need to train once for accelerating sam- pling. arXiv preprint arXiv:2305.10769, 2023.","title":"Catch-up distillation: You only need to train once for accelerating sampling","authors":"Shao","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Shu et al., 2021] Changyong Shu, Yifan Liu, et al. Channel- wise knowledge distillation for dense prediction. In ICCV, pages 5311-5320, Montreal, Canada, Oct. 2021. IEEE.","title":"Channelwise knowledge distillation for dense prediction","authors":"Changyong Shu, Yifan Liu","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Tian et al., 2019] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object de- tection. ICCV, pages 9626-9635, 2019.","title":"Fcos: Fully convolutional one-stage object detection","authors":"Chunhua Zhi Tian, Hao Shen, Tong Chen, He","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Tian et al., 2020] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, Addis Ababa, Ethiopia, Apr. 2020. OpenRe- view.net.","title":"Contrastive representation distillation","authors":"Yonglong Tian, Dilip Krishnan, Phillip Isola","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Tung and Mori, 2019] Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. In ICCV, pages 1365-1374, Seoul, Korea (South), Oct-Nov. 2019. IEEE.","title":"Similarity-preserving knowledge distillation","authors":"Frederick Tung, Greg Mori","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Wan et al., 2020] Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, Péter Vajda, and Joseph Gonzalez. Fbnetv2: Differentiable neural archi- tecture search for spatial and channel dimensions. CVPR, pages 12962-12971, 2020.","title":"Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions","authors":"Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, Péter Vajda, Joseph Gonzalez","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Wang et al., 2019] Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine- grained feature imitation. CVPR, pages 4928-4937, 2019.","title":"Distilling object detectors with finegrained feature imitation","authors":"Tao Wang, Li Yuan, Xiaopeng Zhang, Jiashi Feng","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Wu et al., 2016] Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng. Quantized convolutional neural networks for mobile devices. In CVPR, Las Vegas, NV, USA, Jun. 2016. IEEE.","title":"Quantized convolutional neural networks for mobile devices","authors":"Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, Jian Cheng","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Yang et al., 2021] Jing Yang, Brais Martinez, et al. Knowl- edge distillation via softmax regression representation learning. In ICLR. OpenReview.net, 2021.","title":"Knowledge distillation via softmax regression representation learning","authors":"Yang","year":"2021","venue":"ICLR. OpenReview.net","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Yang et al., 2022a] Zhendong Yang, Zhe Li, et al. Focal and global knowledge distillation for detectors. In CVPR, pages 4643-4652, New Orleans, LA, USA, Jun. 2022. IEEE.","title":"Focal and global knowledge distillation for detectors","authors":"Yang","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Yang et al., 2022b] Zhendong Yang, Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan. Masked gen- erative distillation. In ECCV, 2022.","title":"Masked generative distillation","authors":"Yang","year":"2022","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Yang et al., 2023] Zhendong Yang, Ailing Zeng, Zhe Li, Tianke Zhang, Chun Yuan, and Yu Li. From knowledge distillation to self-knowledge distillation: A unified ap- proach with normalized loss and customized soft labels. ArXiv, abs/2303.13005, 2023.","title":"From knowledge distillation to self-knowledge distillation: A unified approach with normalized loss and customized soft labels","authors":"Zhendong Yang, Ailing Zeng, Zhe Li, Tianke Zhang, Chun Yuan, Yu Li","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Zagoruyko and Komodakis, 2016] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Im- proving the performance of convolutional neural networks via attention transfer. In ICLR, 2016.","title":"Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer","authors":"Komodakis Zagoruyko, Sergey Zagoruyko, Nikos Komodakis","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Zhang and Ma, 2021] Linfeng Zhang and Kaisheng Ma. Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors. In ICLR, pages 1-14. OpenReview.net, 2021.","title":"Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors","authors":"Ma ; Zhang, Linfeng Zhang, Kaisheng Ma","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Zhao et al., 2022] Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distil- lation. In CVPR, pages 11953-11962, New Orleans, LA, USA, Jun. 2022. IEEE.","title":"Decoupled knowledge distillation","authors":"Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, Jiajun Liang","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Zhixing et al., 2021] Du Zhixing, Rui Zhang, Ming Chang, Shaoli Liu, Tianshi Chen, Yunji Chen, et al. Distilling object detectors with feature richness. NeurIPS, 34:5213- 5224, Dec. 2021.","title":"Distilling object detectors with feature richness","authors":"Du Zhixing, Rui Zhang, Ming Chang, Shaoli Liu, Tianshi Chen, Yunji Chen","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}