{"type":"ScholarlyArticle","title":"Teaching What You Should Teach: A Data-Based Distillation Method","authors":[{"name":"Shitong Shao","slug":"shitong-shao","given":"Shitong","family":"Shao","affiliation":"Beijing Key Laboratory of Intelligent Information Technology , School of Computer Science and Technology , Beijing Institute of Technology , China"},{"name":"Huanran Chen","slug":"huanran-chen","given":"Huanran","family":"Chen","affiliation":"Beijing Key Laboratory of Intelligent Information Technology , School of Computer Science and Technology , Beijing Institute of Technology , China"},{"name":"Zhen Huang","slug":"zhen-huang","given":"Zhen","family":"Huang","affiliation":"Univeristy of Science and Technology of China , Hefei , China"},{"name":"Linrui Gong","slug":"linrui-gong","given":"Linrui","family":"Gong","affiliation":"Hunan University , Hunan , China"},{"name":"Shuai Wang","slug":"shuai-wang","given":"Shuai","family":"Wang","affiliation":"Tsinghua University , Beijing , China"},{"name":"Xinxiao Wu","slug":"xinxiao-wu","given":"Xinxiao","family":"Wu","affiliation":"Beijing Key Laboratory of Intelligent Information Technology , School of Computer Science and Technology , Beijing Institute of Technology , China"}],"abstract":"In real teaching scenarios, an excellent teacher always teaches what he (or she) is good at but the student is not. This gives the student the best assistance in making up for his (or her) weaknesses and becoming a good one overall. Enlightened by this, we introduce the \"Teaching what you Should Teach\" strategy into a knowledge distillation framework, and propose a data-based distillation method named \"TST\" that searches for desirable augmented samples to assist in distilling more efficiently and rationally. To be specific, we design a neural network-based data augmentation module with priori bias to find out what meets the teacher's strengths but the student's weaknesses, by learning magnitudes and probabilities to generate suitable data samples. By training the data augmentation module and the generalized distillation paradigm alternately, a student model is learned with excellent generalization ability. To verify the effectiveness of our method, we conducted extensive comparative experiments on object recognition, detection, and segmentation tasks. The results on the CIFAR-100, ImageNet-1k, MS-COCO, and Cityscapes datasets demonstrate that our method achieves state-of-the-art performance on almost all teacher-student pairs. Furthermore, we conduct visualization studies to explore what magnitudes and probabilities are needed for the distillation process.","doi":"10.24963/ijcai.2023/150","url":"https://ijcai.org/proceedings/2023/150","pdf":"https://ijcai.org/proceedings/2023/0150.pdf","markdown":"https://ijcai.org/proceedings/2023/150.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/150","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":150,"pages":{"first":1351,"last":1359},"keywords":["Computer Vision: CV: Applications","Computer Vision: CV: Segmentation","Computer Vision: CV: Recognition (object detection, categorization)"],"videos":[],"references":[{"position":1,"raw":"[Ahn et al., 2019] Sungsoo Ahn, Shell Xu Hu, et al. Variational information distillation for knowledge transfer. In Computer Vi- sion and Pattern Recognition, pages 9163-9171, Long Beach, CA, USA, Jun. 2019. IEEE.","title":"Variational information distillation for knowledge transfer","authors":"Ahn","year":"2019","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Carlini and Wagner, 2017] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In Security and Privacy, pages 39-57. IEEE, 2017.","title":"Towards evaluating the robustness of neural networks","authors":"Wagner Carlini, Nicholas Carlini, David Wagner","year":"2017","venue":"Security and Privacy","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2019] Kai Chen, Jiaqi Wang, and Pang et al. MMDe- tection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155, 2019.","title":"MMDetection: Open mmlab detection toolbox and benchmark","authors":"Kai Chen, Jiaqi Wang, Pang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2021] Pengguang Chen, Shu Liu, et al. Distilling knowledge via knowledge review. In Computer Vision and Pat- tern Recognition, pages 5008-5017, Virtual Event, Jun. 2021. IEEE.","title":"Distilling knowledge via knowledge review","authors":"Chen","year":"2021","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2022] Huanran Chen, Shitong Shao, Ziyi Wang, Zirui Shang, Jin Chen, Xiaofeng Ji, and Xinxiao Wu. Bootstrap gener- alization ability from loss landscape perspective. arXiv preprint arXiv:2209.08473, 2022.","title":"Bootstrap generalization ability from loss landscape perspective","authors":"Huanran Chen, Shitong Shao, Ziyi Wang, Zirui Shang, Jin Chen, Xiaofeng Ji, Xinxiao Wu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2023] Huanran Chen, Shitong Shao, Ziyi Wang, Zirui Shang, Jin Chen, Xiaofeng Ji, and Xinxiao Wu. Bootstrap gen- eralization ability from loss landscape perspective. In Leonid Karlinsky, Tomer Michaeli, and Ko Nishino, editors, ECCV 2022 Workshops, Cham, 2023. Springer. [Contributors, 2020] MMSegmentation Contributors. MMSegmen- tation: Openmmlab semantic segmentation toolbox and bench- mark, 2020.","title":"Bootstrap generalization ability from loss landscape perspective","authors":"Huanran Chen, Shitong Shao, Ziyi Wang, Zirui Shang, Jin Chen, Xiaofeng Ji, Xinxiao Wu","year":"2020","venue":"MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Contributors, 2021] MMRazor Contributors. Openmmlab model compression toolbox and benchmark, 2021.","title":"MMRazor Contributors. Openmmlab model compression toolbox and benchmark","authors":null,"year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cordts et al., 2016] Marius Cordts, Mohamed Omran, and Ramos et al. The cityscapes dataset for semantic urban scene under- standing. In Computer Vision and Pattern Recognition, pages 3213-3223, Las Vegas, NV, USA, Jun.-Jul. 2016. IEEE.","title":"The cityscapes dataset for semantic urban scene understanding","authors":"Marius Cordts, Mohamed Omran, Ramos","year":"2016","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cubuk et al., 2019] Ekin D Cubuk, Barret Zoph, et al. Autoaug- ment: Learning augmentation strategies from data. In Computer Vision and Pattern Recognition, pages 113-123, Long Beach, CA, USA, Jun. 2019. IEEE.","title":"Autoaugment: Learning augmentation strategies from data","authors":"Barret Ekin D Cubuk, Zoph","year":"2019","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Cubuk et al., 2020] Ekin D Cubuk, Barret Zoph, et al. Randaug- ment: Practical automated data augmentation with a reduced search space. In Computer Vision and Pattern Recognition work- shops, pages 702-703, Seattle, WA, USA, Jun. 2020. IEEE.","title":"Randaugment: Practical automated data augmentation with a reduced search space","authors":"Barret Ekin D Cubuk, Zoph","year":"2020","venue":"Computer Vision and Pattern Recognition workshops","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Dai et al., 2019] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer- XL: Attentive language models beyond a fixed-length context. In Association for Computational Linguistics, pages 2978-2988, Florence, Italy, Jul. 2019. ACL.","title":"Transformer-XL: Attentive language models beyond a fixed-length context","authors":"Dai","year":"2019","venue":"Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Dai et al., 2021] Xing Dai, Zeren Jiang, et al. General instance distillation for object detection. In Computer Vision and Pattern Recognition, pages 7842-7851, Virtual Event, Jun. 2021.","title":"General instance distillation for object detection","authors":"Xing Dai, Zeren Jiang","year":"2021","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Fang et al., 2022] Gongfan Fang, Kanya Mo, et al. Up to 100x faster data-free knowledge distillation. In Association for the Ad- vancement of Artificial Intelligence, number 6, pages 6597-6604, Virtual Event, Feb.-Mar. 2022.","title":"Up to 100x faster data-free knowledge distillation","authors":"Fang","year":"2022","venue":"Association for the Advancement of Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Frankle and Carbin, 2019] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Represen- tations, New Orleans, LA, USA, May 2019.","title":"The lottery ticket hypothesis: Finding sparse, trainable neural networks","authors":"Carbin Frankle, Jonathan Frankle, Michael Carbin","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, et al. Deep resid- ual learning for image recognition. In Conference on Computer Vision and Pattern Recognition, pages 770-778, Las Vegas, NV, USA, Jun. 2016. IEEE.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang","year":"2016","venue":"Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[He et al., 2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Computer Vision and Pattern Recog- nition, pages 16000-16009, New Orleans, LA, USA, Jun. 2022. IEEE.","title":"Masked autoencoders are scalable vision learners","authors":"Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick","year":"2022","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Heo et al., 2019] Byeongho Heo, Jeesoo Kim, and Sangdoo et al. A comprehensive overhaul of feature distillation. In International Conference on Computer Vision, pages 1921-1930, Seoul, Korea (South), Oct.-Nov. 2019. IEEE.","title":"A comprehensive overhaul of feature distillation","authors":"Heo","year":"2019","venue":"International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2(7), 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Huang et al., 2022] Tao Huang, Shan You, et al. Knowledge distil- lation from a stronger teacher. In Neural Information Processing Systems, New Orleans, LA, USA, Nov.-Dec. 2022. NIPS.","title":"Knowledge distillation from a stronger teacher","authors":"Tao Huang, Shan You","year":"2022","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Jaderberg et al., 2015] Max Jaderberg, Karen Simonyan, et al. Spatial transformer networks. In Neural Information Process- ing Systems, volume 28, Montréal, Canada, Dec. 2015.","title":"Spatial transformer networks","authors":"Jaderberg","year":"2015","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"Curran Associates, Inc. [Krizhevsky and Hinton, 2009]","title":null,"authors":"Curran Associates, Inc","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Handbook of Sys temic Autoimmune Diseases, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":"Handbook of Sys temic Autoimmune Diseases","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Li et al., 2020] Yonggang Li, Guosheng Hu, Yongtao Wang, Tim- othy M. Hospedales, Neil Martin Robertson, and Yongxing Yang. DADA: differentiable automatic data augmentation. In Euro- pean Conference on Computer Vision, Virtual Event, Aug. 2020. Springer.","title":"DADA: differentiable automatic data augmentation","authors":"Yonggang Li, Guosheng Hu, Yongtao Wang, Timothy M Hospedales, Neil Martin Robertson, Yongxing Yang","year":"2020","venue":"European Conference on Computer Vision, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"Lim et al., ] Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. In Neural Information Processing Systems, pages 6662-6672, Jan.","title":"Fast autoaugment","authors":"Lim","year":null,"venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, et al. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pages 740-755, Zurich, Switzerland, Sept. 2014. Springer.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire","year":"2014","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Liu et al., 2019] Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distil- lation for semantic segmentation. In Computer Vision and Pat- tern Recognition, pages 2604-2613, Long Beach, CA, USA, Jun. 2019. IEEE.","title":"Structured knowledge distillation for semantic segmentation","authors":"Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, Jingdong Wang","year":"2019","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Liu et al., 2020] Yifan Liu, Changyong Shu, Jingdong Wang, and Chunhua Shen. Structured knowledge distillation for dense pre- diction. IEEE transactions on Pattern Analysis and Machine In- telligence, 2020.","title":"Structured knowledge distillation for dense prediction","authors":"Yifan Liu, Changyong Shu, Jingdong Wang, Chunhua Shen","year":"2020","venue":"IEEE transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, et al. Swin transformer: Hi- erarchical vision transformer using shifted windows. In Inter- national Conference on Computer Vision, pages 10012-10022, Virtual Event, Mar. 2021. IEEE.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin","year":"2021","venue":"International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Madry et al., 2018] Aleksander Madry, Aleksandar Makelov, Lud- wig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, Vancouver, Canada, Apr.-May 2018. OpenReview.net.","title":"Towards deep learning models resistant to adversarial attacks","authors":"Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, Adrian Vladu","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Marcel and Rodriguez, 2010] Sébastien Marcel and Yann Ro- driguez. Torchvision the machine-vision package of torch. In ACM international conference on Multimedia, pages 1485-1488, Firenze, Italy, Oct. 2010. ACM.","title":"Torchvision the machine-vision package of torch","authors":"Marcel, Rodriguez ; Yann Rodriguez","year":"2010","venue":"ACM international conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learn- ing. arXiv preprint arXiv:1312.5602, 2013.","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Park et al., 2019] Wonpyo Park, Dongju Kim, et al. Relational knowledge distillation. In Conference on Computer Vision and Pattern Recognition, pages 3967-3976, Long Beach, CA, USA, Jun. 2019. IEEE.","title":"Relational knowledge distillation","authors":"Park","year":"2019","venue":"Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Peng et al., 2019] Baoyun Peng, Xiao Jin, et al. Correlation congruence for knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5007-5016, 2019.","title":"Correlation congruence for knowledge distillation","authors":"Peng","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Purwins et al., 2019] Hendrik Purwins, Bo Li, Tuomas Virtanen, Jan Schlüter, Shuo-Yiin Chang, and Tara Sainath. Deep learning for audio signal processing. IEEE Journal of Selected Topics in Signal Processing, 13(2):206-219, 2019.","title":"Deep learning for audio signal processing","authors":"Hendrik Purwins, Bo Li, Tuomas Virtanen, Jan Schlüter, Shuo-Yiin, Tara Chang, Sainath","year":"2019","venue":"IEEE Journal of Selected Topics in Signal Processing","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Romero et al., 2014] Adriana Romero, Nicolas Ballas, et al. Fit- nets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.","title":"Fitnets: Hints for thin deep nets","authors":"Adriana Romero, Nicolas Ballas","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, et al. Ima- genet large scale visual recognition challenge. International jour- nal of computer vision, 115(3):211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng","year":"2015","venue":"International journal of computer vision","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Shu et al., 2021] Changyong Shu, Yifan Liu, et al. Channel-wise knowledge distillation for dense prediction. In International Conference on Computer Vision, pages 5311-5320, Montreal, Canada, Oct. 2021. IEEE.","title":"Channel-wise knowledge distillation for dense prediction","authors":"Changyong Shu, Yifan Liu","year":"2021","venue":"International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Suzuki, 2022] Teppei Suzuki. Teachaugment: Data augmenta- tion optimization using teacher knowledge. In Computer Vision and Pattern Recognition, pages 10904-10914, New Orleans, LA, USA, Jun. 2022. IEEE.","title":"Teachaugment: Data augmentation optimization using teacher knowledge","authors":"Teppei Suzuki","year":"2022","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Tian et al., 2020] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Interna- tional Conference on Learning Representations, Addis Ababa, Ethiopia, Apr. 2020. OpenReview.net.","title":"International Conference on Learning Representations","authors":"Tian","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Tung and Mori, 2019] Frederick Tung and Greg Mori. Similarity- preserving knowledge distillation. In International Conference on Computer Vision, pages 1365-1374, Seoul, Korea (South), Oct-Nov. 2019. IEEE.","title":"Similaritypreserving knowledge distillation","authors":"Frederick Tung, Greg Mori","year":"2019","venue":"International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Wang et al., 2020] Yukang Wang, Wei Zhou, Tao Jiang, Xiang Bai, and Yongchao Xu. Intra-class feature variation distillation for semantic segmentation. In European Conference on Com- puter Vision, pages 346-362, Virtual Event, Aug. 2020. Springer.","title":"Intra-class feature variation distillation for semantic segmentation","authors":"Yukang Wang, Wei Zhou, Tao Jiang, Xiang Bai, Yongchao Xu","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Wieczorek et al., 2021] Mikołaj Wieczorek, Barbara Rychalska, and Jacek Dabrowski. On the unreasonable effectiveness of cen- troids in image retrieval. In International Conference on Neural Information Processing, pages 212-223, Bali, Indonesia, Dec. 2021. Springer.","title":"On the unreasonable effectiveness of centroids in image retrieval","authors":"Mikołaj Wieczorek, Barbara Rychalska, Jacek Dabrowski","year":"2021","venue":"International Conference on Neural Information Processing","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Wu et al., 2016] Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng. Quantized convolutional neural networks for mobile devices. In Computer Vision and Pattern Recognition, Las Vegas, NV, USA, Jun. 2016. IEEE.","title":"Quantized convolutional neural networks for mobile devices","authors":"Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, Jian Cheng","year":"2016","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Yang et al., 2021] Jing Yang, Brais Martinez, et al. Knowledge distillation via softmax regression representation learning. In In- ternational Conference on Learning Representations. OpenRe- view.net, 2021.","title":"Knowledge distillation via softmax regression representation learning","authors":"Jing Yang, Brais Martinez","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Yang et al., 2022] Zhendong Yang, Zhe Li, et al. Focal and global knowledge distillation for detectors. In Computer Vision and Pattern Recognition, pages 4643-4652, New Orleans, LA, USA, Jun. 2022. IEEE.","title":"Focal and global knowledge distillation for detectors","authors":"Zhendong Yang, Zhe Li","year":"2022","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Yin et al., 2020] Hongxu Yin, Pavlo Molchanov, et al. Dream- ing to distill: Data-free knowledge transfer via deepinversion. In Computer Vision and Pattern Recognition, pages 8715-8724, Seattle, WA, USA, Jun. 2020. IEEE. [Zagoruyko and Komodakis, 2016a] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In International Conference on Learning Representa- tions(ICLR), 2016.","title":"Dreaming to distill: Data-free knowledge transfer via deepinversion","authors":"Yin","year":"2016","venue":"International Conference on Learning Representations(ICLR)","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Zagoruyko and Komodakis, 2016b] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference, pages 1-15, York, UK, Sept. 2016. BMVA.","title":"British Machine Vision Conference","authors":"Komodakis Zagoruyko, Sergey Zagoruyko, Nikos Komodakis","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Zhang and Ma, 2021] Linfeng Zhang and Kaisheng Ma. Improve object detection with feature-based knowledge distillation: To- wards accurate and efficient detectors. In International Confer- ence on Learning Representations, Vienna, Austria, May 2021. OpenReview.net, 2021.","title":"Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors","authors":"Ma ; Zhang, Linfeng Zhang, Kaisheng Ma","year":"2021","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Zhao et al., 2017] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing net- work. In Computer Vision and Pattern Recognition, Honolulu, HI, USA, Jul, 2017. IEEE.","title":"Pyramid scene parsing network","authors":"Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, Jiaya Jia","year":"2017","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Zhao et al., 2020] Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring self-attention for image recognition. In Com- puter Vision and Pattern Recognition, pages 10076-10085, Seat- tle, WA, USA, Jun. 2020. IEEE.","title":"Exploring self-attention for image recognition","authors":"Zhao","year":"2020","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Zhao et al., 2022] Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Confer- ence on Computer Vision and Pattern Recognition, pages 11953- 11962, New Orleans, LA, USA, Jun. 2022. IEEE.","title":"Decoupled knowledge distillation","authors":"Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, Jiajun Liang","year":"2022","venue":"Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Zhixing et al., 2021] Du Zhixing, Rui Zhang, Ming Chang, Shaoli Liu, Tianshi Chen, Yunji Chen, et al. Distilling object detectors with feature richness. Neural Information Processing Systems, 34:5213-5224, Dec. 2021.","title":"Distilling object detectors with feature richness","authors":"Du Zhixing, Rui Zhang, Ming Chang, Shaoli Liu, Tianshi Chen, Yunji Chen","year":"2021","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}