{"type":"ScholarlyArticle","title":"Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillation","authors":[{"name":"Jingxuan Wei","slug":"jingxuan-wei","given":"Jingxuan","family":"Wei","affiliation":"Shenyang Institute of Computing Technology , Chinese Academy of Sciences; University of Chinese Academy of Sciences"},{"name":"Linzhuang Sun","slug":"linzhuang-sun","given":"Linzhuang","family":"Sun","affiliation":"Shenyang Institute of Computing Technology , Chinese Academy of Sciences; University of Chinese Academy of Sciences"},{"name":"Yichong Leng","slug":"yichong-leng","given":"Yichong","family":"Leng","affiliation":"University of Science and Technology of China"},{"name":"Xu Tan","slug":"xu-tan","given":"Xu","family":"Tan","affiliation":"Independent Researcher"},{"name":"Bihui Yu","slug":"bihui-yu","given":"Bihui","family":"Yu","affiliation":"Shenyang Institute of Computing Technology , Chinese Academy of Sciences; University of Chinese Academy of Sciences"},{"name":"Ruifeng Guo","slug":"ruifeng-guo","given":"Ruifeng","family":"Guo","affiliation":"Shenyang Institute of Computing Technology , Chinese Academy of Sciences; University of Chinese Academy of Sciences"}],"abstract":"Knowledge distillation, transferring knowledge from a teacher model to a student model, has emerged as a powerful technique in neural machine translation for compressing models or simplifying training targets. Knowledge distillation encompasses two primary methods: sentence-level distillation and token-level distillation. In sentence-level distillation, the student model is trained to align with the output of the teacher model, which can alleviate the training difficulty and give student model a comprehensive understanding of global structure. Differently, token-level distillation requires the student model to learn the output distribution of the teacher model, facilitating a more fine-grained transfer of knowledge. Studies have revealed divergent performances between sentence-level and token-level distillation across different scenarios, leading to the confusion on the empirical selection of knowledge distillation methods. In this study, we argue that token-level distillation, with its more complex objective (i.e., distribution), is better suited for “simple” scenarios, while sentence-level distillation excels in “complex” scenarios. To substantiate our hypothesis, we systematically analyze the performance of distillation methods by varying the model size of student models, the complexity of text, and the difficulty of decoding procedure. While our experimental results validate our hypothesis, defining the complexity level of a given scenario remains a challenging task. So we further introduce a novel hybrid method that combines token-level and sentence-level distillation through a gating mechanism, aiming to leverage the advantages of both individual methods. Experiments demonstrate that the hybrid method surpasses the performance of token-level or sentence-level distillation methods and the previous works by a margin, demonstrating the effectiveness of the proposed hybrid method.","doi":"10.24963/ijcai.2024/722","url":"https://ijcai.org/proceedings/2024/722","pdf":"https://ijcai.org/proceedings/2024/0722.pdf","markdown":"https://ijcai.org/proceedings/2024/722.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/722","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Natural Language Processing","track":"Main Track","number":722,"pages":{"first":6531,"last":6540},"keywords":["Natural Language Processing: NLP: Machine translation and multilinguality","Natural Language Processing: NLP: Interpretability and analysis of models for NLP","Natural Language Processing: NLP: Other","Natural Language Processing: NLP: Summarization"],"videos":[],"references":[{"position":1,"raw":"[Baskar et al., 2019] Murali Karthick Baskar, Lukáš Bur- get, Shinji Watanabe, Martin Karafiát, Takaaki Hori, and Jan Honza Černockỳ. Promising accurate prefix boosting for sequence-to-sequence asr. In ICASSP, pages 5646-5650. IEEE, 2019.","title":"Promising accurate prefix boosting for sequence-to-sequence asr","authors":"Murali Karthick Baskar, Lukáš Burget, Shinji Watanabe, Martin Karafiát, Takaaki Hori, Jan Honza Černockỳ","year":"2019","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Blain et al., 2023] Frédéric Blain, Chrysoula Zerva, Ricardo Ribeiro, Nuno M Guerreiro, Diptesh Kanojia, José GC de Souza, Beatriz Silva, Tânia Vaz, Yan Jingxuan, Fatemeh Azadi, et al. Findings of the wmt 2023 shared task on quality estimation. In Proceedings of the Eighth Conference on Machine Translation, pages 629-653, 2023.","title":"Findings of the wmt 2023 shared task on quality estimation","authors":"Frédéric Blain, Chrysoula Zerva, Ricardo Ribeiro, M Nuno, Diptesh Guerreiro, José Gc De Kanojia, Beatriz Souza, Tânia Silva, Yan Vaz, Fatemeh Jingxuan, Azadi","year":"2023","venue":"Proceedings of the Eighth Conference on Machine Translation","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2020] Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu, and Jingjing Liu. Distilling knowledge learned in bert for text generation. In ACL, pages 7893- 7905, 2020.","title":"Distilling knowledge learned in bert for text generation","authors":"Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu, Jingjing Liu","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Deng et al., 2023] Hexuan Deng, Liang Ding, Xuebo Liu, Meishan Zhang, Dacheng Tao, and Min Zhang. Improving simultaneous machine translation with monolingual data. In AAAI, pages 12728-12736, 2023.","title":"Improving simultaneous machine translation with monolingual data","authors":"Deng","year":"2023","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ding et al., 2020] Liang Ding, Longyue Wang, Xuebo Liu, Derek F Wong, Dacheng Tao, and Zhaopeng Tu. Understanding and improving lexical choice in non- autoregressive translation. In ICLR, 2020.","title":"Understanding and improving lexical choice in nonautoregressive translation","authors":"Liang Ding, Longyue Wang, Xuebo Liu, Derek F Wong, Dacheng Tao, Zhaopeng Tu","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ding et al., 2021] Liang Ding, Longyue Wang, Xuebo Liu, Derek F Wong, Dacheng Tao, and Zhaopeng Tu. Pro- gressive multi-granularity training for non-autoregressive translation. In Findings of the ACL-IJCNLP 2021, pages 2797-2803, 2021.","title":"Progressive multi-granularity training for non-autoregressive translation","authors":"Liang Ding, Longyue Wang, Xuebo Liu, Derek F Wong, Dacheng Tao, Zhaopeng Tu","year":"2021","venue":"Findings of the ACL-IJCNLP 2021","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Edunov et al., 2018] Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back-translation at scale. In EMNLP, pages 489-500, 2018.","title":"Understanding back-translation at scale","authors":"Sergey Edunov, Myle Ott, Michael Auli, David Grangier","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Farinha et al., 2022] Ana C Farinha, M Amin Farajian, Mar- ianna Buchicchio, Patrick Fernandes, José GC De Souza, Helena Moniz, and André FT Martins. Findings of the wmt 2022 shared task on chat translation. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 724-743, 2022.","title":"Findings of the wmt 2022 shared task on chat translation","authors":"M Ana C Farinha, Marianna Amin Farajian, Patrick Buchicchio, José Gc De Fernandes, Helena Souza, André Ft Moniz, Martins","year":"2022","venue":"Proceedings of the Seventh Conference on Machine Translation (WMT)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Freitag et al., 2017] Markus Freitag, Yaser Al-Onaizan, and Baskaran Sankaran. Ensemble distillation for neural ma- chine translation. arXiv preprint arXiv:1702.01802, 2017.","title":"Ensemble distillation for neural machine translation","authors":"Markus Freitag, Yaser Al-Onaizan, Baskaran Sankaran","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gajbhiye et al., 2021] Amit Gajbhiye, Marina Fomicheva, Fernando Alva-Manchego, Frédéric Blain, Abiola Oba- muyide, Nikolaos Aletras, and Lucia Specia. Knowledge distillation for quality estimation. In ACL, pages 5091- 5099, 2021.","title":"Knowledge distillation for quality estimation","authors":"Amit Gajbhiye, Marina Fomicheva, Fernando Alva-Manchego, Frédéric Blain, Abiola Obamuyide, Nikolaos Aletras, Lucia Specia","year":"2021","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gao et al., 2022] Pengzhi Gao, Zhongjun He, Hua Wu, and Haifeng Wang. Bi-simcut: A simple strategy for boosting neural machine translation. In NAACL, pages 3938-3948, 2022.","title":"Bi-simcut: A simple strategy for boosting neural machine translation","authors":"Gao","year":"2022","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ge et al., 2023] Ling Ge, Chunming Hu, Guanghui Ma, Hong Zhang, and Jihong Liu. Prokd: an unsupervised pro- totypical knowledge distillation network for zero-resource cross-lingual named entity recognition. In AAAI, pages 12818-12826, 2023.","title":"Prokd: an unsupervised prototypical knowledge distillation network for zero-resource cross-lingual named entity recognition","authors":"Ling Ge, Chunming Hu, Guanghui Ma, Hong Zhang, Jihong Liu","year":"2023","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gou et al., 2021] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A sur- vey. International Journal of Computer Vision, 129:1789- 1819, 2021.","title":"Knowledge distillation: A survey","authors":"Jianping Gou, Baosheng Yu, Stephen J Maybank, Dacheng Tao","year":"2021","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[He et al., 2021] Haoyu He, Xingjian Shi, Jonas Mueller, Sheng Zha, Mu Li, and George Karypis. Distiller: A systematic study of model distillation methods in natural language processing. In Proceedings of the Second Work- shop on Simple and Efficient Natural Language Processing, pages 119-133, 2021.","title":"Distiller: A systematic study of model distillation methods in natural language processing","authors":"Haoyu He, Xingjian Shi, Jonas Mueller, Sheng Zha, Mu Li, George Karypis","year":"2021","venue":"Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Iyer et al., 2023] Nikhil Iyer, V. Thejas, Nipun Kwatra, Ra- machandran Ramjee, and Muthian Sivathanu. Wide- minima density hypothesis and the explore-exploit learning rate schedule. Journal of Machine Learning Research, 24(65):1-37, 2023.","title":"Wideminima density hypothesis and the explore-exploit learning rate schedule","authors":"Nikhil Iyer, V Thejas, Nipun Kwatra, Ramachandran Ramjee, Muthian Sivathanu","year":"2023","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Jaszkiewicz and Słowiński, 1999] Andrzej Jaszkiewicz and Roman Słowiński. The 'light beam search'approach-an overview of methodology applications. European Journal of Operational Research, 113(2):300-314, 1999.","title":"Andrzej Jaszkiewicz and Roman Słowiński. The 'light beam search'approach-an overview of methodology applications","authors":"Słowiński Jaszkiewicz","year":"1999","venue":"European Journal of Operational Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kambhatla et al., 2022] Nishant Kambhatla, Logan Born, and Anoop Sarkar. Cipherdaug: Ciphertext based data augmentation for neural machine translation. In Proceed- ings of the ACL, pages 201-218, 2022.","title":"Cipherdaug: Ciphertext based data augmentation for neural machine translation","authors":"Nishant Kambhatla, Logan Born, Anoop Sarkar","year":"2022","venue":"Proceedings of the ACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kim and Rush, 2016] Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In EMNLP, pages 1317-1327, 2016.","title":"Sequence-level knowledge distillation","authors":"Yoon Kim, Alexander M Rush","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lamb et al., 2016] Alex M Lamb, Anirudh Goyal ALIAS PARTH GOYAL, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. Professor forcing: A new algorithm for training recurrent networks. NeurIPS, 29, 2016.","title":"Professor forcing: A new algorithm for training recurrent networks","authors":"Alex M Lamb, Anirudh Goyal, Alias Parth Goyal, Ying Zhang, Saizheng Zhang, Aaron C Courville, Yoshua Bengio","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lee et al., 2022] Dongkyu Lee, Zhiliang Tian, Yingxiu Zhao, Ka Chun Cheung, and Nevin Zhang. Hard gate knowledge distillation-leverage calibration for robust and reliable lan- guage model. In EMNLP, pages 9793-9803, 2022.","title":"Hard gate knowledge distillation-leverage calibration for robust and reliable language model","authors":"Lee","year":"2022","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lei et al., 2022] Yuanyuan Lei, Ruihong Huang, Lu Wang, and Nick Beauchamp. Sentence-level media bias analysis informed by discourse structures. In EMNLP, pages 10040- 10050, 2022.","title":"Sentence-level media bias analysis informed by discourse structures","authors":"Yuanyuan Lei, Ruihong Huang, Lu Wang, Nick Beauchamp","year":"2022","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Li et al., 2021] Zheng Li, Danqing Zhang, Tianyu Cao, Ying Wei, Yiwei Song, and Bing Yin. Metats: Meta teacher- student network for multilingual sequence labeling with minimal supervision. In EMNLP, pages 3183-3196, 2021.","title":"Metats: Meta teacherstudent network for multilingual sequence labeling with minimal supervision","authors":"Zheng Li, Danqing Zhang, Tianyu Cao, Ying Wei, Yiwei Song, Bing Yin","year":"2021","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Liao et al., 2020] Baohao Liao, Yingbo Gao, and Hermann Ney. Multi-agent mutual learning at sentence-level and token-level for neural machine translation. In EMNLP, pages 1715-1724, 2020.","title":"Multi-agent mutual learning at sentence-level and token-level for neural machine translation","authors":"Liao","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Liu et al., 2020] Yuang Liu, Wei Zhang, and Jun Wang. Adaptive multi-teacher multi-level knowledge distillation. Neurocomputing, 415:106-113, 2020.","title":"Adaptive multi-teacher multi-level knowledge distillation","authors":"Yuang Liu, Wei Zhang, Jun Wang","year":"2020","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Lohrenz et al., 2023] Timo Lohrenz, Björn Möller, Zhengyang Li, and Tim Fingscheidt. Relaxed attention for transformer models. In IJCNN, pages 1-10, 2023.","title":"Relaxed attention for transformer models","authors":"Lohrenz","year":"2023","venue":"IJCNN","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ma et al., 2023] Cong Ma, Yaping Zhang, Mei Tu, Yang Zhao, Yu Zhou, and Chengqing Zong. Multi-teacher knowl- edge distillation for end-to-end text image machine transla- tion. In ICDAR, pages 484-501. Springer, 2023.","title":"Multi-teacher knowledge distillation for end-to-end text image machine translation","authors":"Cong Ma, Yaping Zhang, Mei Tu, Yang Zhao, Yu Zhou, Chengqing Zong","year":"2023","venue":"ICDAR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Mhamdi et al., 2023] Meryem Mhamdi, Jonathan May, Franck Dernoncourt, Trung Bui, and Seunghyun Yoon. Multilingual sentence-level semantic search using meta- distillation learning. arXiv preprint arXiv:2309.08185, 2023.","title":"Meryem Mhamdi","authors":"Mhamdi","year":"2023","venue":"Multilingual sentence-level semantic search using metadistillation learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Mun'im et al., 2019] Raden Mu'az Mun'im, Nakamasa In- oue, and Koichi Shinoda. Sequence-level knowledge distil- lation for model compression of attention-based sequence- to-sequence speech recognition. In ICASSP, pages 6151- 6155, 2019.","title":"Sequence-level knowledge distillation for model compression of attention-based sequenceto-sequence speech recognition","authors":"Mun'im","year":"2019","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Oka et al., 2021] Yui Oka, Katsuhito Sudoh, and Satoshi Nakamura. Using perturbed length-aware positional en- coding for non-autoregressive neural machine translation. arXiv preprint arXiv:2107.13689, 2021.","title":"Using perturbed length-aware positional encoding for non-autoregressive neural machine translation","authors":"Oka","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Ott et al., 2019] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for se- quence modeling. In NAACL, pages 48-53, 2019.","title":"fairseq: A fast, extensible toolkit for sequence modeling","authors":"Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, Michael Auli","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Peng et al., 2023] Keqin Peng, Liang Ding, Qihuang Zhong, Yuanxin Ouyang, Wenge Rong, Zhang Xiong, and Dacheng Tao. Token-level self-evolution training for sequence-to- sequence learning. In Proceedings of the ACL, pages 841- 850, 2023.","title":"Token-level self-evolution training for sequence-tosequence learning","authors":"Keqin Peng, Liang Ding, Qihuang Zhong, Yuanxin Ouyang, Wenge Rong, Zhang Xiong, Dacheng Tao","year":"2023","venue":"Proceedings of the ACL","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Phuong and Lampert, 2019] Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In ICML, pages 5142-5151. PMLR, 2019.","title":"Towards understanding knowledge distillation","authors":"Phuong, Lampert, Mary Phuong, Christoph Lampert","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Ren et al., 2019] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech: Fast, robust and controllable text to speech. NeurIPS, 32, 2019.","title":"Fastspeech: Fast, robust and controllable text to speech","authors":"Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, Tie-Yan Liu","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Shao et al., 2022] Chenze Shao, Xuanfu Wu, and Yang Feng. One reference is not enough: Diverse distillation with reference selection for non-autoregressive translation. In NAACL, pages 3779-3791, 2022.","title":"One reference is not enough: Diverse distillation with reference selection for non-autoregressive translation","authors":"Shao","year":"2022","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Shen et al., 2020] Dinghan Shen, Mingzhi Zheng, Yelong Shen, Yanru Qu, and Weizhu Chen. A simple but tough-to-beat data augmentation approach for natural lan- guage understanding and generation. arXiv preprint arXiv:2009.13818, 2020.","title":"A simple but tough-to-beat data augmentation approach for natural language understanding and generation","authors":"Shen","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Stahlberg, 2020] Felix Stahlberg. Neural machine transla- tion: A review. Journal of Artificial Intelligence Research, 69:343-418, 2020.","title":"Neural machine translation: A review","authors":"Felix Stahlberg, Stahlberg","year":"2020","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Sun et al., 2019] Hao Sun, Xu Tan, Jun-Wei Gan, Hongzhi Liu, Sheng Zhao, Tao Qin, and Tie-Yan Liu. Token-level ensemble distillation for grapheme-to-phoneme conversion. In ISCA, pages 2115-2119, 2019.","title":"Token-level ensemble distillation for grapheme-to-phoneme conversion","authors":"Hao Sun, Xu Tan, Jun-Wei Gan, Hongzhi Liu, Sheng Zhao, Tao Qin, Tie-Yan Liu","year":"2019","venue":"ISCA","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Sun et al., 2020] Haipeng Sun, Rui Wang, Kehai Chen, Masao Utiyama, Eiichiro Sumita, and Tiejun Zhao. Knowl- edge distillation for multilingual unsupervised neural ma- chine translation. In ACL, pages 3525-3535, 2020.","title":"Knowledge distillation for multilingual unsupervised neural machine translation","authors":"Haipeng Sun, Rui Wang, Kehai Chen, Masao Utiyama, Eiichiro Sumita, Tiejun Zhao","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Tan et al., 2019] Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and Tie-Yan Liu. Multilingual neural machine trans- lation with knowledge distillation. In ICLR, 2019.","title":"Multilingual neural machine translation with knowledge distillation","authors":"Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, Tie-Yan Liu","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Tan et al., 2022] Qingyu Tan, Ruidan He, Lidong Bing, and Hwee Tou Ng. Document-level relation extraction with adaptive focal loss and knowledge distillation. In ACL, pages 1672-1681, 2022.","title":"Document-level relation extraction with adaptive focal loss and knowledge distillation","authors":"Qingyu Tan, Ruidan He, Lidong Bing, Hwee Tou Ng","year":"2022","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Tang et al., 2019] Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task- specific knowledge from bert into simple neural networks. arXiv preprint arXiv:1903.12136, 2019.","title":"Distilling taskspecific knowledge from bert into simple neural networks","authors":"Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, Jimmy Lin","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Tang et al., 2021] Zineng Tang, Jaemin Cho, Hao Tan, and Mohit Bansal. Vidlankd: Improving language under- standing via video-distilled knowledge transfer. NeurIPS, 34:24468-24481, 2021.","title":"Vidlankd: Improving language understanding via video-distilled knowledge transfer","authors":"Tang","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Toomarian and Barhen, 1992] Nikzad Benny Toomarian and Jacob Barhen. Learning a trajectory using adjoint functions and teacher forcing. Neural networks, 5(3):473-484, 1992.","title":"Learning a trajectory using adjoint functions and teacher forcing","authors":"Benny Nikzad, Jacob Toomarian, Barhen","year":"1992","venue":"Neural networks","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Wang and Yoon, 2021] Lin Wang and Kuk-Jin Yoon. Knowl- edge distillation and student-teacher learning for visual in- telligence: A review and new outlooks. IEEE transactions on pattern analysis and machine intelligence, 44(6):3048- 3068, 2021.","title":"Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks","authors":"Lin Wang, Kuk-Jin Yoon","year":"2021","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Wang et al., 2020] Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Fei Huang, and Kewei Tu. Structure-level knowl- edge distillation for multilingual sequence labeling. In ACL, pages 3317-3330, 2020.","title":"Structure-level knowledge distillation for multilingual sequence labeling","authors":"Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Fei Huang, Kewei Tu","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Wang et al., 2021] Fusheng Wang, Jianhao Yan, Fandong Meng, and Jie Zhou. Selective knowledge distillation for neural machine translation. In ACL and IJCAI, pages 6456- 6466, 2021.","title":"Selective knowledge distillation for neural machine translation","authors":"Fusheng Wang, Jianhao Yan, Fandong Meng, Jie Zhou","year":"2021","venue":"ACL and IJCAI","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Wu et al., 2021] Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, Tie-Yan Liu, et al. R- drop: Regularized dropout for neural networks. NeurIPS, 34:10890-10905, 2021.","title":"Rdrop: Regularized dropout for neural networks","authors":"Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, Tie-Yan Liu","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Xiao et al., 2023] Yisheng Xiao, Lijun Wu, Junliang Guo, Juntao Li, Min Zhang, Tao Qin, and Tie-yan Liu. A survey on non-autoregressive generation for neural machine trans- lation and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.","title":"A survey on non-autoregressive generation for neural machine translation and beyond","authors":"Yisheng Xiao, Lijun Wu, Junliang Guo, Juntao Li, Min Zhang, Tao Qin, Tie-Yan Liu","year":"2023","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Xu et al., 2021a] Haoran Xu, Benjamin Van Durme, and Kenton W. Murray. Bert, mbert, or bibert? A study on contextualized embeddings for neural machine translation. In EMNLP, pages 6663-6675, 2021.","title":"mbert, or bibert? A study on contextualized embeddings for neural machine translation","authors":"Xu","year":"2021","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Xu et al., 2021b] Weijia Xu, Shuming Ma, Dongdong Zhang, and Marine Carpuat. How does distilled data complexity impact the quality and confidence of non- autoregressive machine translation? In ACL, pages 4392- 4400, 2021.","title":"How does distilled data complexity impact the quality and confidence of nonautoregressive machine translation? In ACL","authors":"Xu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Yang et al., 2022a] Zhixian Yang, Renliang Sun, and Xiao- jun Wan. Nearest neighbor knowledge distillation for neural machine translation. In NAACL, pages 5546-5556, 2022.","title":"Nearest neighbor knowledge distillation for neural machine translation","authors":"Yang","year":"2022","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Yang et al., 2022b] Zhixian Yang, Renliang Sun, and Xiao- jun Wan. Nearest neighbor knowledge distillation for neural machine translation. In NAACL, pages 5546-5556, 2022.","title":"Nearest neighbor knowledge distillation for neural machine translation","authors":"Yang","year":"2022","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":54,"raw":"[Zhang et al., 2019] Biao Zhang, Deyi Xiong, Jinsong Su, and Jiebo Luo. Future-aware knowledge distillation for neural machine translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(12):2278- 2287, 2019.","title":"Future-aware knowledge distillation for neural machine translation","authors":"Biao Zhang, Deyi Xiong, Jinsong Su, Jiebo Luo","year":"2019","venue":"IEEE/ACM Transactions on Audio, Speech, and Language Processing","doi":null,"url":null,"internal_url":null},{"position":55,"raw":"[Zhang et al., 2022] Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling-Yu Duan. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In CVPR, pages 10174-10183, 2022.","title":"Fine-tuning global model via data-free knowledge distillation for non-iid federated learning","authors":"Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, Ling-Yu Duan","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":56,"raw":"[Zhang et al., 2024] Shuoxi Zhang, Hanpeng Liu, and Kun He. Knowledge distillation via token-level relationship graph based on the big data technologies. Big Data Re- search, page 100438, 2024.","title":"Knowledge distillation via token-level relationship graph based on the big data technologies","authors":"Zhang","year":"2024","venue":"Big Data Research","doi":null,"url":null,"internal_url":null},{"position":57,"raw":"[Zhou et al., 2020] Chunting Zhou, Jiatao Gu, and Graham Neubig. Understanding knowledge distillation in non- autoregressive machine translation. In ICLR, 2020.","title":"Understanding knowledge distillation in nonautoregressive machine translation","authors":"Zhou","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}