{"type":"ScholarlyArticle","title":"Prompting to Distill: Boosting Data-Free Knowledge Distillation via Reinforced Prompt","authors":[{"name":"Xinyin Ma","slug":"xinyin-ma","given":"Xinyin","family":"Ma","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Xinchao Wang","slug":"xinchao-wang","given":"Xinchao","family":"Wang","affiliation":"Department of Electrical and Computer Engineering , National University of Singapore"},{"name":"Gongfan Fang","slug":"gongfan-fang","given":"Gongfan","family":"Fang","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Yongliang Shen","slug":"yongliang-shen","given":"Yongliang","family":"Shen","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Weiming Lu","slug":"weiming-lu","given":"Weiming","family":"Lu","affiliation":"College of Computer Science and Technology , Zhejiang University"}],"abstract":"Data-free knowledge distillation (DFKD) conducts knowledge distillation via eliminating the dependence of original training data, and has recently achieved impressive results in accelerating pre-trained language models.  At the heart of DFKD is to reconstruct a synthetic dataset by inverting the parameters of the uncompressed model. Prior DFKD approaches, however, have largely relied on hand-crafted priors of the target data distribution for the reconstruction, which can be inevitably biased and often incompetent to capture the intrinsic distributions. To address this problem, we propose a prompt-based method, termed as PromptDFD, that allows us to take advantage of learned language priors, which effectively harmonizes the synthetic sentences to be semantically and grammatically correct. Specifically, PromptDFD leverages a pre-trained generative model to provide language priors and introduces a reinforced topic prompter to control data synthesis, making the generated samples thematically relevant and semantically plausible, and thus friendly to downstream tasks. As shown in our experiments, the proposed method substantially improves the synthesis quality and achieves considerable improvements on distillation performance. In some cases, PromptDFD even gives rise to results on par with those from the data-driven knowledge distillation with access to the original training data.","doi":"10.24963/ijcai.2022/596","url":"https://ijcai.org/proceedings/2022/596","pdf":"https://ijcai.org/proceedings/2022/0596.pdf","markdown":"https://ijcai.org/proceedings/2022/596.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/596","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":596,"pages":{"first":4296,"last":4302},"keywords":["Natural Language Processing: Language Models","Natural Language Processing: Text Classification","Natural Language Processing: Other","Machine Learning: Multi-task and Transfer Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986450","length_sec":87,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984593","length_sec":345,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Auer et al., 2007] Sören Auer, Christian Bizer, Georgi Ko- bilarov, Jens Lehmann, Richard Cyganiak, and Zachary G. Ives. Dbpedia: A nucleus for a web of open data. In Proc. of ISWC, 2007.","title":"Dbpedia: A nucleus for a web of open data","authors":"Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, Zachary G Ives","year":"2007","venue":"Proc. of ISWC","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2019] Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student net- works. In Proc. of ICCV, 2019.","title":"Data-free learning of student networks","authors":"Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, Qi Tian","year":"2019","venue":"Proc. of ICCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2020] Daoyuan Chen, Yaliang Li, Minghui Qiu, Zhen Wang, Bofang Li, Bolin Ding, Hongbo Deng, Jun Huang, Wei Lin, and Jingren Zhou. Adabert: Task- adaptive BERT compression with differentiable neural ar- chitecture search. In Proc. of IJCAI, 2020.","title":"Adabert: Taskadaptive BERT compression with differentiable neural architecture search","authors":"Daoyuan Chen, Yaliang Li, Minghui Qiu, Zhen Wang, Bofang Li, Bolin Ding, Hongbo Deng, Jun Huang, Wei Lin, Jingren Zhou","year":"2020","venue":"Proc. of IJCAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fang et al., 2021] Gongfan Fang, Jie Song, Xinchao Wang, Chengchao Shen, Xingen Wang, and Mingli Song. Con- trastive model invertion for data-free knolwedge distilla- tion. In Proc. of IJCAI, 2021.","title":"Contrastive model invertion for data-free knolwedge distillation","authors":"Fang","year":"2021","venue":"Proc. of IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/327"},{"position":5,"raw":"[Goodfellow et al., 2015] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Proc. of ICLR, 2015.","title":"Explaining and harnessing adversarial examples","authors":"Ian J Goodfellow, Jonathon Shlens, Christian Szegedy","year":"2015","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hinton et al., 2015] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural net- work. CoRR, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey E Hinton, Oriol Vinyals, Jeffrey Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Holtzman et al., 2020] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neu- ral text degeneration. In Proc. of ICLR, 2020.","title":"The curious case of neural text degeneration","authors":"Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi","year":"2020","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hou et al., 2020] Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic BERT with adaptive width and depth. In Proc. of NeuIPS, 2020.","title":"Dynabert: Dynamic BERT with adaptive width and depth","authors":"Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, Qun Liu","year":"2020","venue":"Proc. of NeuIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiao et al., 2020] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling BERT for natural language un- derstanding. In Proc. of ACL, 2020.","title":"Tinybert: Distilling BERT for natural language understanding","authors":"Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, Qun Liu","year":"2020","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li and Liang, 2021] Xiang Lisa Li and Percy Liang. Prefix- tuning: Optimizing continuous prompts for generation. In Proc. of ACL, 2021.","title":"Prefixtuning: Optimizing continuous prompts for generation","authors":"Lisa Xiang, Percy Li, Liang","year":"2021","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lopes et al., 2017] Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. arXiv preprint arXiv:1710.07535, 2017.","title":"Data-free knowledge distillation for deep neural networks","authors":"Gontijo Raphael, Stefano Lopes, Thad Fenu, Starner","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ma et al., 2020] Xinyin Ma, Yongliang Shen, Gongfan Fang, Chen Chen, Chenghao Jia, and Weiming Lu. Ad- versarial self-supervised data-free distillation for text clas- sification. In Proc. of EMNLP, 2020.","title":"Adversarial self-supervised data-free distillation for text classification","authors":"Xinyin Ma, Yongliang Shen, Gongfan Fang, Chen Chen, Chenghao Jia, Weiming Lu","year":"2020","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Maas et al., 2011] Andrew L. Maas, Raymond E. Daly, Pe- ter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proc. of ACL, 2011.","title":"Learning word vectors for sentiment analysis","authors":"Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, Christopher Potts","year":"2011","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mahendran and Vedaldi, 2015] Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representa- tions by inverting them. In CVPR, 2015. [Micaelli and Storkey, 2019] Paul Micaelli and Amos J Storkey. Zero-shot knowledge transfer via adversarial be- lief matching. In Proc. of NeuIPS, 2019.","title":"Zero-shot knowledge transfer via adversarial belief matching","authors":"Vedaldi Mahendran, Aravindh Mahendran, Andrea Vedaldi, Paul Micaelli, Amos J Storkey","year":"2015","venue":"Proc. of NeuIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Nayak et al., 2019] Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj, Venkatesh Babu Radhakrishnan, and Anirban Chakraborty. Zero-shot knowledge distilla- tion in deep networks. In Proc. of ICML, 2019.","title":"Zero-shot knowledge distillation in deep networks","authors":"Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj, Venkatesh Babu Radhakrishnan, Anirban Chakraborty","year":"2019","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 2019.","title":"Language models are unsupervised multitask learners","authors":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever","year":"2019","venue":"OpenAI Blog","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rashid et al., 2021] Ahmad Rashid, Vasileios Lioutas, Ab- bas Ghaddar, and Mehdi Rezagholizadeh. Towards zero- shot knowledge distillation for natural language process- ing. In Proc. of EMNLP, 2021.","title":"Towards zeroshot knowledge distillation for natural language processing","authors":"Ahmad Rashid, Vasileios Lioutas, Abbas Ghaddar, Mehdi Rezagholizadeh","year":"2021","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sanh et al., 2019] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled ver- sion of BERT: smaller, faster, cheaper and lighter. CoRR, 2019.","title":"Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter","authors":"Victor Sanh, Lysandre Debut, Julien Chaumond, Thomas Wolf","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shin et al., 2020] Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Auto- prompt: Eliciting knowledge from language models with automatically generated prompts. In Proc. of EMNLP, 2020.","title":"Autoprompt: Eliciting knowledge from language models with automatically generated prompts","authors":"Taylor Shin, Yasaman Razeghi, Robert L Logan, I V, Eric Wallace, Sameer Singh","year":"2020","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Socher et al., 2013] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for seman- tic compositionality over a sentiment treebank. In Proc. of EMNLP, 2013.","title":"Recursive deep models for semantic compositionality over a sentiment treebank","authors":"Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, Christopher Potts","year":"2013","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sun et al., 2019] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for BERT model com- pression. In Proc. of EMNLP, 2019.","title":"Patient knowledge distillation for BERT model compression","authors":"Siqi Sun, Yu Cheng, Zhe Gan, Jingjing Liu","year":"2019","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sutton et al., 1999] Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Proc. of NeuIPS, 1999.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"Richard S Sutton, David A Mcallester, P Satinder, Yishay Singh, Mansour","year":"1999","venue":"Proc. of NeuIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2021] Yimeng Wu, Mehdi Rezagholizadeh, Ab- bas Ghaddar, Md Akmal Haidar, and Ali Ghodsi. Universal-KD: Attention-based output-grounded interme- diate layer knowledge distillation. In Proc. of EMNLP, 2021.","title":"Universal-KD: Attention-based output-grounded intermediate layer knowledge distillation","authors":"Yimeng Wu, Mehdi Rezagholizadeh, Abbas Ghaddar, Md Akmal Haidar, Ali Ghodsi","year":"2021","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yin et al., 2020] Hongxu Yin, Pavlo Molchanov, Jose M. Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Ni- raj K. Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proc. of CVPR, 2020.","title":"Dreaming to distill: Data-free knowledge transfer via deepinversion","authors":"Yin","year":"2020","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yu et al., 2017] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Proc. of AAAI, 2017.","title":"Seqgan: Sequence generative adversarial nets with policy gradient","authors":"Lantao Yu, Weinan Zhang, Jun Wang, Yong Yu","year":"2017","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2015] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Proc. of NeuIPS, 2015.","title":"Character-level convolutional networks for text classification","authors":"Xiang Zhang, Junbo Zhao, Yann Lecun","year":"2015","venue":"Proc. of NeuIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}