{"type":"ScholarlyArticle","title":"Dual Policy Distillation","authors":[{"name":"Kwei-Herng Lai","slug":"kwei-herng-lai","given":"Kwei-Herng","family":"Lai","affiliation":"Department of Computer Science and Engineering , Texas A&M University"},{"name":"Daochen Zha","slug":"daochen-zha","given":"Daochen","family":"Zha","affiliation":"Department of Computer Science and Engineering , Texas A&M University"},{"name":"Yuening Li","slug":"yuening-li","given":"Yuening","family":"Li","affiliation":"Department of Computer Science and Engineering , Texas A&M University"},{"name":"Xia Hu","slug":"xia-hu","given":"Xia","family":"Hu","affiliation":"Department of Computer Science and Engineering , Texas A&M University"}],"abstract":"Policy distillation, which transfers a teacher policy to a student policy has achieved great success in challenging tasks of deep reinforcement learning. This teacher-student framework requires a well-trained teacher model which is computationally expensive. Moreover, the performance of the student model could be limited by the teacher model if the teacher model is not optimal. In the light of collaborative learning, we study the feasibility of involving joint intellectual efforts from diverse perspectives of student models. In this work, we introduce dual policy distillation (DPD), a student-student framework in which two learners operate on the same environment to explore different perspectives of the environment and extract knowledge from each other to enhance their learning. The key challenge in developing this dual learning framework is to identify the beneficial knowledge from the peer learner for contemporary learning-based reinforcement learning algorithms, since it is unclear whether the knowledge distilled from an imperfect and noisy peer learner would be helpful. To address the challenge, we theoretically justify that distilling knowledge from a peer learner will lead to policy improvement and propose a disadvantageous distillation strategy based on the theoretical results. The conducted experiments on several continuous control tasks show that the proposed framework achieves superior performance with a learning-based agent and function approximation without the use of expensive teacher models.","doi":"10.24963/ijcai.2020/435","url":"https://ijcai.org/proceedings/2020/435","pdf":"https://ijcai.org/proceedings/2020/0435.pdf","markdown":"https://ijcai.org/proceedings/2020/435.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/435","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":435,"pages":{"first":3146,"last":3152},"keywords":["Machine Learning: Reinforcement Learning","Agent-based and Multi-agent Systems: Multi-agent Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24581/file_24581.m3u8","length_sec":295,"provider":"hls"}],"references":[{"position":1,"raw":"Abbeel and Ng, ] Pieter Abbeel and Andrew Y Ng. Appren- ticeship learning via inverse reinforcement learning. page 1.","title":"Apprenticeship learning via inverse reinforcement learning","authors":"Abbeel, Pieter Ng, Andrew Y Abbeel, Ng","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bain and Sommut, 1999] Michael Bain and Claude Sommut. A framework for behavioural cloning. Machine Intelligence, pages 103-129, 1999.","title":"A framework for behavioural cloning","authors":"Michael Bain, Claude Sommut","year":"1999","venue":"Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Lud- wig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Czarnecki et al., 2019] Wojciech Marian Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant M Jayakumar, Grze- gorz Swirszcz, and Max Jaderberg. Distilling policy distil- lation. arXiv preprint arXiv:1902.02186, 2019.","title":"Distilling policy distillation","authors":"Marian Wojciech, Razvan Czarnecki, Simon Pascanu, Osindero, M Siddhant, Grzegorz Jayakumar, Max Swirszcz, Jaderberg","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Da Silva et al., 2017] Felipe Leno Da Silva, Ruben Glatt, and Anna Helena Reali Costa. Simultaneously learning and advising in multiagent reinforcement learning. In AAMAS, pages 1100-1108, 2017.","title":"Simultaneously learning and advising in multiagent reinforcement learning","authors":"Da Silva","year":"2017","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dillenbourg, 1999] Pierre Dillenbourg. Collaborative learn- ing: Cognitive and computational approaches. advances in learning and instruction series. ERIC, 1999.","title":"Collaborative learning: Cognitive and computational approaches","authors":"Dillenbourg","year":"1999","venue":"advances in learning and instruction series","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Finn et al., 2016] Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In ICML, pages 49-58, 2016.","title":"Guided cost learning: Deep inverse optimal control via policy optimization","authors":"Finn","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hadfield-Menell et al., 2016] Dylan Hadfield-Menell, Stu- art J Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning. In NeurIPS, pages 3909- 3917, 2016.","title":"Cooperative inverse reinforcement learning","authors":"Hadfield-Menell","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hester et al., 2018] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Hor- gan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In AAAI, 2018.","title":"Deep q-learning from demonstrations","authors":"Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ho and Ermon, 2016] Jonathan Ho and Stefano Ermon. Gen- erative adversarial imitation learning. In NeurIPS, pages 4565-4573, 2016.","title":"Generative adversarial imitation learning","authors":"Jonathan Ho, Stefano Ermon","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Levine et al., 2016] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, pages 1334-1373, 2016.","title":"End-to-end training of deep visuomotor policies","authors":"Sergey Levine, Chelsea Finn, Trevor Darrell, Pieter Abbeel","year":"2016","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lillicrap et al., 2016] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Littman, 1994] Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Ma- chine learning proceedings 1994, pages 157-163. 1994.","title":"Markov games as a framework for multi-agent reinforcement learning","authors":"Michael L Littman","year":"1994","venue":"Machine learning proceedings","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Oh et al., 2018] Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In ICML, pages 3878-3887, 2018.","title":"Self-imitation learning","authors":"Junhyuk Oh, Yijie Guo, Satinder Singh, Honglak Lee","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Omidshafiei et al., 2019] Shayegan Omidshafiei, Dong-Ki Kim, Miao Liu, Gerald Tesauro, Matthew Riemer, Christo- pher Amato, Murray Campbell, and Jonathan P How. Learn- ing to teach in cooperative multiagent reinforcement learn- ing. In AAAI, pages 6128-6136, 2019.","title":"Learning to teach in cooperative multiagent reinforcement learning","authors":"Shayegan Omidshafiei, Dong-Ki Kim, Miao Liu, Gerald Tesauro, Matthew Riemer, Christopher Amato, Murray Campbell, Jonathan P How","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rusu et al., 2016] Andrei A Rusu, Sergio Gomez Col- menarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. In ICLR, 2016.","title":"Policy distillation","authors":"Rusu","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Salimans et al., 2017] Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864, 2017.","title":"Evolution strategies as a scalable alternative to reinforcement learning","authors":"Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, Ilya Sutskever","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Shoham et al., 2003] Yoav Shoham, Rob Powers, and Trond Grenager. Multi-agent reinforcement learning: a critical survey. Web manuscript, 2003.","title":"Multi-agent reinforcement learning: a critical survey","authors":"Yoav Shoham, Rob Powers, Trond Grenager","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, page 354, 2017.","title":"Mastering the game of go without human knowledge","authors":"David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Smith and MacGregor, 1992] Barbara Leigh Smith and Jean T MacGregor. What is collaborative learning, 1992.","title":"What is collaborative learning","authors":"Barbara Leigh, Smith, Jean T Macgregor","year":"1992","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tan, 1993] Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In ICML, pages 330- 337, 1993.","title":"Multi-agent reinforcement learning: Independent vs. cooperative agents","authors":"Ming Tan, Tan","year":"1993","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Teh et al., 2017] Yee Teh, Victor Bapst, Wojciech M Czar- necki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask rein- forcement learning. In NeurIPS, pages 4496-4506, 2017.","title":"Distral: Robust multitask reinforcement learning","authors":"Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, Razvan Pascanu","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2018a] Tianbing Xu, Qiang Liu, Liang Zhao, and Jian Peng. Learning to explore via meta-policy gradient. In ICML, pages 5463-5472, 2018.","title":"Learning to explore via meta-policy gradient","authors":"Xu","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xu et al., 2018b] Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning. In NeurIPS, pages 2396-2407, 2018.","title":"Meta-gradient reinforcement learning","authors":"Xu","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yin and Pan, 2017] Haiyan Yin and Sinno Jialin Pan. Knowl- edge transfer for deep reinforcement learning with hierar- chical experience replay. In AAAI, 2017.","title":"Knowledge transfer for deep reinforcement learning with hierarchical experience replay","authors":"Haiyan Yin, Sinno Jialin Pan","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zha et al., 2019a] Daochen Zha, Kwei-Herng Lai, Yuanpu Cao, Songyi Huang, Ruzhe Wei, Junyu Guo, and Xia Hu. Rlcard: A toolkit for reinforcement learning in card games. arXiv preprint arXiv:1910.04376, 2019.","title":"Rlcard: A toolkit for reinforcement learning in card games","authors":"Zha","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zha et al., 2019b] Daochen Zha, Kwei-Herng Lai, Kaixiong Zhou, and Xia Hu. Experience replay optimization. In IJCAI, pages 4243-4249, 2019.","title":"Experience replay optimization","authors":"Zha","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/589"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}