{"type":"ScholarlyArticle","title":"Diversity-Inducing Policy Gradient: Using Maximum Mean Discrepancy to Find a Set of Diverse Policies","authors":[{"name":"Muhammad Masood","slug":"muhammad-masood","given":"Muhammad","family":"Masood","affiliation":"Harvard University"},{"name":"Finale Doshi-Velez","slug":"finale-doshi-velez","given":"Finale","family":"Doshi-Velez","affiliation":"Harvard University"}],"abstract":"Standard reinforcement learning methods aim to master one way of solving a task whereas there may exist multiple near-optimal policies. Being able to identify this collection of near-optimal policies can allow a domain expert to efficiently explore the space of reasonable solutions.  Unfortunately, existing approaches that quantify uncertainty over policies are not ultimately relevant to finding policies with qualitatively distinct behaviors.  In this work, we formalize the difference between policies as a difference between the distribution of trajectories induced by each policy, which encourages diversity with respect to both state visitation and action choices.  We derive a gradient-based optimization technique that can be combined with existing policy gradient methods to now identify diverse collections of well-performing policies.  We demonstrate our approach on benchmarks and a healthcare task.","doi":"10.24963/ijcai.2019/821","url":"https://ijcai.org/proceedings/2019/821","pdf":"https://ijcai.org/proceedings/2019/0821.pdf","markdown":"https://ijcai.org/proceedings/2019/821.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/821","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"AI for Improving Human Well-being","track":"AI for Improving Human Well-being","number":821,"pages":{"first":5923,"last":5929},"keywords":["Special Track on AI for Improving Human-Well Being: Health applications (Special Track on AI and Human Wellbeing)"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2018] W. Y. Chen, L. Mackey, J. Gorham, F.-X. Briol, and C. J. Oates. Stein Points. ArXiv e-prints, March 2018.","title":"Stein Points","authors":"W Y Chen, L Mackey, J Gorham, F.-X Briol, C J Oates","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cohen et al., 2019] Andrew Cohen, Xingye Qiao, Lei Yu, Elliot Way, and Xiangrong Tong. Diverse exploration via conjugate policies for policy gradient methods. arXiv preprint arXiv:1902.03633, 2019.","title":"Diverse exploration via conjugate policies for policy gradient methods","authors":"Andrew Cohen, Xingye Qiao, Lei Yu, Elliot Way, Xiangrong Tong","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Eysenbach et al., 2018] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070, 2018.","title":"Diversity is all you need: Learning skills without a reward function","authors":"Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, Sergey Levine","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fard and Pineau, 2011] Mahdi Milani Fard and Joelle Pineau. Non-deterministic policies in markovian decision processes. J. Artif. Intell. Res.(JAIR), 40:1-24, 2011.","title":"Non-deterministic policies in markovian decision processes","authors":"Pineau ; Fard, Milani Mahdi, Joelle Fard, Pineau","year":"2011","venue":"J. Artif. Intell. Res.(JAIR)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gangwani et al., 2018] Tanmay Gangwani, Qiang Liu, and Jian Peng. Learning self-imitating diverse policies. arXiv preprint arXiv:1805.10309, 2018.","title":"Learning self-imitating diverse policies","authors":"Tanmay Gangwani, Qiang Liu, Jian Peng","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ghavamzadeh et al., 2015] Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, Aviv Tamar, et al. Bayesian reinforcement learning: A survey. Foundations and Trends R in Machine Learning, 8(5-6):359-483, 2015.","title":"Bayesian reinforcement learning: A survey","authors":"Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, Aviv Tamar","year":"2015","venue":"Foundations and Trends R in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gretton et al., 2007] Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex J Smola. A kernel method for the two-sample-problem. In Advances in neural information processing systems, pages 513-520, 2007.","title":"A kernel method for the two-sample-problem","authors":"Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, Alex J Smola","year":"2007","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gretton et al., 2012] Arthur Gretton, Dino Sejdinovic, Heiko Strathmann, Sivaraman Balakrishnan, Mas- similiano Pontil, Kenji Fukumizu, and Bharath K Sriperumbudur. Optimal kernel choice for large-scale two-sample tests. In Advances in neural information processing systems, pages 1205-1213, 2012.","title":"Optimal kernel choice for large-scale two-sample tests","authors":"Arthur Gretton, Dino Sejdinovic, Heiko Strathmann, Sivaraman Balakrishnan, Massimiliano Pontil, Kenji Fukumizu, Bharath K Sriperumbudur","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Haarnoja et al., 2017] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learn- ing with deep energy-based policies. arXiv preprint arXiv:1702.08165, 2017.","title":"Reinforcement learning with deep energy-based policies","authors":"Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, Sergey Levine","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hong et al., 2018] Zhang-Wei Hong, Tzu-Yun Shann, Shih- Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, and Chun-Yi Lee. Diversity-driven exploration strategy for deep reinforce- ment learning. In Advances in Neural Information Pro- cessing Systems, pages 10489-10500, 2018.","title":"Diversity-driven exploration strategy for deep reinforcement learning","authors":"Zhang-Wei Hong, Tzu-Yun Shann, Shih-Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, Chun-Yi Lee","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Johnson et al., 2017] Alistair EW Johnson, David J Stone, Leo A Celi, and Tom J Pollard. The mimic code repos- itory: enabling reproducibility in critical care research. Journal of the American Medical Informatics Association, 25(1):32-39, 2017.","title":"The mimic code repository: enabling reproducibility in critical care research","authors":"E W Alistair, David J Johnson, Leo A Stone, Tom J Celi, Pollard","year":"2017","venue":"Journal of the American Medical Informatics Association","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lange et al., 2012] Sascha Lange, Thomas Gabel, and Mar- tin Riedmiller. Batch reinforcement learning. In Rein- forcement learning, pages 45-73. Springer, 2012.","title":"Batch reinforcement learning","authors":"Sascha Lange, Thomas Gabel, Martin Riedmiller","year":"2012","venue":"Reinforcement learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lehman and Stanley, 2011] Joel Lehman and Kenneth O Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computation, 19(2):189-223, 2011.","title":"Abandoning objectives: Evolution through the search for novelty alone","authors":"Stanley ; Lehman, Joel Lehman, Kenneth O Stanley","year":"2011","venue":"Evolutionary computation","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Levine and Koltun, 2013] Sergey Levine and Vladlen Koltun. Guided policy search. In International Confer- ence on Machine Learning, pages 1-9, 2013.","title":"Guided policy search","authors":"Koltun Levine, Sergey Levine, Vladlen Koltun","year":"2013","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2014] Chunming Liu, Xin Xu, and Dewen Hu. Multiobjective reinforcement learning: A comprehensive overview. IEEE Transactions on Systems, Man, and Cy- bernetics: Systems, 45(3):385-398, 2014.","title":"Multiobjective reinforcement learning: A comprehensive overview","authors":"Chunming Liu, Xin Xu, Dewen Hu","year":"2014","venue":"IEEE Transactions on Systems, Man, and Cybernetics: Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2017] Yang Liu, Prajit Ramachandran, Qiang Liu, and Jian Peng. Stein variational policy gradient. arXiv preprint arXiv:1704.02399, 2017.","title":"Stein variational policy gradient","authors":"Yang Liu, Prajit Ramachandran, Qiang Liu, Jian Peng","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Masood and Doshi-Velez, 2019] Muhammad A Masood and Finale Doshi-Velez. Diversity-inducing policy gra- dient: Using maximum mean discrepancy to find a set of diverse policies. arXiv preprint arXiv:1906.00088, 2019.","title":"Muhammad A Masood and Finale Doshi-Velez. Diversity-inducing policy gradient: Using maximum mean discrepancy to find a set of diverse policies","authors":"Doshi-Velez Masood","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Smith et al., 2018] Matthew Smith, Herke Hoof, and Joelle Pineau. An inference-based policy gradient method for learning options. In International Conference on Machine Learning, pages 4710-4719, 2018.","title":"An inference-based policy gradient method for learning options","authors":"Matthew Smith, Herke Hoof, Joelle Pineau","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sohrabi et al., 2016] Shirin Sohrabi, Anton V Riabov, Octa- vian Udrea, and Oktie Hassanzadeh. Finding diverse high- quality plans for hypothesis generation. In ECAI, pages 1581-1582, 2016.","title":"Finding diverse highquality plans for hypothesis generation","authors":"Shirin Sohrabi, Anton V Riabov, Octavian Udrea, Oktie Hassanzadeh","year":"2016","venue":"ECAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sriperumbudur et al., 2010] Bharath K Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, and Gert RG Lanckriet. Non-parametric estimation of integral probability metrics. In Information Theory Proceedings (ISIT), 2010 IEEE International Symposium on, pages 1428-1432. IEEE, 2010.","title":"Non-parametric estimation of integral probability metrics","authors":"Kenji Bharath K Sriperumbudur, Arthur Fukumizu, Bernhard Gretton, Gert Rg Schölkopf, Lanckriet","year":"2010","venue":"Information Theory Proceedings (ISIT), 2010 IEEE International Symposium on","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Srivastava et al., 2007] Biplav Srivastava, Tuan Anh Nguyen, Alfonso Gerevini, Subbarao Kambhampati, Minh Binh Do, and Ivan Serina. Domain independent approaches for finding diverse plans. In IJCAI, pages 2016-2022, 2007.","title":"Domain independent approaches for finding diverse plans","authors":"Srivastava","year":"2007","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2007/325"},{"position":23,"raw":"[Sutton et al., 2000] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approxi- mation. In Advances in neural information processing sys- tems, pages 1057-1063, 2000.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"David A Richard S Sutton, Mcallester, P Satinder, Yishay Singh, Mansour","year":"2000","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Thomas, 2015] Philip S Thomas. Safe reinforcement learn- ing. PhD thesis, University of Massachusetts Libraries, 2015.","title":"Safe reinforcement learning","authors":"Thomas Philip","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Toussaint and Lopes, 2017] Marc Toussaint and Manuel Lopes. Multi-bound tree search for logic-geometric pro- gramming in cooperative manipulation domains. In (ICRA 2017), 2017.","title":"Multi-bound tree search for logic-geometric programming in cooperative manipulation domains","authors":"Lopes Toussaint, Marc Toussaint, Manuel Lopes","year":"2017","venue":"ICRA 2017","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}