{"type":"ScholarlyArticle","title":"Policy Optimization with Second-Order Advantage Information","authors":[{"name":"Jiajin Li","slug":"jiajin-li","given":"Jiajin","family":"Li","affiliation":"The Chinese University of Hong Kong"},{"name":"Baoxiang Wang","slug":"baoxiang-wang","given":"Baoxiang","family":"Wang","affiliation":"The Chinese University of Hong Kong"},{"name":"Shengyu Zhang","slug":"shengyu-zhang","given":"Shengyu","family":"Zhang","affiliation":"The Chinese University of Hong Kong; Tencent"}],"abstract":"Policy optimization on high-dimensional continuous control tasks exhibits its difficulty caused by the large variance of the policy gradient estimators. We present the action subspace dependent gradient (ASDG) estimator which incorporates the Rao-Blackwell theorem (RB) and Control Variates (CV) into a unified framework to reduce the variance. To invoke RB, our proposed algorithm (POSA) learns the underlying factorization structure among the action space based on the second-order advantage information.  POSA captures the quadratic information explicitly and efficiently by utilizing the wide \\& deep architecture. Empirical studies show that our proposed approach demonstrates the performance improvements on high-dimensional synthetic settings and OpenAI Gym's MuJoCo continuous control tasks.","doi":"10.24963/ijcai.2018/699","url":"https://ijcai.org/proceedings/2018/699","pdf":"https://ijcai.org/proceedings/2018/0699.pdf","markdown":"https://ijcai.org/proceedings/2018/699.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/699","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Uncertainty in AI","track":"Main track","number":699,"pages":{"first":5038,"last":5044},"keywords":["Planning and Scheduling: Markov Decisions Processes","Machine Learning: Reinforcement Learning","Uncertainty in AI: Markov Decision Processes"],"videos":[],"references":[{"position":1,"raw":"[Casella and Robert, 1996] George Casella and Christian P Robert. Rao-blackwellisation of sampling schemes. Biometrika, 83(1):81-94, 1996.","title":"Rao-blackwellisation of sampling schemes","authors":"Robert ; Casella, George Casella, Christian P Robert","year":"1996","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chakrabarti et al., 2006] Deepayan Chakrabarti, Ravi Ku- mar, and Andrew Tomkins. Evolutionary clustering. In Proceedings of the 12th ACM SIGKDD international con- ference on Knowledge discovery and data mining, pages 554-560. ACM, 2006.","title":"Deepayan Chakrabarti, Ravi","authors":"Chakrabarti","year":"2006","venue":"Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cheng et al., 2016] Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recom- mender systems. In Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, pages 7-10. ACM, 2016.","title":"Wide & deep learning for recommender systems","authors":"Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir","year":"2016","venue":"Proceedings of the 1st Workshop on Deep Learning for Recommender Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Degris et al., 2012] Thomas Degris, Martha White, and Richard S Sutton. Off-policy actor-critic. arXiv preprint arXiv:1205.4839, 2012.","title":"Off-policy actor-critic","authors":"Thomas Degris, Martha White, Richard S Sutton","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Duan et al., 2016] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In Inter- national Conference on Machine Learning, pages 1329- 1338, 2016.","title":"Benchmarking deep reinforcement learning for continuous control","authors":"Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Grathwohl et al., 2018] Will Grathwohl, Dami Choi, Yuhuai Wu, Geoff Roeder, and David Duvenaud. Back- propagation through the void: Optimizing control variates for black-box gradient estimation. In International Conference on Learning Representations, 2018.","title":"Backpropagation through the void: Optimizing control variates for black-box gradient estimation","authors":"Grathwohl","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gu et al., 2016] Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard E Turner, and Sergey Levine. Q- prop: Sample-efficient policy gradient with an off-policy critic. arXiv preprint arXiv:1611.02247, 2016.","title":"Qprop: Sample-efficient policy gradient with an off-policy critic","authors":"Gu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kingma and Welling, 2013] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.","title":"Auto-encoding variational bayes","authors":"P Diederik, Max Kingma, Welling","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Levine et al., 2016] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Re- search, 17(39):1-40, 2016.","title":"End-to-end training of deep visuomotor policies","authors":"Sergey Levine, Chelsea Finn, Trevor Darrell, Pieter Abbeel","year":"2016","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2016] Shuai Li, Baoxiang Wang, Shengyu Zhang, and Wei Chen. Contextual combinatorial cascading ban- dits. In International Conference on Machine Learning, pages 1245-1253, 2016.","title":"Contextual combinatorial cascading bandits","authors":"Shuai Li, Baoxiang Wang, Shengyu Zhang, Wei Chen","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous con- trol with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2018] Hao Liu, Yihao Feng, Yi Mao, Dengyong Zhou, Jian Peng, and Qiang Liu. Action-dependent control variates for policy optimization via stein identity. In Inter- national Conference on Learning Representations, 2018. [Mnih et al., 2015] Volodymyr Mnih, KoPieterray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Action-dependent control variates for policy optimization via stein identity","authors":"Hao Liu, Yihao Feng, Yi Mao, Dengyong Zhou, Jian Peng, Qiang Liu, ; Mnih","year":"2015","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asyn- chronous methods for deep reinforcement learning. In In- ternational Conference on Machine Learning, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Oates et al., 2017] Chris J Oates, Mark Girolami, and Nico- las Chopin. Control functionals for monte carlo integra- tion. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 79(3):695-718, 2017.","title":"Control functionals for monte carlo integration","authors":"Chris J Oates, Mark Girolami, Nicolas Chopin","year":"2017","venue":"Journal of the Royal Statistical Society: Series B (Statistical Methodology)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ranganath et al., 2014] Rajesh Ranganath, Sean Gerrish, and David Blei. Black box variational inference. In Ar- tificial Intelligence and Statistics, pages 814-822, 2014.","title":"Black box variational inference","authors":"Rajesh Ranganath, Sean Gerrish, David Blei","year":"2014","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ranganath et al., 2016] Rajesh Ranganath, Dustin Tran, and David Blei. Hierarchical variational models. In In- ternational Conference on Machine Learning, pages 324- 333, 2016.","title":"Hierarchical variational models","authors":"Rajesh Ranganath, Dustin Tran, David Blei","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Schulman et al., 2015] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advan- tage estimation. arXiv preprint arXiv:1506.02438, 2015.","title":"Highdimensional continuous control using generalized advantage estimation","authors":"John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017.","title":"Mastering the game of go without human knowledge","authors":"David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tucker et al., 2018] George Tucker, Surya Bhupatiraju, Shixiang Gu, Richard E Turner, Zoubin Ghahramani, and Sergey Levine. The mirage of action-dependent baselines in reinforcement learning. arXiv preprint arXiv:1802.10031, 2018.","title":"The mirage of action-dependent baselines in reinforcement learning","authors":"George Tucker, Surya Bhupatiraju, Shixiang Gu, Richard E Turner, Zoubin Ghahramani, Sergey Levine","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3-4):229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wu et al., 2017] Cathy Wu, Kanaad Parvate, Nishant Kheterpal, Leah Dickstein, Ankur Mehta, Eugene Vinit- sky, and Alexandre Bayen. Framework for control and deep reinforcement learning in traffic. In Intelligent Trans- portation Systems (ITSC), 2017 IEEE 20th International Conference on, pages 1-8. IEEE, 2017.","title":"Framework for control and deep reinforcement learning in traffic","authors":"Cathy Wu, Kanaad Parvate, Nishant Kheterpal, Leah Dickstein, Ankur Mehta, Eugene Vinitsky, Alexandre Bayen","year":"2017","venue":"Intelligent Transportation Systems (ITSC)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2018] Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre M.Bayen, Sham Kakade, Igor Mordatch, and Pieter Abbeel. Variance reduction for policy gradient with action-dependent factorized base- lines. In International Conference on Learning Represen- tations, 2018.","title":"Variance reduction for policy gradient with action-dependent factorized baselines","authors":"Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre M Bayen, Sham Kakade, Igor Mordatch, Pieter Abbeel","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}