{"type":"ScholarlyArticle","title":"Soft Policy Gradient Method for Maximum Entropy Deep Reinforcement Learning","authors":[{"name":"Wenjie Shi","slug":"wenjie-shi","given":"Wenjie","family":"Shi","affiliation":"Department of Automation , Tsinghua University , Beijing , China"},{"name":"Shiji Song","slug":"shiji-song","given":"Shiji","family":"Song","affiliation":"Department of Automation , Tsinghua University , Beijing , China"},{"name":"Cheng Wu","slug":"cheng-wu","given":"Cheng","family":"Wu","affiliation":"Department of Automation , Tsinghua University , Beijing , China"}],"abstract":"Maximum entropy deep reinforcement learning (RL) methods have been demonstrated on a range of challenging continuous tasks. However, existing methods either suffer from severe instability when training on large off-policy data or cannot scale to tasks with very high state and action dimensionality such as 3D humanoid locomotion. Besides, the optimality of desired Boltzmann policy set for non-optimal soft value function is not persuasive enough. In this paper, we first derive soft policy gradient based on entropy regularized expected reward objective for RL with continuous actions. Then, we present an off-policy actor-critic, model-free maximum entropy deep RL algorithm called deep soft policy gradient (DSPG) by combining soft policy gradient with soft Bellman equation. To ensure stable learning while eliminating the need of two separate critics for soft value functions, we leverage double sampling approach to making the soft Bellman equation tractable. The experimental results demonstrate that our method outperforms in performance over off-policy prior methods.","doi":"10.24963/ijcai.2019/475","url":"https://ijcai.org/proceedings/2019/475","pdf":"https://ijcai.org/proceedings/2019/0475.pdf","markdown":"https://ijcai.org/proceedings/2019/475.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/475","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":475,"pages":{"first":3425,"last":3431},"keywords":["Machine Learning Applications: Applications of Reinforcement Learning","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Bertsekas and Tsitsiklis, 1996] D Bertsekas and J Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, 1996.","title":"Neuro-Dynamic Programming","authors":"D Bertsekas, Tsitsiklis","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bhatnagar et al., 2008] Shalabh Bhatnagar, Mohammad Ghavamzadeh, Mark Lee, and Richard S Sutton. Incre- mental natural actor-critic algorithms. In Advances in neural information processing systems, pages 105-112, 2008.","title":"Incremental natural actor-critic algorithms","authors":"Shalabh Bhatnagar, Mohammad Ghavamzadeh, Mark Lee, Richard S Sutton","year":"2008","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Degris et al., 2012] Thomas Degris, Patrick M Pilarski, and Richard S Sutton. Model-free reinforcement learning with continuous action in practice. In American Control Con- ference (ACC), 2012, pages 2177-2182. IEEE, 2012.","title":"Model-free reinforcement learning with continuous action in practice","authors":"Thomas Degris, Patrick M Pilarski, Richard S Sutton","year":"2012","venue":"American Control Conference (ACC)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fox et al., 2015] Roy Fox, Ari Pakman, and Naftali Tish- by. Taming the noise in reinforcement learning via soft updates. arXiv preprint arXiv:1512.08562, 2015.","title":"Taming the noise in reinforcement learning via soft updates","authors":"Roy Fox, Ari Pakman, Naftali Tishby","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gu et al., 2017] Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard E Turner, and Sergey Levine. Q- prop: Sample-efficient policy gradient with an off-policy critic. In International Conference on Learning Represen- tations, 2017.","title":"Qprop: Sample-efficient policy gradient with an off-policy critic","authors":"Gu","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Haarnoja et al., 2017] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learn- ing with deep energy-based policies. arXiv preprint arX- iv:1702.08165, 2017.","title":"Reinforcement learning with deep energy-based policies","authors":"Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, Sergey Levine","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290, 2018.","title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","authors":"Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Henderson et al., 2017] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. arXiv preprint arXiv:1709.06560, 2017.","title":"Deep reinforcement learning that matters","authors":"Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, David Meger","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Levine et al., 2016] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Re- search, 17(1):1334-1373, 2016.","title":"End-to-end training of deep visuomotor policies","authors":"Sergey Levine, Chelsea Finn, Trevor Darrell, Pieter Abbeel","year":"2016","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tas- sa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arX- iv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, et al. Human- level control through deep reinforcement learning. Nature, 518(7540):529, 2015.","title":"Humanlevel control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Nachum et al., 2017a] Ofir Nachum, Mohammad Norouz- i, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. In Advances in Neural Information Processing Systems, pages 2775-2785, 2017.","title":"Bridging the gap between value and policy based reinforcement learning","authors":"Nachum","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Nachum et al., 2017b] Ofir Nachum, Mohammad Norouz- i, Kelvin Xu, and Dale Schuurmans. Trust-pcl: An off- policy trust region method for continuous control. arXiv preprint arXiv:1707.01891, 2017.","title":"Trust-pcl: An offpolicy trust region method for continuous control","authors":"Nachum","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[O'Donoghue et al., 2016] Brendan O'Donoghue, Remi Munos, Koray Kavukcuoglu, and Volodymyr Mnih. Combining policy gradient and q-learning. arXiv preprint arXiv:1611.01626, 2016.","title":"Combining policy gradient and q-learning","authors":"O'donoghue","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Prashanth et al., 2016] LA Prashanth, Cheng Jie, Michael Fu, Steve Marcus, and Csaba Szepesvári. Cumulative prospect theory meets reinforcement learning: Prediction and control. In International Conference on Machine Learning, pages 1406-1415, 2016.","title":"Cumulative prospect theory meets reinforcement learning: Prediction and control","authors":"Prashanth","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pages 1889-1897, 2015.","title":"Trust region policy optimization","authors":"John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, Philipp Moritz","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Schulman et al., 2017] John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q- learning. arXiv preprint arXiv:1704.06440, 2017.","title":"Equivalence between policy gradients and soft qlearning","authors":"John Schulman, Xi Chen, Pieter Abbeel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shortreed et al., 2011] Susan M Shortreed, Eric Laber, Daniel J Lizotte, T Scott Stroup, Joelle Pineau, and Su- san A Murphy. Informing sequential clinical decision- making through reinforcement learning: an empirical s- tudy. Machine Learning, 84(1-2):109-136, 2011.","title":"Informing sequential clinical decisionmaking through reinforcement learning: an empirical study","authors":"Susan M Shortreed, Eric Laber, Scott Daniel J Lizotte, Joelle Stroup, Susan A Pineau, Murphy","year":"2011","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Silver et al., 2014] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning, 2014.","title":"Deterministic policy gradient algorithms","authors":"David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, Martin Riedmiller","year":"2014","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J. Mad- dison, Arthur Guez, Laurent Sifre, et al. Mastering the game of go with deep neural networks and tree search. Na- ture, 529(7587):484-489, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Singh et al., 1994] Satinder P Singh, Tommi Jaakkola, and Michael I Jordan. Learning without state-estimation in partially observable markovian decision processes. In Ma- chine Learning Proceedings 1994, pages 284-292. Elsevi- er, 1994.","title":"Learning without state-estimation in partially observable markovian decision processes","authors":"P Satinder, Tommi Singh, Michael I Jaakkola, Jordan","year":"1994","venue":"Machine Learning Proceedings 1994","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sutton et al., 2000] Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approxi- mation. In Advances in neural information processing sys- tems, pages 1057-1063, 2000.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"David A Richard S Sutton, Mcallester, P Satinder, Yishay Singh, Mansour","year":"2000","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Todorov, 2008] Emanuel Todorov. General duality between optimal control and estimation. In Decision and Control, 2008. CDC 2008. 47th IEEE Conference on, pages 4286- 4292. IEEE, 2008.","title":"General duality between optimal control and estimation","authors":"Emanuel Todorov","year":"2008","venue":"CDC 2008. 47th IEEE Conference on","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Toussaint, 2009] Marc Toussaint. Robot trajectory opti- mization using approximate inference. In Proceedings of the 26th annual international conference on machine learning, pages 1049-1056. ACM, 2009.","title":"Robot trajectory optimization using approximate inference","authors":"Marc Toussaint","year":"2009","venue":"Proceedings of the 26th annual international conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3-4):229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Ziebart et al., 2008] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.","title":"Maximum entropy inverse reinforcement learning","authors":"Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey","year":"2008","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}