{"type":"ScholarlyArticle","title":"Exploiting the Sign of the Advantage Function to Learn Deterministic Policies in Continuous Domains","authors":[{"name":"Matthieu Zimmer","slug":"matthieu-zimmer","given":"Matthieu","family":"Zimmer","affiliation":"Shanghai Jiao Tong University , UM-SJTU Joint Institute , Shanghai , China"},{"name":"Paul Weng","slug":"paul-weng","given":"Paul","family":"Weng","affiliation":"Shanghai Jiao Tong University , UM-SJTU Joint Institute , Shanghai , China"}],"abstract":"In the context of learning deterministic policies in continuous domains, we revisit an approach, which was first proposed in Continuous Actor Critic Learning Automaton (CACLA) and later extended in Neural Fitted Actor Critic (NFAC). This approach is based on a policy update different from that of deterministic policy gradient (DPG). Previous work has observed its excellent performance empirically, but a theoretical justification is lacking. To fill this gap, we provide a theoretical explanation to motivate this unorthodox policy update by relating it to another update and making explicit the objective function of the latter. We furthermore discuss in depth the properties of these updates to get a deeper understanding of the overall approach. In addition, we extend it and propose a new trust region algorithm, Penalized NFAC (PeNFAC). Finally, we experimentally demonstrate in several classic control problems that it surpasses the state-of-the-art algorithms to learn deterministic policies.","doi":"10.24963/ijcai.2019/625","url":"https://ijcai.org/proceedings/2019/625","pdf":"https://ijcai.org/proceedings/2019/0625.pdf","markdown":"https://ijcai.org/proceedings/2019/625.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/625","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":625,"pages":{"first":4496,"last":4502},"keywords":["Machine Learning: Reinforcement Learning","Planning and Scheduling: Markov Decisions Processes"],"videos":[],"references":[{"position":1,"raw":"[Colas et al., 2018] Cédric Colas, Olivier Sigaud, and Pierre- Yves Oudeyer. GEP-PG: Decoupling Exploration and Ex- ploitation in Deep Reinforcement Learning Algorithms. In International Conference on Machine Learning, Stock- holm, Sweden, July 2018.","title":"GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms","authors":"Cédric Colas, Olivier Sigaud, Pierre-Yves Oudeyer","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Henderson et al., 2018] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.","title":"Deep reinforcement learning that matters","authors":"Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, David Meger","year":"2018","venue":"Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv preprint arXiv:1502.03167, 2015.","title":"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift","authors":"Sergey Ioffe, Christian Szegedy","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Kakade and Langford, 2002] Sham Kakade and John Lang- ford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learn- ing, volume 2, pages 267-274, 2002.","title":"Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning","authors":"Langford Kakade","year":"2002","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy L. Ba. Adam: a Method for Stochastic Optimization. Inter- national Conference on Learning Representations, pages 1-13, 2015.","title":"Adam: a Method for Stochastic Optimization","authors":"P Diederik, Jimmy L Kingma, Ba","year":"2015","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Konda and Tsitsiklis, 1999] Vijay R. Konda and John N. Tsitsiklis. Actor-Critic Algorithms. Neural Information Processing Systems, 13:1008-1014, 1999.","title":"Actor-Critic Algorithms","authors":"Vijay R Konda, John N Tsitsiklis","year":"1999","venue":"Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Lillicrap et al., 2016] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yu- val Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. International Conference on Learning Representations, 2016.","title":"Continuous control with deep reinforcement learning","authors":"Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra","year":"2016","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Maatyas, 1965] Josef Maatyas. Random optimization. Au- tomation and Remote control, 26(2):246-253, 1965.","title":"Josef Maatyas. Random optimization","authors":"Maatyas","year":"1965","venue":"Automation and Remote control","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei a Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fid- jeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Has- sabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Prokhorov and Wunsch, 1997] Danil V. Prokhorov and Donald C. Wunsch. Adaptive critic designs. IEEE Trans- actions on Neural Networks, 8(5):997-1007, 1997.","title":"Adaptive critic designs","authors":"V Danil, Donald C Prokhorov, Wunsch","year":"1997","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Schulman et al., 2015] John Schulman, Sergey Levine, Michael Jordan, and Pieter Abbeel. Trust Region Pol- icy Optimization. International Conference on Machine Learning, page 16, 2015.","title":"Trust Region Policy Optimization","authors":"John Schulman, Sergey Levine, Michael Jordan, Pieter Abbeel","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Silver et al., 2014] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic Policy Gradient Algorithms. Proceedings of the 31st International Conference on Machine Learning, pages 387-395, 2014.","title":"Deterministic Policy Gradient Algorithms","authors":"David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, Martin Riedmiller","year":"2014","venue":"Proceedings of the 31st International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Smith, 2005] Russell Smith. Open dynamics engine. 2005.","title":"Open dynamics engine","authors":"Russell Smith","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sutton et al., 1999] Richard S. Sutton, David Mcallester, Satinder Singh, and Yishay Mansour. Policy Gradient Methods for Reinforcement Learning with Function Ap- proximation. In Advances in Neural Information Process- ing Systems 12, pages 1057-1063, 1999.","title":"Policy Gradient Methods for Reinforcement Learning with Function Approximation","authors":"Richard S Sutton, David Mcallester, Satinder Singh, Yishay Mansour","year":"1999","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sutton, 1988] Richard S. Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3(1):9-44, 1988.","title":"Learning to predict by the methods of temporal differences","authors":"Richard S Sutton","year":"1988","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Van Hasselt and Wiering, 2007] Hado Van Hasselt and Marco A. Wiering. Reinforcement learning in continuous action spaces. In Proceedings of the IEEE Symposium on Approximate Dynamic Programming and Reinforcement Learning, pages 272-279, 2007.","title":"Reinforcement learning in continuous action spaces","authors":"Van Hasselt, Wiering ; Hado Van Hasselt, Marco A Wiering","year":"2007","venue":"Proceedings of the IEEE Symposium on Approximate Dynamic Programming and Reinforcement Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Zimmer et al., 2016] Matthieu Zimmer, Yann Boniface, and Alain Dutech. Neural Fitted Actor-Critic. In European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, 2016.","title":"Neural Fitted Actor-Critic","authors":"Matthieu Zimmer, Yann Boniface, Alain Dutech","year":"2016","venue":"European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zimmer et al., 2018] Matthieu Zimmer, Yann Boniface, and Alain Dutech. Developmental reinforcement learning through sensorimotor space enlargement. In The 8th Joint IEEE International Conference on Development and Learning and on Epigenetic Robotics, September 2018.","title":"Developmental reinforcement learning through sensorimotor space enlargement","authors":"Matthieu Zimmer, Yann Boniface, Alain Dutech","year":"2018","venue":"The 8th Joint IEEE International Conference on Development and Learning and on Epigenetic Robotics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}