{"type":"ScholarlyArticle","title":"Self-Adaptive Double Bootstrapped DDPG","authors":[{"name":"Zhuobin Zheng","slug":"zhuobin-zheng","given":"Zhuobin","family":"Zheng","affiliation":"Department of Computer Science and Technologies , Tsinghua University"},{"name":"Chun Yuan","slug":"chun-yuan","given":"Chun","family":"Yuan","affiliation":"Graduate School at Shenzhen , Tsinghua University {zhengzb16 , lin-zh14 ,"},{"name":"Zhihui Lin","slug":"zhihui-lin","given":"Zhihui","family":"Lin","affiliation":"Department of Computer Science and Technologies , Tsinghua University"},{"name":"Yangyang Cheng","slug":"yangyang-cheng","given":"Yangyang","family":"Cheng","affiliation":"Department of Computer Science and Technologies , Tsinghua University"},{"name":"Hanghao Wu","slug":"hanghao-wu","given":"Hanghao","family":"Wu","affiliation":"Department of Computer Science and Technologies , Tsinghua University"}],"abstract":"Deep Deterministic Policy Gradient (DDPG) algorithm has been successful for state-of-the-art performance in high-dimensional continuous control tasks. However, due to the complexity and randomness of the environment, DDPG tends to suffer from inefficient exploration and unstable training. In this work, we propose Self-Adaptive Double Bootstrapped DDPG (SOUP), an algorithm that extends DDPG to bootstrapped actor-critic architecture. SOUP improves the efficiency of exploration by multiple actor heads capturing more potential actions and multiple critic heads evaluating more reasonable Q-values collaboratively. The crux of double bootstrapped architecture is to tackle the fluctuations in performance, caused by multiple heads of spotty capacity varying throughout training. To alleviate the instability, a self-adaptive confidence mechanism is introduced to dynamically adjust the weights of bootstrapped heads and enhance the ensemble performance effectively and efficiently. We demonstrate that SOUP achieves faster learning by at least 45% while improving cumulative reward and stability substantially in comparison to vanilla DDPG on OpenAI Gym's MuJoCo environments.","doi":"10.24963/ijcai.2018/444","url":"https://ijcai.org/proceedings/2018/444","pdf":"https://ijcai.org/proceedings/2018/0444.pdf","markdown":"https://ijcai.org/proceedings/2018/444.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/444","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":444,"pages":{"first":3198,"last":3204},"keywords":["Machine Learning: Reinforcement Learning","Machine Learning Applications: Applications of Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Brockman et al., 2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Woj- ciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.","title":"Openai gym","authors":"Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2018] Richard Y. Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. UCB EXPLORATION VIA q- ENSEMBLES, 2018.","title":"UCB EXPLORATION VIA q-ENSEMBLES","authors":"Richard Y Chen, Szymon Sidor, Pieter Abbeel, John Schulman","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Efron and Tibshirani, 1994] Bradley Efron and Robert J Tibshi- rani. An introduction to the bootstrap. CRC press, 1994.","title":"Bradley Efron and Robert J Tibshirani. An introduction to the bootstrap","authors":"Tibshirani Efron","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gu et al., 2017] Shixiang Gu, Timothy Lillicrap, Zoubin Ghahra- mani, Richard E. Turner, and Sergey Levine. Q-prop: Sample- efficient policy gradient with an off-policy critic. In Proceedings of International Conference on Learning Representations, 2017.","title":"Q-prop: Sampleefficient policy gradient with an off-policy critic","authors":"Gu","year":"2017","venue":"Proceedings of International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Henderson et al., 2017] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. arXiv preprint arXiv:1709.06560, 2017.","title":"Deep reinforcement learning that matters","authors":"Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, David Meger","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Islam et al., 2017] Riashat Islam, Peter Henderson, Maziar Gom- rokchi, and Doina Precup. Reproducibility of benchmarked deep reinforcement learning tasks for continuous control. arXiv preprint arXiv:1708.04133, 2017.","title":"Reproducibility of benchmarked deep reinforcement learning tasks for continuous control","authors":"Islam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jaderberg et al., 2017] Max Jaderberg, Volodymyr Mnih, Woj- ciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsuper- vised auxiliary tasks. In Proceedings of International Conference on Learning Representations, 2017.","title":"Reinforcement learning with unsupervised auxiliary tasks","authors":"Jaderberg","year":"2017","venue":"Proceedings of International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kalweit and Boedecker, 2017] Gabriel Kalweit and Joschka Boedecker. Uncertainty-driven imagination for continuous deep reinforcement learning. In Conference on Robot Learning, pages 195-206, 2017.","title":"Uncertainty-driven imagination for continuous deep reinforcement learning","authors":"Boedecker Kalweit, Gabriel Kalweit, Joschka Boedecker","year":"2017","venue":"Conference on Robot Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kingma and Ba, 2015] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of Interna- tional Conference on Learning Representations, 2015.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2015","venue":"Proceedings of International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Konda and Tsitsiklis, 2000] Vijay R Konda and John N Tsitsiklis. Actor-critic algorithms. In Advances in neural information pro- cessing systems, pages 1008-1014, 2000.","title":"Actor-critic algorithms","authors":"R Vijay, John N Konda, Tsitsiklis","year":"2000","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Levine et al., 2016] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor poli- cies. Journal of Machine Learning Research, 17(39):1-40, 2016.","title":"End-to-end training of deep visuomotor policies","authors":"Sergey Levine, Chelsea Finn, Trevor Darrell, Pieter Abbeel","year":"2016","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lillicrap et al., 2016] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep rein- forcement learning. In Proceedings of International Conference on Learning Representations, 2016.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2016","venue":"Proceedings of International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Maas et al., 2013] Andrew L Maas, Awni Y Hannun, and An- drew Y Ng. Rectifier nonlinearities improve neural network acoustic models. In Proc. ICML, volume 30, 2013.","title":"Rectifier nonlinearities improve neural network acoustic models","authors":"Awni Y Andrew L Maas, Andrew Y Hannun, Ng","year":"2013","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih et al., 2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostro- vski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.","title":"Human-level control through deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, pages 1928-1937, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Osband et al., 2016] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via boot- strapped dqn. In Advances in Neural Information Processing Systems, pages 4026-4034, 2016.","title":"Deep exploration via bootstrapped dqn","authors":"Ian Osband, Charles Blundell, Alexander Pritzel, Benjamin Van, Roy","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pathak et al., 2017] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self- supervised prediction. In ICML, 2017.","title":"Curiosity-driven exploration by selfsupervised prediction","authors":"Deepak Pathak, Pulkit Agrawal, Alexei A Efros, Trevor Darrell","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Plappert et al., 2018] Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim As- four, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. In International Conference on Learning Representations, 2018.","title":"Parameter space noise for exploration","authors":"Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, Marcin Andrychowicz","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Silver et al., 2014] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. De- terministic policy gradient algorithms. In Proceedings of the 31st International Conference on Machine Learning (ICML-14), pages 387-395, 2014.","title":"Deterministic policy gradient algorithms","authors":"David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, Martin Riedmiller","year":"2014","venue":"Proceedings of the 31st International Conference on Machine Learning (ICML-14)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Silver et al., 2017] David Silver, Julian Schrittwieser, Karen Si- monyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mas- tering the game of go without human knowledge. Nature, 550(7676):354, 2017.","title":"Mastering the game of go without human knowledge","authors":"David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton","year":"2017","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sutton and Barto, 1998] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.","title":"Reinforcement learning: An introduction","authors":"S Richard, Andrew G Sutton, Barto","year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sutton et al., 2000] Richard S Sutton, David A McAllester, Satin- der P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Ad- vances in neural information processing systems, pages 1057- 1063, 2000.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"David A Richard S Sutton, Mcallester, P Satinder, Yishay Singh, Mansour","year":"2000","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Todorov et al., 2012] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ Interna- tional Conference on, pages 5026-5033. IEEE, 2012.","title":"Mujoco: A physics engine for model-based control","authors":"Emanuel Todorov, Tom Erez, Yuval Tassa","year":"2012","venue":"Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256, 1992.","title":"Simple statistical gradientfollowing algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2017] Zhaoyang Yang, Kathryn Merrick, Hussein Ab- bass, and Lianwen Jin. Multi-task deep reinforcement learning for continuous action control. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI- 17, pages 3301-3307, 2017.","title":"Multi-task deep reinforcement learning for continuous action control","authors":"Zhaoyang Yang, Kathryn Merrick, Hussein Abbass, Lianwen Jin","year":"2017","venue":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17","doi":null,"url":null,"internal_url":"/proceedings/2017/461"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}