{"type":"ScholarlyArticle","title":"Ensemble Reinforcement Learning in Continuous Spaces -- A Hierarchical Multi-Step Approach for Policy Training","authors":[{"name":"Gang Chen","slug":"gang-chen","given":"Gang","family":"Chen","affiliation":"School of Engineering and Computer Science , Victoria University of Wellington , New Zealand"},{"name":"Victoria Huang","slug":"victoria-huang","given":"Victoria","family":"Huang","affiliation":"National Institute of Water and Atmospheric Research , New Zealand"}],"abstract":"Actor-critic deep reinforcement learning (DRL) algorithms have recently achieved prominent success in tackling various challenging reinforcement learning (RL) problems, particularly complex control tasks with high-dimensional continuous state and action spaces. Nevertheless, existing research showed that actor-critic DRL algorithms often failed to explore their learning environments effectively, resulting in limited learning stability and performance. To address this limitation, several ensemble DRL algorithms have been proposed lately to boost exploration and stabilize the learning process. However, most of existing ensemble algorithms do not explicitly train all base learners towards jointly optimizing the performance of the ensemble. In this paper, we propose a new technique to train an ensemble of base learners based on an innovative multi-step integration method. This training technique enables us to develop a new hierarchical learning algorithm for ensemble DRL that effectively promotes inter-learner collaboration through stable inter-learner parameter sharing. The design of our new algorithm is verified theoretically. The algorithm is also shown empirically to outperform several state-of-the-art DRL algorithms on multiple benchmark RL problems.","doi":"10.24963/ijcai.2023/391","url":"https://ijcai.org/proceedings/2023/391","pdf":"https://ijcai.org/proceedings/2023/0391.pdf","markdown":"https://ijcai.org/proceedings/2023/391.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/391","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":391,"pages":{"first":3514,"last":3522},"keywords":["Machine Learning: ML: Deep reinforcement learning","Machine Learning: ML: Ensemble methods"],"videos":[],"references":[{"position":1,"raw":"[Achiam, 2018] J. Achiam. Spinning Up in Deep Reinforce- ment Learning. https://github.com/openai/spinningup, 2018. Accessed: 2022-12-20.","title":"Spinning Up in Deep Reinforcement Learning","authors":"J Achiam","year":"2022","venue":null,"doi":null,"url":"https://github.com/openai/spinningup,2018","internal_url":null},{"position":2,"raw":"[An et al., 2021] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline rein- forcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.","title":"Uncertainty-based offline reinforcement learning with diversified q-ensemble","authors":"Seungyong Gaon An, Jang-Hyun Moon, Hyun Oh Kim, Song","year":"2021","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baek et al., 2020] J. Baek, H. Jun, J. Park, H. Lee, and S. Han. Sparse variational deterministic policy gradient for continuous real time control. IEEE Transactions on Industrial Electronics, 2020.","title":"Sparse variational deterministic policy gradient for continuous real time control","authors":"J Baek, H Jun, J Park, H Lee, S Han","year":"2020","venue":"IEEE Transactions on Industrial Electronics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Barth-Maron et al., 2018] G. Barth-Maron, M. W. Hoff- man, D. Budden, W. Dabney, D. Horgan, D. Tb, A. Mul- dal, N. Heess, and T. Lillicrap. Distributed distribu- tional deterministic policy gradients. arXiv preprint arXiv:1804.08617, 2018.","title":"Distributed distributional deterministic policy gradients","authors":"G Barth-Maron, M W Hoffman, D Budden, W Dabney, D Horgan, D Tb, A Muldal, N Heess, T Lillicrap","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bellemare et al., 2016] M. Bellemare, S. Srinivasan, G. Os- trovski, T. Schaul, D. Saxton, and R. Munos. Uni- fying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29:1471-1479, 2016.","title":"Unifying count-based exploration and intrinsic motivation","authors":"M Bellemare, S Srinivasan, G Ostrovski, T Schaul, D Saxton, R Munos","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Brockman et al., 2016] G. Brockman, V. Cheung, L. Pet- tersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv:1606.01540, 2016.","title":"Openai gym","authors":"G Brockman, V Cheung, L Pettersson, J Schneider, J Schulman, J Tang, W Zaremba","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chan et al., 2019] C. Y. S. Chan, S. Fishman, J. Canny, A. Korattikara, and S. Guadarrama. Measuring the relia- bility of reinforcement learning algorithms. arXiv preprint arXiv:1912.05663, 2019.","title":"Measuring the reliability of reinforcement learning algorithms","authors":"C Y S Chan, S Fishman, J Canny, A Korattikara, S Guadarrama","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2017] R. Y. Chen, S. Sidor, P. Abbeel, and J. Schulman. Ucb exploration via q-ensembles. arXiv preprint arXiv:1706.01502, 2017.","title":"Ucb exploration via q-ensembles","authors":"R Y Chen, S Sidor, P Abbeel, J Schulman","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2018] G. Chen, Y. Peng, and M. Zhang. Effec- tive exploration for deep reinforcement learning via boot- strapped q-ensembles under tsallis entropy regularization. arXiv preprint arXiv:1809.00403, 2018.","title":"Effective exploration for deep reinforcement learning via bootstrapped q-ensembles under tsallis entropy regularization","authors":"G Chen, Y Peng, M Zhang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Cobbe et al., 2021] K. W. Cobbe, J. Hilton, O. Klimov, and J. Schulman. Phasic policy gradient. In International Con- ference on Machine Learning, pages 2020-2027. PMLR, 2021.","title":"Phasic policy gradient","authors":"K W Cobbe, J Hilton, O Klimov, J Schulman","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"Ellenberger, 2018 2019]","title":null,"authors":"Ellenberger","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"B. Ellenberger. Pybullet gym- perium. https://github.com/benelot/pybullet-gym, 2018- 2019. Accessed: 2022-12-20.","title":"Pybullet gymperium","authors":"B Ellenberger","year":"2022","venue":null,"doi":null,"url":"https://github.com/benelot/pybullet-gym,2018-2019","internal_url":null},{"position":13,"raw":"[Fujimoto et al., 2018] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learn- ing, pages 1587-1596. PMLR, 2018.","title":"Addressing function approximation error in actor-critic methods","authors":"S Fujimoto, H Hoof, D Meger","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Haarnoja et al., 2018] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In In- ternational conference on machine learning, pages 1861- 1870. PMLR, 2018.","title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","authors":"T Haarnoja, A Zhou, P Abbeel, S Levine","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hasselt et al., 2016] H. Van Hasselt, A. Guez, and D. Sil- ver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intel- ligence, volume 30, 2016.","title":"Deep reinforcement learning with double q-learning","authors":"H Van Hasselt, A Guez, D Silver","year":"2016","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Huang et al., 2017] Z. Huang, S. Zhou, B. Zhuang, and X. Zhou. Learning to run with actor-critic ensemble. arXiv preprint arXiv:1712.08987, 2017.","title":"Learning to run with actor-critic ensemble","authors":"Z Huang, S Zhou, B Zhuang, X Zhou","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ibarz et al., 2021] J. Ibarz, J. Tan, C. Finn, M. Kalakrishnan, P. Pastor, and S. Levine. How to train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Robotics Research, 40(4-5):698- 721, 2021.","title":"How to train your robot with deep reinforcement learning: lessons we have learned","authors":"J Ibarz, J Tan, C Finn, M Kalakrishnan, P Pastor, S Levine","year":"2021","venue":"The International Journal of Robotics Research","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Januszewski et al., 2021] P. Januszewski, M. Olko, M. Królikowski, J. Światkowski, M. Andrychowicz, L. Kuciński, and P. Miloś. Continuous control with ensemble deep deterministic policy gradients. arXiv preprint arXiv:2111.15382, 2021.","title":"Continuous control with ensemble deep deterministic policy gradients","authors":"P Januszewski, M Olko, M Królikowski, J Światkowski, M Andrychowicz, L Kuciński, P Miloś","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Kurutach et al., 2018] T. Kurutach, I. Clavera, Y. Duan, A. Tamar, and P. Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv:1802.10592, 2018.","title":"Model-ensemble trust-region policy optimization","authors":"T Kurutach, I Clavera, Y Duan, A Tamar, P Abbeel","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lai et al., 2020] Kwei-Herng Lai, Daochen Zha, Yuening Li, and Xia Hu. Dual policy distillation. arXiv preprint arXiv:2006.04061, 2020.","title":"Dual policy distillation","authors":"Kwei-Herng Lai, Daochen Zha, Yuening Li, Xia Hu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lakshminarayanan et al., 2016] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. arXiv preprint arXiv:1612.01474, 2016.","title":"Simple and scalable predictive uncertainty estimation using deep ensembles","authors":"B Lakshminarayanan, A Pritzel, C Blundell","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lee et al., 2021] K. Lee, M. Laskin, A. Srinivas, and P. Abbeel. Sunrise: A simple unified framework for en- semble learning in deep reinforcement learning. In Inter- national Conference on Machine Learning, pages 6131- 6141. PMLR, 2021.","title":"Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning","authors":"K Lee, M Laskin, A Srinivas, P Abbeel","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Lillicrap et al., 2015] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wier- stra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"T P Lillicrap, J J Hunt, A Pritzel, N Heess, T Erez, Y Tassa, D Silver, D Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Liu et al., 2021] R. Liu, F. Nageotte, P. Zanne, M. de Math- elin, and B. Dresp-Langley. Deep reinforcement learning for the control of robotic manipulation: a focussed mini- review. Robotics, 10(1):22, 2021.","title":"Deep reinforcement learning for the control of robotic manipulation: a focussed minireview","authors":"R Liu, F Nageotte, P Zanne, M De Mathelin, B Dresp-Langley","year":"2021","venue":"Robotics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Mnih et al., 2016] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep rein- forcement learning. In International conference on ma- chine learning, pages 1928-1937. PMLR, 2016.","title":"Asynchronous methods for deep reinforcement learning","authors":"V Mnih, A P Badia, M Mirza, A Graves, T Lillicrap, T Harley, D Silver, K Kavukcuoglu","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Osband and Roy, 2017] I. Osband and B. Van Roy. Why is posterior sampling better than optimism for reinforcement learning? In International conference on machine learn- ing, pages 2701-2710. PMLR, 2017.","title":"Why is posterior sampling better than optimism for reinforcement learning?","authors":"Roy ; Osband, I Osband, B Van Roy","year":"2017","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Osband et al., 2016] I.","title":null,"authors":"Osband","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29:4026-4034, 2016.","title":"Deep exploration via bootstrapped dqn","authors":"C Osband, A Blundell, B Pritzel, Van Roy","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Osband et al., 2018] I.","title":null,"authors":"Osband","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"Osband, J. Aslanides, and A. Cas- sirer. Randomized prior functions for deep reinforcement learning. arXiv preprint arXiv:1806.03335, 2018.","title":"Randomized prior functions for deep reinforcement learning","authors":"J Osband, A Aslanides, Cassirer","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Paine et al., 2020] T. L. Paine, C. Paduraru, A. Michi, C. Gulcehre, K. Zolna, A. Novikov, Z. Wang, and N. de Freitas. Hyperparameter selection for offline re- inforcement learning. arXiv preprint arXiv:2007.09055, 2020.","title":"Hyperparameter selection for offline reinforcement learning","authors":"T L Paine, C Paduraru, A Michi, C Gulcehre, K Zolna, A Novikov, Z Wang, N De Freitas","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Plappert et al., 2017] M. Plappert, R. Houthooft, P. Dhari- wal, S. Sidor, R. Y. Chen, X. Chen, T. Asfour, P. Abbeel, and M. Andrychowicz. Parameter space noise for explo- ration. arXiv preprint arXiv:1706.01905, 2017.","title":"Parameter space noise for exploration","authors":"M Plappert, R Houthooft, P Dhariwal, S Sidor, R Y Chen, X Chen, T Asfour, P Abbeel, M Andrychowicz","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Reizinger and Szemenyei, 2020] P. Reizinger and M. Sze- menyei. Attention-based curiosity-driven exploration in deep reinforcement learning. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3542-3546. IEEE, 2020.","title":"Attention-based curiosity-driven exploration in deep reinforcement learning","authors":"P Reizinger, M Szemenyei","year":"2020","venue":"ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Schulman et al., 2015] J. Schulman, N. Heess, T. Weber, and P. Abbeel. Gradient estimation using stochastic com- putation graphs. Advances in Neural Information Process- ing Systems, 28:3528-3536, 2015.","title":"Gradient estimation using stochastic computation graphs","authors":"J Schulman, N Heess, T Weber, P Abbeel","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Schulman et al., 2017] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"J Schulman, F Wolski, P Dhariwal, A Radford, O Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Scieur et al., 2017] D. Scieur, V. Roulet, F. Bach, and A. d'Aspremont. Integration methods and accelerated op- timization algorithms. arXiv preprint arXiv:1702.06751, 2017.","title":"Integration methods and accelerated optimization algorithms","authors":"D Scieur, V Roulet, F Bach, A","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Shani et al., 2020] L. Shani, Y. Efroni, and S. Mannor. Adaptive trust region policy optimization: Global conver- gence and faster rates for regularized mdps. In Proceed- ings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pages 5668-5675, 2020.","title":"Adaptive trust region policy optimization: Global convergence and faster rates for regularized mdps","authors":"L Shani, Y Efroni, S Mannor","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Silver et al., 2014] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gra- dient algorithms. In International conference on machine learning, pages 387-395. PMLR, 2014.","title":"Deterministic policy gradient algorithms","authors":"D Silver, G Lever, N Heess, T Degris, D Wierstra, M Riedmiller","year":"2014","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Wang et al., 2020] C. Wang, Y. Wu, Q. Vuong, and K. Ross. Striving for simplicity and performance in off-policy drl: Output normalization and non-uniform sampling. In Inter- national Conference on Machine Learning, pages 10070- 10080. PMLR, 2020.","title":"Striving for simplicity and performance in off-policy drl: Output normalization and non-uniform sampling","authors":"C Wang, Y Wu, Q Vuong, K Ross","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Wiering and Hasselt, 2008] M. A. Wiering and H. Van Has- selt. Ensemble algorithms in reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 38(4):930-936, 2008.","title":"Ensemble algorithms in reinforcement learning","authors":"M A Wiering, H Van","year":"2008","venue":"IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Wu et al., 2017] Y. Wu, E. Mansimov, R. B. Grosse, S. Liao, and J. Ba. Scalable trust-region method for deep reinforcement learning using kronecker-factored approxi- mation. Advances in neural information processing sys- tems, 30:5279-5288, 2017.","title":"Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation","authors":"Y Wu, E Mansimov, R B Grosse, S Liao, J Ba","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Zhelo et al., 2018] O. Zhelo, J. Zhang, L. Tai, M. Liu, and W. Burgard. Curiosity-driven exploration for mapless nav- igation with deep reinforcement learning. arXiv preprint arXiv:1804.00456, 2018.","title":"Curiosity-driven exploration for mapless navigation with deep reinforcement learning","authors":"O Zhelo, J Zhang, L Tai, M Liu, W Burgard","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}