{"type":"ScholarlyArticle","title":"Towards Applicable Reinforcement Learning: Improving the Generalization and Sample Efficiency with Policy Ensemble","authors":[{"name":"Zhengyu Yang","slug":"zhengyu-yang","given":"Zhengyu","family":"Yang","affiliation":"Shanghai Jiao Tong University"},{"name":"Kan Ren","slug":"kan-ren","given":"Kan","family":"Ren","affiliation":"Microsoft Research"},{"name":"Xufang Luo","slug":"xufang-luo","given":"Xufang","family":"Luo","affiliation":"Microsoft Research"},{"name":"Minghuan Liu","slug":"minghuan-liu","given":"Minghuan","family":"Liu","affiliation":"Shanghai Jiao Tong University"},{"name":"Weiqing Liu","slug":"weiqing-liu","given":"Weiqing","family":"Liu","affiliation":"Microsoft Research"},{"name":"Jiang Bian","slug":"jiang-bian","given":"Jiang","family":"Bian","affiliation":"Microsoft Research"},{"name":"Weinan Zhang","slug":"weinan-zhang","given":"Weinan","family":"Zhang","affiliation":"Shanghai Jiao Tong University"},{"name":"Dongsheng Li","slug":"dongsheng-li","given":"Dongsheng","family":"Li","affiliation":"Microsoft Research"}],"abstract":"It is challenging for reinforcement learning (RL) algorithms to succeed in real-world applications. Take financial trading as an example, the market information is noisy yet imperfect and the macroeconomic regulation or other factors may shift between training and evaluation, thus it requires both generalization and high sample efficiency for resolving the task. However, directly applying typical RL algorithms can lead to poor performance in such scenarios. To derive a robust and applicable RL algorithm, in this work, we design a simple but effective method named Ensemble Proximal Policy Optimization (EPPO), which learns ensemble policies in an end-to-end manner. Notably, EPPO combines each policy and the policy ensemble organically and optimizes both simultaneously. In addition, EPPO adopts a diversity enhancement regularization over the policy space which helps to generalize to unseen states and promotes exploration. We theoretically prove that EPPO can increase exploration efficacy, and through comprehensive experimental evaluations on various tasks, we demonstrate that EPPO achieves higher efficiency and is robust for real-world applications compared with vanilla policy optimization algorithms and other ensemble methods. Code and supplemental materials are available at https://seqml.github.io/eppo.","doi":"10.24963/ijcai.2022/508","url":"https://ijcai.org/proceedings/2022/508","pdf":"https://ijcai.org/proceedings/2022/0508.pdf","markdown":"https://ijcai.org/proceedings/2022/508.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/508","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":508,"pages":{"first":3659,"last":3665},"keywords":["Machine Learning: Deep Reinforcement Learning","Machine Learning: Ensemble Methods"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986805","length_sec":90,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984948","length_sec":329,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Anschel et al., 2017] Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dqn: Variance reduction and stabiliza- tion for deep reinforcement learning. In ICML, pages 176- 185. PMLR, 2017.","title":"Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning","authors":"Anschel","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellemare et al., 2013] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. JAIR, 47:253-279, 2013.","title":"The arcade learning environment: An evaluation platform for general agents","authors":"Yavar Marc G Bellemare, Joel Naddaf, Michael Veness, Bowling","year":"2013","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chevalier-Boisvert et al., 2018] Maxime Chevalier- Boisvert, Lucas Willems, and Suman Pal. Min- imalistic gridworld environment for openai gym. https://github.com/maximecb/gym-minigrid, 2018. Accessed: 2021-08-30.","title":"Minimalistic gridworld environment for openai gym","authors":"Maxime Chevalier-Boisvert, Lucas Willems, Suman Pal","year":"2021","venue":null,"doi":null,"url":"https://github.com/maximecb/gym-minigrid,2018","internal_url":null},{"position":4,"raw":"[Chua et al., 2018] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforce- ment learning in a handful of trials using probabilistic dynamics models. In NeurIPS, pages 4759-4770, 2018.","title":"Deep reinforcement learning in a handful of trials using probabilistic dynamics models","authors":"Chua","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Duell and Udluft, 2013] Siegmund Duell and Steffen Ud- luft. Ensembles for continuous actions in reinforcement learning. In ESANN. Citeseer, 2013.","title":"Ensembles for continuous actions in reinforcement learning","authors":"Udluft Duell, Siegmund Duell, Steffen Udluft","year":"2013","venue":"ESANN. Citeseer","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fang et al., 2021] Yuchen Fang, Kan Ren, Weiqing Liu, Dong Zhou, Weinan Zhang, Jiang Bian, Yong Yu, and Tie- Yan Liu. Universal trading for order execution with ora- cle policy distillation. arXiv preprint arXiv:2103.10860, 2021.","title":"Universal trading for order execution with oracle policy distillation","authors":"Yuchen Fang, Weiqing Kan Ren, Dong Liu, Weinan Zhou, Jiang Zhang, Yong Bian, Tie-Yan Yu, Liu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Faußer and Schwenker, 2015] Stefan Faußer and Friedhelm Schwenker. Neural network ensembles in reinforcement learning. Neural Processing Letters, 41(1):55-69, 2015.","title":"Neural network ensembles in reinforcement learning","authors":"Schwenker Faußer, Stefan Faußer, Friedhelm Schwenker","year":"2015","venue":"Neural Processing Letters","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ghosh et al., 2017] Dibya Ghosh, Avi Singh, Aravind Ra- jeswaran, Vikash Kumar, and Sergey Levine. Divide- and-conquer reinforcement learning. arXiv preprint arXiv:1711.09874, 2017.","title":"Divideand-conquer reinforcement learning","authors":"Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, Sergey Levine","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goyal et al., 2019] Anirudh Goyal, Shagun Sodhani, Jonathan Binas, Xue Bin Peng, Sergey Levine, and Yoshua Bengio. Reinforcement learning with competitive ensembles of information-constrained primitives. arXiv preprint arXiv:1906.10667, 2019.","title":"Reinforcement learning with competitive ensembles of information-constrained primitives","authors":"Anirudh Goyal, Shagun Sodhani, Jonathan Binas, Xue Bin Peng, Sergey Levine, Yoshua Bengio","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hong et al., 2018] Zhang-Wei Hong, Tzu-Yun Shann, Shih- Yang Su, Yi-Hsiang Chang, and Chun-Yi Lee. Diversity- driven exploration strategy for deep reinforcement learn- ing. arXiv preprint arXiv:1802.04564, 2018.","title":"Diversitydriven exploration strategy for deep reinforcement learning","authors":"Zhang-Wei Hong, Tzu-Yun Shann, Shih-Yang Su, Yi-Hsiang Chang, Chun-Yi Lee","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jacobs et al., 1991] Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mix- tures of local experts. Neural computation, 3(1):79-87, 1991.","title":"Adaptive mixtures of local experts","authors":"Robert A Jacobs, Michael I Jordan, Steven J Nowlan, Geoffrey E Hinton","year":"1991","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kurutach et al., 2018] Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model- ensemble trust-region policy optimization. arXiv preprint arXiv:1802.10592, 2018.","title":"Modelensemble trust-region policy optimization","authors":"Kurutach","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lee et al., 2021] Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In ICML, pages 6131-6141. PMLR, 2021.","title":"Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning","authors":"Lee","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2012] Nan Li, Yang Yu, and Zhi-Hua Zhou. Diver- sity regularized ensemble pruning. In ECMLPKDD, pages 330-345. Springer, 2012.","title":"Diversity regularized ensemble pruning","authors":"Nan Li, Yang Yu, Zhi-Hua Zhou","year":"2012","venue":"ECMLPKDD","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2020] Junjie Li, Sotetsu Koyamada, Qiwei Ye, Guoqing Liu, Chao Wang, Ruihan Yang, Li Zhao, Tao Qin, Tie-Yan Liu, and Hsiao-Wuen Hon. Suphx: Mas- tering mahjong with deep reinforcement learning. arXiv preprint arXiv:2003.13590, 2020.","title":"Suphx: Mastering mahjong with deep reinforcement learning","authors":"Junjie Li, Sotetsu Koyamada, Qiwei Ye, Guoqing Liu, Chao Wang, Ruihan Yang, Li Zhao, Tao Qin, Tie-Yan Liu, Hsiao-Wuen Hon","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Masood and Doshi-Velez, 2019] Muhammad A Masood and Finale Doshi-Velez. Diversity-inducing policy gra- dient: Using maximum mean discrepancy to find a set of diverse policies. arXiv preprint arXiv:1906.00088, 2019.","title":"Muhammad A Masood and Finale Doshi-Velez. Diversity-inducing policy gradient: Using maximum mean discrepancy to find a set of diverse policies","authors":"Doshi-Velez Masood","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Parker-Holder et al., 2020] Jack Parker-Holder, Aldo Pac- chiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective diversity in population based reinforcement learning. NeurIPS, 33:18050-18062, 2020.","title":"Effective diversity in population based reinforcement learning","authors":"Parker-Holder","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ren et al., 2021] Jie Ren, Yewen Li, Zihan Ding, Wei Pan, and Hao Dong. Probabilistic mixture-of-experts for efficient deep reinforcement learning. arXiv preprint arXiv:2104.09122, 2021.","title":"Probabilistic mixture-of-experts for efficient deep reinforcement learning","authors":"Jie Ren, Yewen Li, Zihan Ding, Wei Pan, Hao Dong","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Saphal et al., 2020] Rohan Saphal, Balaraman Ravindran, Dheevatsa Mudigere, Sasikanth Avancha, and Bharat Kaul. Seerl: Sample efficient ensemble reinforcement learning. arXiv preprint arXiv:2001.05209, 2020.","title":"Seerl: Sample efficient ensemble reinforcement learning","authors":"Rohan Saphal, Balaraman Ravindran, Dheevatsa Mudigere, Sasikanth Avancha, Bharat Kaul","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schulman et al., 2015] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advan- tage estimation. arXiv preprint arXiv:1506.02438, 2015.","title":"Highdimensional continuous control using generalized advantage estimation","authors":"John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Prox- imal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wiering and Van Hasselt, 2008] Marco A Wiering and Hado Van Hasselt. Ensemble algorithms in reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 38(4):930-936, 2008.","title":"Ensemble algorithms in reinforcement learning","authors":"A Marco, Hado Wiering, Van Hasselt","year":"2008","venue":"IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2021] Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08140, 2021.","title":"Uncertainty weighted actor-critic for offline reinforcement learning","authors":"Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, Hanlin Goh","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yang et al., 2022] Guan Yang, Minghuan Liu, Weijun Hong, Weinan Zhang, Fei Fang, Guangjun Zeng, and Yue Lin. Perfectdou: Dominating doudizhu with perfect in- formation distillation. arXiv preprint arXiv:2203.16406, 2022.","title":"Perfectdou: Dominating doudizhu with perfect information distillation","authors":"Guan Yang, Minghuan Liu, Weijun Hong, Weinan Zhang, Fei Fang, Guangjun Zeng, Yue Lin","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ye et al., 2020] Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, et al. Mastering complex control in moba games with deep reinforcement learning. AAAI, 34, 2020.","title":"Mastering complex control in moba games with deep reinforcement learning","authors":"Ye","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2020] Shaofeng Zhang, Meng Liu, and Junchi Yan. The diversified ensemble neural network. NeurIPS, 33, 2020.","title":"The diversified ensemble neural network","authors":"Shaofeng Zhang, Meng Liu, Junchi Yan","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhou et al., 2002] Zhi-Hua Zhou, Jianxin Wu, and Wei Tang. Ensembling neural networks: many could be better than all. Artificial intelligence, 137(1-2):239-263, 2002.","title":"Ensembling neural networks: many could be better than all","authors":"Zhi-Hua Zhou, Jianxin Wu, Wei Tang","year":"2002","venue":"Artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou et al., 2018] Tianyi Zhou, Shengjie Wang, and Jeff A Bilmes. Diverse ensemble evolution: Curriculum data- model marriage. In NeurIPS, pages 5909-5920, 2018.","title":"Diverse ensemble evolution: Curriculum datamodel marriage","authors":"Tianyi Zhou, Shengjie Wang, Jeff A Bilmes","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}