{"type":"ScholarlyArticle","title":"JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learning","authors":[{"name":"Zichuan Lin","slug":"zichuan-lin","given":"Zichuan","family":"Lin","affiliation":"Tencent AI Lab , Shenzhen , China"},{"name":"Junyou Li","slug":"junyou-li","given":"Junyou","family":"Li","affiliation":"Tencent AI Lab , Shenzhen , China"},{"name":"Jianing Shi","slug":"jianing-shi","given":"Jianing","family":"Shi","affiliation":"Tencent AI Lab , Shenzhen , China"},{"name":"Deheng Ye","slug":"deheng-ye","given":"Deheng","family":"Ye","affiliation":"Tencent AI Lab , Shenzhen , China"},{"name":"Qiang Fu","slug":"qiang-fu","given":"Qiang","family":"Fu","affiliation":"Tencent AI Lab , Shenzhen , China"},{"name":"Wei Yang","slug":"wei-yang","given":"Wei","family":"Yang","affiliation":"Tencent AI Lab , Shenzhen , China"}],"abstract":"Learning rational behaviors in open-world games like Minecraft remains to be challenging for Reinforcement Learning (RL) research due to the compound challenge of partial observability, high-dimensional visual perception and delayed reward. To address this, we propose JueWu-MC, a sample-efficient hierarchical RL approach equipped with representation learning and imitation learning to deal with perception and exploration. Specifically, our approach includes two levels of hierarchy, where the high-level controller learns a policy to control over options and the low-level workers learn to solve each sub-task. To boost the learning of sub-tasks, we propose a combination of techniques including 1) action-aware representation learning which captures underlying relations between action and representation, 2) discriminator-based self-imitation learning for efficient exploration, and 3) ensemble behavior cloning with consistency filtering for policy robustness. Extensive experiments show that JueWu-MC significantly improves sample efficiency and outperforms a set of baselines by a large margin. Notably, we won the championship of the NeurIPS MineRL 2021 research competition and achieved the highest performance score ever.","doi":"10.24963/ijcai.2022/452","url":"https://ijcai.org/proceedings/2022/452","pdf":"https://ijcai.org/proceedings/2022/0452.pdf","markdown":"https://ijcai.org/proceedings/2022/452.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/452","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":452,"pages":{"first":3257,"last":3263},"keywords":["Search: Game Playing","Machine Learning: Deep Reinforcement Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986978","length_sec":77,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985121","length_sec":706,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Amiranashvili et al., 2020] Artemij Amiranashvili, Nicolai Dorka, et al. Scaling imitation learning in minecraft. arXiv:2007.02701, 2020.","title":"Scaling imitation learning in minecraft","authors":"Amiranashvili","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Berner et al., 2019] Christopher Berner, Greg Brockman, et al. Dota 2 with large scale deep reinforcement learn- ing. arXiv:1912.06680, 2019.","title":"Dota 2 with large scale deep reinforcement learning","authors":"Christopher Berner, Greg Brockman","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2021] Sheng Chen, Menghui Zhu, et al. Which heroes to pick? learning to draft in moba games with neural networks and tree search. IEEE Trans. on Games, 2021.","title":"Which heroes to pick? learning to draft in moba games with neural networks and tree search","authors":"Sheng Chen, Menghui Zhu","year":"2021","venue":"IEEE Trans. on Games","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Guss et al., 2019] William H. Guss, Brandon Houghton, et al. Minerl: A large-scale dataset of minecraft demonstra- tions. In IJCAI, 2019.","title":"Minerl: A large-scale dataset of minecraft demonstrations","authors":"H William, Brandon Guss, Houghton","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/339"},{"position":5,"raw":"[Guss et al., 2021] William H Guss, Mario Castro, et al. The minerl 2020 competition on sample efficient reinforcement learning using human priors. arXiv:2101.11071, 2021.","title":"The minerl 2020 competition on sample efficient reinforcement learning using human priors","authors":"Mario William H Guss, Castro","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hessel et al., 2018] Matteo Hessel, Joseph Modayil, et al. Rainbow: Combining improvements in deep reinforcement learning. In AAAI, 2018.","title":"Rainbow: Combining improvements in deep reinforcement learning","authors":"Matteo Hessel, Joseph Modayil","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hester et al., 2018] Todd Hester, Matej Vecerik, et al. Deep q-learning from demonstrations. In AAAI, 2018.","title":"Deep q-learning from demonstrations","authors":"Todd Hester, Matej Vecerik","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2019] Shiyu Huang, Hang Su, et al. Combo- action: Training agent for fps game with auxiliary tasks. In AAAI, 2019.","title":"Comboaction: Training agent for fps game with auxiliary tasks","authors":"Shiyu Huang, Hang Su","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2021] Shiyu Huang, Wenze Chen, et al. Ti- kick: Towards playing multi-agent football full games from single-agent demonstrations. arXiv:2110.04507, 2021.","title":"Tikick: Towards playing multi-agent football full games from single-agent demonstrations","authors":"Shiyu Huang, Wenze Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kanervisto et al., 2020] Anssi Kanervisto, Janne Karttunen, and Ville Hautamäki. Playing minecraft with behavioural cloning. In NeurIPS 2019 Competition and Demonstration Track, 2020.","title":"Playing minecraft with behavioural cloning","authors":"Anssi Kanervisto, Janne Karttunen, Ville Hautamäki","year":"2020","venue":"NeurIPS 2019 Competition and Demonstration Track","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krishna and Murty, 1999] K Krishna and M Narasimha Murty. Genetic k-means algorithm. IEEE Trans. on Sys- tems, Man and Cyber., 1999.","title":"Genetic k-means algorithm","authors":"K Krishna, M Narasimha, Murty","year":"1999","venue":"IEEE Trans. on Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Le et al., 2018] Hoang Le, Nan Jiang, et al. Hierarchical imitation and reinforcement learning. In ICML, 2018.","title":"Hierarchical imitation and reinforcement learning","authors":"Hoang Le, Nan Jiang","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lesort et al., 2018] Timothée Lesort, Natalia Díaz- Rodríguez, et al. State representation learning for control: An overview. Neural Networks, 2018.","title":"State representation learning for control: An overview","authors":"Timothée Lesort, Natalia Díaz-Rodríguez","year":"2018","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mao et al., 2021] Hangyu Mao, Chao Wang, et al. Seihai: A sample-efficient hierarchical ai for the minerl competition. arXiv:2111.08857, 2021.","title":"Seihai: A sample-efficient hierarchical ai for the minerl competition","authors":"Mao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, et al. Playing atari with deep reinforcement learning. arXiv:1312.5602, 2013.","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nair et al., 2018] Ashvin Nair, Bob McGrew, et al. Over- coming exploration in reinforcement learning with demon- strations. In ICRA, 2018.","title":"Overcoming exploration in reinforcement learning with demonstrations","authors":"Ashvin Nair, Bob Mcgrew","year":"2018","venue":"ICRA","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Oh et al., 2016] Junhyuk Oh, Valliappa Chockalingam, et al. Control of memory, active perception, and action in minecraft. In ICML, 2016.","title":"Junhyuk Oh, Valliappa Chockalingam, et al. Control of memory, active perception, and action in minecraft","authors":"Oh","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Oh et al., 2018] Junhyuk Oh, Yijie Guo, et al. Self-imitation learning. In ICML, 2018.","title":"Self-imitation learning","authors":"Junhyuk Oh, Yijie Guo","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pathak et al., 2017] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In ICML, 2017.","title":"Curiosity-driven exploration by self-supervised prediction","authors":"Deepak Pathak, Pulkit Agrawal, Alexei A Efros, Trevor Darrell","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pertsch et al., 2020] Karl Pertsch, Youngwoon Lee, and Joseph J Lim. Accelerating reinforcement learning with learned skill priors. arXiv:2010.11944, 2020.","title":"Accelerating reinforcement learning with learned skill priors","authors":"Karl Pertsch, Youngwoon Lee, Joseph J Lim","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Reddy et al., 2019] Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv:1905.11108, 2019.","title":"Sqil: Imitation learning via reinforcement learning with sparse rewards","authors":"Siddharth Reddy, Anca D Dragan, Sergey Levine","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schaul et al., 2015] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience re- play. arXiv:1511.05952, 2015.","title":"Prioritized experience replay","authors":"Tom Schaul, John Quan, Ioannis Antonoglou, David Silver","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schulman et al., 2017] John Schulman, Filip Wolski, et al. Proximal policy optimization algorithms. arXiv:1707.06347, 2017.","title":"Proximal policy optimization algorithms","authors":"John Schulman, Filip Wolski","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Silver et al., 2016] David Silver, Aja Huang, et al. Mastering the game of go with deep neural networks and tree search. nature, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang","year":"2016","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Skrynnik et al., 2021] Alexey Skrynnik, Aleksey Staroverov, et al. Forgetful experience replay in hierarchical reinforce- ment learning from expert demonstrations. Knowledge- Based Systems, 2021.","title":"Forgetful experience replay in hierarchical reinforcement learning from expert demonstrations","authors":"Alexey Skrynnik, Aleksey Staroverov","year":"2021","venue":"Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Srinivas et al., 2020] Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. arXiv:2004.04136, 2020.","title":"Curl: Contrastive unsupervised representations for reinforcement learning","authors":"Aravind Srinivas, Michael Laskin, Pieter Abbeel","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Tessler et al., 2017] Chen Tessler, Shahar Givony, et al. A deep hierarchical approach to lifelong learning in minecraft. In AAAI, 2017.","title":"A deep hierarchical approach to lifelong learning in minecraft","authors":"Chen Tessler, Shahar Givony","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Van Hasselt et al., 2016] Hado Van Hasselt, Arthur Guez, et al. Deep reinforcement learning with double q-learning. In AAAI, 2016.","title":"Deep reinforcement learning with double q-learning","authors":"Van Hasselt","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Vinyals et al., 2019] Oriol Vinyals, Igor Babuschkin, et al. Grandmaster level in starcraft ii using multi-agent reinforce- ment learning. Nature, 575(7782):350-354, 2019.","title":"Grandmaster level in starcraft ii using multi-agent reinforcement learning","authors":"Oriol Vinyals, Igor Babuschkin","year":"2019","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2016] Ziyu Wang, Tom Schaul, et al. Dueling network architectures for deep reinforcement learning. In ICML, 2016.","title":"Dueling network architectures for deep reinforcement learning","authors":"Ziyu Wang, Tom Schaul","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Whitney et al., 2019] William Whitney, Rajat Agarwal, et al. Dynamics-aware embeddings. arXiv:1908.09357, 2019.","title":"Dynamics-aware embeddings","authors":"William Whitney, Rajat Agarwal","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wu et al., 2019] Yueh-Hua Wu, Nontawat Charoenphakdee, et al. Imitation learning from imperfect demonstration. In ICML, 2019.","title":"Imitation learning from imperfect demonstration","authors":"Yueh-Hua Wu, Nontawat Charoenphakdee","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wu et al., 2021] Haiping Wu, Khimya Khetarpal, et al. Self- supervised attention-aware reinforcement learning. In AAAI, 2021.","title":"Selfsupervised attention-aware reinforcement learning","authors":"Haiping Wu, Khimya Khetarpal","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Ye et al., 2020a] Deheng Ye, Guibin Chen, et al. Supervised learning achieves human-level performance in moba games: A case study of honor of kings. IEEE TNNLS, 2020.","title":"Supervised learning achieves human-level performance in moba games: A case study of honor of kings","authors":"Ye","year":"2020","venue":"IEEE TNNLS","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Ye et al., 2020b] Deheng Ye, Guibin Chen, et al. Towards playing full moba games with deep reinforcement learning. In NeurIPS, 2020.","title":"Towards playing full moba games with deep reinforcement learning","authors":"Ye","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Ye et al., 2020c] Deheng Ye, Zhao Liu, et al. Mastering Complex Control in MOBA Games with Deep Reinforce- ment Learning. In AAAI, 2020.","title":"Mastering Complex Control in MOBA Games with Deep Reinforcement Learning","authors":"Ye","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}