{"type":"ScholarlyArticle","title":"I²HRL: Interactive Influence-based Hierarchical Reinforcement Learning","authors":[{"name":"Rundong Wang","slug":"rundong-wang","given":"Rundong","family":"Wang","affiliation":"School of Computer Science and Engineering , Nanyang Technological University , Singapore"},{"name":"Runsheng Yu","slug":"runsheng-yu","given":"Runsheng","family":"Yu","affiliation":"School of Computer Science and Engineering , Nanyang Technological University , Singapore"},{"name":"Bo An","slug":"bo-an","given":"Bo","family":"An","affiliation":"School of Computer Science and Engineering , Nanyang Technological University , Singapore"},{"name":"Zinovi Rabinovich","slug":"zinovi-rabinovich","given":"Zinovi","family":"Rabinovich","affiliation":"School of Computer Science and Engineering , Nanyang Technological University , Singapore"}],"abstract":"Hierarchical reinforcement learning (HRL) is a promising approach to solve tasks with long time horizons and sparse rewards. It is often implemented as a high-level policy assigning subgoals to a low-level policy. However, it suffers the high-level non-stationarity problem since the low-level policy is constantly changing. The non-stationarity also leads to the data efficiency problem: policies need more data at non-stationary states to stabilize training. To address these issues, we propose a novel HRL method: Interactive Influence-based Hierarchical Reinforcement Learning (I^2HRL). First, inspired by agent modeling, we enable the interaction between the low-level and high-level policies to stabilize the high-level policy training. The high-level policy makes decisions conditioned on the received low-level policy representation as well as the state of the environment. Second, we furthermore stabilize the high-level policy via an information-theoretic regularization with minimal dependence on the changing low-level policy. Third, we propose the influence-based exploration to more frequently visit the non-stationary states where more transition data is needed. We experimentally validate the effectiveness of the proposed solution in several tasks in MuJoCo domains by demonstrating that our approach can significantly boost the learning performance and accelerate learning compared with state-of-the-art HRL methods.","doi":"10.24963/ijcai.2020/433","url":"https://ijcai.org/proceedings/2020/433","pdf":"https://ijcai.org/proceedings/2020/0433.pdf","markdown":"https://ijcai.org/proceedings/2020/433.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/433","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":433,"pages":{"first":3131,"last":3138},"keywords":["Machine Learning: Deep Reinforcement Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25384/file_25384.m3u8","length_sec":298,"provider":"hls"}],"references":[{"position":1,"raw":"[Alemi et al., 2016] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410, 2016.","title":"Deep variational information bottleneck","authors":"Ian Alexander A Alemi, Joshua V Fischer, Kevin Dillon, Murphy","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bacon et al., 2017] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In AAAI Conference on Artificial Intelligence, pages 1726-1734, 2017.","title":"The option-critic architecture","authors":"Bacon","year":"2017","venue":"AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bellemare et al., 2016] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Process- ing Systems, pages 1471-1479, 2016.","title":"Unifying count-based exploration and intrinsic motivation","authors":"Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, Remi Munos","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cover and Thomas, 2012] Thomas M Cover and Joy A Thomas. Elements of Information Theory. John Wiley & Sons, 2012.","title":"Elements of Information Theory","authors":"M Thomas, Joy A Cover, Thomas","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Duan et al., 2016] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In Inter- national Conference on Machine Learning, pages 1329- 1338, 2016.","title":"Benchmarking deep reinforcement learning for continuous control","authors":"Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel","year":"2016","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Eysenbach et al., 2018] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070, 2018.","title":"Diversity is all you need: Learning skills without a reward function","authors":"Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, Sergey Levine","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Florensa et al., 2017] Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforcement learning. arXiv preprint arXiv:1704.03012, 2017.","title":"Stochastic neural networks for hierarchical reinforcement learning","authors":"Carlos Florensa, Yan Duan, Pieter Abbeel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fujimoto et al., 2018] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv:1802.09477, 2018.","title":"Addressing function approximation error in actor-critic methods","authors":"Scott Fujimoto, Herke Van Hoof, David Meger","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Heess et al., 2016] Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap, Martin Riedmiller, and David Silver. Learning and transfer of modulated locomotor con- trollers. arXiv preprint arXiv:1610.05182, 2016.","title":"Learning and transfer of modulated locomotor controllers","authors":"Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap, Martin Riedmiller, David Silver","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kim et al., 2019] Youngjin Kim, Wontae Nam, Hyunwoo Kim, Ji-Hoon Kim, and Gunhee Kim. Curiosity- bottleneck: Exploration by distilling task-specific novelty. In International Conference on Machine Learning, pages 3379-3388, 2019.","title":"Curiositybottleneck: Exploration by distilling task-specific novelty","authors":"Youngjin Kim, Wontae Nam, Hyunwoo Kim, Ji-Hoon Kim, Gunhee Kim","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Konidaris and Barto, 2009] George Konidaris and Andrew Barto. Efficient skill learning using abstraction selec- tion. In International Joint Conference on Artificial In- telligence, pages 1107-1112, 2009.","title":"Efficient skill learning using abstraction selection","authors":"Barto Konidaris, George Konidaris, Andrew Barto","year":"2009","venue":"International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2009/187"},{"position":12,"raw":"[Kulkarni et al., 2016] Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Ad- vances in Neural Information Processing Systems, pages 3675-3683, 2016.","title":"Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation","authors":"D Tejas, Karthik Kulkarni, Ardavan Narasimhan, Josh Saeedi, Tenenbaum","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Levy et al., 2019] Andrew Levy, Robert Platt, and Kate Saenko. Hierarchical reinforcement learning with hind- sight. In International Conference on Learning Rep- resentations, 2019. https://openreview.net/forum?id= ryzECoAcY7.","title":"Hierarchical reinforcement learning with hindsight","authors":"Levy","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":"https://openreview.net/forum?id=ryzECoAcY7","internal_url":null},{"position":14,"raw":"[Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous con- trol with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.","title":"Continuous control with deep reinforcement learning","authors":"Jonathan J Timothy P Lillicrap, Alexander Hunt, Nicolas Pritzel, Tom Heess, Yuval Erez, David Tassa, Daan Silver, Wierstra","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Maaten and Hinton, 2008] Laurens van der Maaten and Ge- offrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(Nov):2579-2605, 2008.","title":"Laurens van der Maaten and Geoffrey Hinton","authors":"Hinton Maaten","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nachum et al., 2018] Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierar- chical reinforcement learning. In Advances in Neural In- formation Processing Systems, pages 3303-3313, 2018.","title":"Data-efficient hierarchical reinforcement learning","authors":"Ofir Nachum, Shane Shixiang, Honglak Gu, Sergey Lee, Levine","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nachum et al., 2019] Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? arXiv preprint arXiv:1909.10618, 2019.","title":"Why does hierarchy (sometimes) work so well in reinforcement learning","authors":"Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, Sergey Levine","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Papoudakis et al., 2019] Georgios Papoudakis, Filippos Christianos, Arrasy Rahman, and Stefano V Albrecht. Dealing with non-stationarity in multi-agent deep rein- forcement learning. arXiv preprint arXiv:1906.04737, 2019.","title":"Dealing with non-stationarity in multi-agent deep reinforcement learning","authors":"Georgios Papoudakis, Filippos Christianos, Arrasy Rahman, Stefano V Albrecht","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rabinowitz et al., 2018] Neil C Rabinowitz, Frank Perbet, H Francis Song, Chiyuan Zhang, SM Eslami, and Matthew Botvinick. Machine theory of mind. arXiv preprint arXiv:1802.07740, 2018.","title":"Machine theory of mind","authors":"Frank Neil C Rabinowitz, Francis Perbet, Chiyuan Song, S M Zhang, Matthew Eslami, Botvinick","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rafati and Noelle, 2019] Jacob Rafati and David C Noelle. Efficient exploration through intrinsic motivation learn- ing for unsupervised subgoal discovery in model-free hierarchical reinforcement learning. arXiv preprint arXiv:1911.10164, 2019.","title":"Efficient exploration through intrinsic motivation learning for unsupervised subgoal discovery in model-free hierarchical reinforcement learning","authors":"Noelle ; Rafati, Jacob Rafati, David C Noelle","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Silver et al., 2016] David Silver, Aja Huang, Chris J Maddi- son, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Pan- neershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484, 2016.","title":"Mastering the game of go with deep neural networks and tree search","authors":"David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den, Marc Driessche, Lanctot","year":"2016","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tishby and Zaslavsky, 2015] Naftali Tishby and Noga Za- slavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pages 1-5, 2015.","title":"Deep learning and the information bottleneck principle","authors":"Naftali Tishby, Noga Zaslavsky","year":"2015","venue":"IEEE Information Theory Workshop (ITW)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vezhnevets et al., 2017] Alexander Sasha Vezhnevets, Si- mon Osindero, Tom Schaul, Nicolas Heess, Max Jader- berg, David Silver, and Koray Kavukcuoglu. Feudal net- works for hierarchical reinforcement learning. In Inter- national Conference on Machine Learning, pages 3540- 3549, 2017.","title":"Feudal networks for hierarchical reinforcement learning","authors":"Sasha Alexander, Simon Vezhnevets, Tom Osindero, Nicolas Schaul, Max Heess, David Jaderberg, Koray Silver, Kavukcuoglu","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}