{"type":"ScholarlyArticle","title":"Robustly Learning Composable Options in Deep Reinforcement Learning","authors":[{"name":"Akhil Bagaria","slug":"akhil-bagaria","given":"Akhil","family":"Bagaria","affiliation":""},{"name":"Jason Senthil","slug":"jason-senthil","given":"Jason","family":"Senthil","affiliation":""},{"name":"Matthew Slivinski","slug":"matthew-slivinski","given":"Matthew","family":"Slivinski","affiliation":""},{"name":"George Konidaris","slug":"george-konidaris","given":"George","family":"Konidaris","affiliation":""}],"abstract":"Hierarchical reinforcement learning (HRL) is only effective for long-horizon problems  when high-level skills can be reliably sequentially executed. Unfortunately, learning reliably composable skills is difficult, because all the components of every skill are constantly changing during learning. We propose three methods for improving the composability of learned skills: representing skill initiation regions using a combination of pessimistic and optimistic classifiers; learning re-targetable policies that are robust to non-stationary subgoal regions; and learning robust option policies using model-based RL. We test these improvements on four sparse-reward maze navigation tasks involving a simulated quadrupedal robot. Each method successively improves the robustness of a baseline skill discovery method, substantially outperforming state-of-the-art flat and hierarchical methods.","doi":"10.24963/ijcai.2021/298","url":"https://ijcai.org/proceedings/2021/298","pdf":"https://ijcai.org/proceedings/2021/0298.pdf","markdown":"https://ijcai.org/proceedings/2021/298.md","bibtex":"https://ijcai.org/proceedings/2021/bibtex/298","volume":{"slug":"2021","title":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2021","url":"https://ijcai.org/proceedings/2021/","year":2021,"location":"Montreal","dates":"19-27 August 2021","editors":"Prof. Zhi-Hua Zhou","isbn":"978-0-9992411-9-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2021"},"section":"Machine Learning","track":"Main Track","number":298,"pages":{"first":2161,"last":2169},"keywords":["Machine Learning: Deep Reinforcement Learning","Machine Learning: Reinforcement Learning","Robotics: Learning in Robotics"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}