{"type":"ScholarlyArticle","title":"Hindsight Trust Region Policy Optimization","authors":[{"name":"Hanbo Zhang","slug":"hanbo-zhang","given":"Hanbo","family":"Zhang","affiliation":""},{"name":"Site Bai","slug":"site-bai","given":"Site","family":"Bai","affiliation":""},{"name":"Xuguang Lan","slug":"xuguang-lan","given":"Xuguang","family":"Lan","affiliation":""},{"name":"David Hsu","slug":"david-hsu","given":"David","family":"Hsu","affiliation":""},{"name":"Nanning Zheng","slug":"nanning-zheng","given":"Nanning","family":"Zheng","affiliation":""}],"abstract":"Reinforcement Learning (RL) with sparse rewards is a major challenge. We pro-\r\npose Hindsight Trust Region Policy Optimization (HTRPO), a new RL algorithm\r\nthat extends the highly successful TRPO algorithm with hindsight to tackle the\r\nchallenge of sparse rewards. Hindsight refers to the algorithm’s ability to learn\r\nfrom information across goals, including past goals not intended for the current\r\ntask. We derive the hindsight form of TRPO, together with QKL, a quadratic\r\napproximation to the KL divergence constraint on the trust region. QKL reduces\r\nvariance in KL divergence estimation and improves stability in policy updates. We\r\nshow that HTRPO has similar convergence property as TRPO. We also present\r\nHindsight Goal Filtering (HGF), which further improves the learning performance\r\nfor suitable tasks. HTRPO has been evaluated on various sparse-reward tasks,\r\nincluding Atari games and simulated robot control. Experimental results show that\r\nHTRPO consistently outperforms TRPO, as well as HPG, a state-of-the-art policy\r\n14 gradient algorithm for RL with sparse rewards.","doi":"10.24963/ijcai.2021/459","url":"https://ijcai.org/proceedings/2021/459","pdf":"https://ijcai.org/proceedings/2021/0459.pdf","markdown":"https://ijcai.org/proceedings/2021/459.md","bibtex":"https://ijcai.org/proceedings/2021/bibtex/459","volume":{"slug":"2021","title":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2021","url":"https://ijcai.org/proceedings/2021/","year":2021,"location":"Montreal","dates":"19-27 August 2021","editors":"Prof. Zhi-Hua Zhou","isbn":"978-0-9992411-9-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2021"},"section":"Machine Learning","track":"Main Track","number":459,"pages":{"first":3335,"last":3341},"keywords":["Machine Learning: Reinforcement Learning","Machine Learning: Deep Reinforcement Learning"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}