---
title: "Hindsight Trust Region Policy Optimization"
authors:
  - "Hanbo Zhang"
  - "Site Bai"
  - "Xuguang Lan"
  - "David Hsu"
  - "Nanning Zheng"
doi: "10.24963/ijcai.2021/459"
url: "https://ijcai.org/proceedings/2021/459"
pdf: "https://ijcai.org/proceedings/2021/0459.pdf"
venue: "Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence"
year: 2021
pages: "3335-3341"
track: "Main Track"
keywords:
  - "Machine Learning: Reinforcement Learning"
  - "Machine Learning: Deep Reinforcement Learning"
---

# Hindsight Trust Region Policy Optimization

Hanbo Zhang, Site Bai, Xuguang Lan, David Hsu, Nanning Zheng

*Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence*, pages 3335-3341. DOI: [10.24963/ijcai.2021/459](https://doi.org/10.24963/ijcai.2021/459)

## Abstract

Reinforcement Learning (RL) with sparse rewards is a major challenge. We pro-
pose Hindsight Trust Region Policy Optimization (HTRPO), a new RL algorithm
that extends the highly successful TRPO algorithm with hindsight to tackle the
challenge of sparse rewards. Hindsight refers to the algorithm’s ability to learn
from information across goals, including past goals not intended for the current
task. We derive the hindsight form of TRPO, together with QKL, a quadratic
approximation to the KL divergence constraint on the trust region. QKL reduces
variance in KL divergence estimation and improves stability in policy updates. We
show that HTRPO has similar convergence property as TRPO. We also present
Hindsight Goal Filtering (HGF), which further improves the learning performance
for suitable tasks. HTRPO has been evaluated on various sparse-reward tasks,
including Atari games and simulated robot control. Experimental results show that
HTRPO consistently outperforms TRPO, as well as HPG, a state-of-the-art policy
14 gradient algorithm for RL with sparse rewards.