---
title: "Reinforcement Learning for Sparse-Reward Object-Interaction Tasks in a First-person Simulated 3D Environment"
authors:
  - "Wilka Carvalho"
  - "Anthony Liang"
  - "Kimin Lee"
  - "Sungryull Sohn"
  - "Honglak Lee"
  - "Richard Lewis"
  - "Satinder Singh"
doi: "10.24963/ijcai.2021/306"
url: "https://ijcai.org/proceedings/2021/306"
pdf: "https://ijcai.org/proceedings/2021/0306.pdf"
venue: "Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence"
year: 2021
pages: "2219-2226"
track: "Main Track"
keywords:
  - "Machine Learning: Deep Reinforcement Learning"
  - "Machine Learning: Deep Learning"
  - "Machine Learning: Relational Learning"
  - "Machine Learning: Reinforcement Learning"
---

# Reinforcement Learning for Sparse-Reward Object-Interaction Tasks in a First-person Simulated 3D Environment

Wilka Carvalho, Anthony Liang, Kimin Lee, Sungryull Sohn, Honglak Lee, Richard Lewis, Satinder Singh

*Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence*, pages 2219-2226. DOI: [10.24963/ijcai.2021/306](https://doi.org/10.24963/ijcai.2021/306)

## Abstract

Learning how to execute complex tasks involving multiple objects in a 3D world is challenging when there is no ground-truth information about the objects or any demonstration to learn from.
When an agent only receives a signal from task-completion, this makes it challenging to learn the object-representations which support learning the correct object-interactions needed to complete the task.
In this work, we formulate learning an attentive object dynamics model as a classification problem, using random object-images to define incorrect labels for our object-dynamics model. 
We show empirically that this enables object-representation learning that captures an object's category (is it a toaster?), its properties (is it on?), and object-relations (is something inside of it?).
With this, our core learner (a relational RL agent) receives the dense training signal it needs to rapidly learn object-interaction tasks.
We demonstrate results in the 3D AI2Thor simulated kitchen environment with a range of challenging food preparation tasks.
We compare our method's performance to several related approaches and against the performance of an oracle: an agent that is supplied with ground-truth information about objects in the scene.
We find that our agent achieves performance closest to the oracle in terms of both learning speed and maximum success rate.