Reinforcement Learning for Sparse-Reward Object-Interaction Tasks in a First-person Simulated 3D Environment

1 Introduction

Consider a robotic home-aid agent that learns objectinteraction tasks that involve using multiple objects together to accomplish various tasks such as chopping vegetables or heating meals. Such tasks are important for artificial intelligence (AI) to make progress on because of their large potential to impact our everday world: nursing robots can serve healthcare workers in hospitals, and home-aid robots can help busy families, the disabled, and the elderly.

Prior work on object-interaction tasks has focused on achieving strong training performance using expert demonstrations [Zhu et al., 2017; Shridhar et al., 2019]. Unfortunately, [Zhu et al., 2017] found they were unable to learn relatively simple pick and place tasks when only learning from a sparse task-completion signal. Other work has relaxed the learning problem by relying on domain knowledge in the form of shaped rewards or object-affordance knowledge [Jain et al., 2019; Gordon et al., 2018].

∗Contact author: wcarvalh@umich.edu

Unfortunately, expert demonstrations and shaped rewards can be challenging to obtain for tasks novel to an agent. Additionally, it can be tedious or impossible to obtain ground-truth information about all novel objects an agent may encounter. Ideally, agents are capable of learning object-interaction tasks without this information. To work towards this, we focus on the setting where none of these are available.

Learning object-interaction tasks without expert demonstrations or shaped rewards is challenging because selecting between object-interactions induces a branching factor that scales with the number of visible objects, leading the agent choose from 50-100 actions at a given time-step. This leads the agent to infrequently experience a successful episode. When the agent does, task completion typically occurs after many hundred time-steps. Consider learning to toast bread. The agent should learn to turn on the toaster after a bread slice is placed inside, i.e. it needs to learn to represent containment relationships (the bread is inside the toaster) and object properties (the toaster is on or off). Without domain knowledge about objects, task-completion alone provides a weak learning signal for learning both to represent 3D object categories, properties, and relationships. When episodes last for hundreds of time-steps and the agent interacts with many objects, this makes it challenging to learn about about how the agent's object-interactions led to reward.

In this work, we find that we can achieve strong training performance on object-interaction tasks without expert demonstrations, shaped rewards, or ground-truth object-knowledge by incorporating inter-object attention and an object-centric model into a reinforcement learning agent.We call our agent the Learning Object Attention & Dynamics (or LOAD) agent. LOAD is composed of a base object-centric relational policy (Attentive Object-DQN, §4.1) that leverages inter-object attention to incorporate object-relationships when estimating object-interaction action-values. Without ground-truth information to identify object categories, properties, or relationships, LOAD learns object-representations with a novel learning objective that frames learning an object-model as a classification problem, where random object-embeddings are incorrect labels (Attentive Object-Model, §4.2). By doing so, we provide the object-model with a dense learning signal for learning represent both object categories, but also changes in object-properties caused by different object-interactions. Additionally, by sharing inter-object attention between the policy and the model, learning the model helps drive learning of inter-object attention helpful for speedening task learning.

In order to study object-interaction tasks and evaluate our agent, we adopt the virtual home-environment AI2Thor [Kolve et al., 2017] (or Thor). Thor is an open-source environment that is high-fidelity, 3D, partially observable, and enables object-interactions. We show that LOAD is able to significantly reduce sample complexity in this domain where no prior work has yet learned sparse-reward object-interaction tasks without expert demonstrations or shaped rewards.

In our main evaluation, we compare pairing Attentive Object-DQN with our Attentive Object-Model to alternative representation learning methods, and show that learning with our object-model best closes the performance gap to an agent supplied with ground-truth information about object categories, properties, and relationships (§5.1). Through an analysis of the learned object-representations and inter-object attention learned by each auxiliary task, we provide quantitative evidence that our Attentive Object-Model best learns representations that capture the ground-truth information present in our oracle (§5.2). We hypothesize that this is the source of our strong performance. Afterwards, we perform a series of ablations to study the importance of object-representations which capture object-properites and object-relations for reducing sample-complexity (§5.3).

In summary, the key contributions of our proposal are: (1) LOAD: an RL agent that demonstrates how to learn sparsereward object-interaction tasks with first-persosn vision without expert demonstrations, shaped rewards, or ground-truth object-knowledge. (2) A novel Attentive Object-Model auxilliary task, which frames learning an object-model as a classification problem. With our analysis, we provide evidence that for our 3D, high-fidelty domain and our architecture, it is key to learn object-representations which not only capture objectcategories but also object-properties and object-relations.

2 RelatedWork

Learning Object-interaction Tasks in 3D, First-person Environments. Due to the large branching factor induced by object-interactions, most work here has relied extensively on expert demonstrations [Zhu et al., 2017; Shridhar et al., 2019; Xu et al., 2019] or avoided this problem by hard-coding objectselection [Jain et al., 2019; Gordon et al., 2018]. The work most closely related to ours is [Oh et al., 2017] (in Minecraft) and [Zhu et al., 2017] (in Thor). Both develop a hierarchical reinforcement learning agent where a meta-controller provides goal object-interactions for a low-level controller to complete using ground-truth object-information. Both provide agents with knowledge of all objects and both assume lower-level policies pretrained to navigate to objects and to select interactions with a desired object. In contrast, we do not provide the agent with any ground-truth object information; nor do we pretrain navigation to objects or selection of them.

Object-Centric Relational RL. An intutive approach to tasks with objects is object-centric relational RL. Most work here has used hand-designed representations of objects and their relations, showing things like improved sampleefficiency [Xu et al., 2020], improved policy quality [Zaragoza et al., 2010], and generalization to unseen objects [Van Hoof et al., 2015]. In contrast, we seek to learn object-representations and object-relations implicitly with our network. Most similar to our work is [Zambaldi et al., 2018]-which applies attention to the feature vector outputs of a CNN. In this work, Attentive Object-DQN is a novel architecture extension for a setting with an object-centric observation- and action-space. Additionally, we show that learning an object-model as an auxilliary task can help drive learning of attention.

Learning an Object-model as an Auxiliary Task. Most prior work here has focused on how an object-model can be used in model-based reinforcement learning by enabling superior planning [Ye et al., 2020; Veerapaneni et al., 2020; Watters et al., 2019]. In contrast, we do not use our objectmodel for planning and instead show that it can be leveraged to learn object-representation and inter-object attention to support faster policy learning in a model-free setting. Additionally, other work focused on domains where representation-learning only had to differntiate object-categories. We show that our method can additionally differentiate object-properties and does so significantly better than the object-model of [Watters et al., 2019]. Our attentive object-model is most similar to the Contrastive Structured World Model (CSWM) [Kipf et al., 2019], which uses a maximum margin contrastive learning objective [Hadsell et al., 2006] to learn an object-model. Instead, we formulate a novel object-model contrastive objective as learning a classification problem. We note that they applied their model towards video-prediction and not reinforcement learning.

3 Sparse-reward Object-interaction Tasks in a First-person Simulated 3D Environment

Observations. We focus on an agent that has a 2D camera for experiencing egocentric observations xego of the environment. Our agent also has a pretrained vision system that enables it to extract bounding box image-patches corresponding to the visible objects in its observation Xo = {xo,i }. Besides boxes around objects, no other information is extracted (i.e. no labels, identifiers, poses, etc.). We assume the agent has access to its (x, y, z) location and body rotation (ϕ1, ϕ2, ϕ3) in a global coordinate frame, xloc = (x, y, z, ϕ1, ϕ2, ϕ3).

Actions. In this work, we focus on the Thor environment. Here, the agent has 8 base object-interactions: I = {Pickup, Put, Open, Close, Turn on, Turn off, Slice, Fill}. The agent interacts with objects by selecting (object-image-patch, interaction) pairs a = (b, xo,c ) ∈ I × Xo , where xo,c corresponds to the chosen image-patch. For example, the agent can turn on the stove by selecting the image-patch containing the stoveknob and the Turn on interaction (see Figure 2 for a diagram). Each action is available at every time-step and can be applied to all objects (i.e. no affordance information is given/used). Interactions occur over one time-step, though their effect may occur over multiple. For the example above, when the agent applies "Turn on" to the stove knob, food on the stove will take several time-steps to heat. In addition to object-interactions, the agent can select from 8 base navigation actions: AN = {Move ahead, Move back, Move right, Move left, Look up, Look down, Rotate right, Rotate left}. With {Look up, Look down}, the agent can rotate its head up or down in increments of 30◦ between angles {0◦ ,±30◦ ,±60◦ }. 0◦ represents looking straight ahead. With {Rotate Left, Rotate Right}, the agent can rotate its body by {±90o }.

Tasks. We construct 8 tasks with the following 4 challenges. Challenge (A): the visual complexity of task objects (e.g. the cup is translucent). Challenge (B): the number of objects to be interacted with (e.g., "Slice Apple, Potato, Lettuce" requires the agent interact with 4 objects). Challenge (C): whether object-containment must be recognized and used (e.g. toasting bread in a toaster). Challenge (D): whether object-properties change (e.g. bread get's cooked). See Figure 1 for a description of the challenges associated with each task and Figure 1 for example panels of 2 tasks.

Reward. We consider a single-task setting where the agent receives a terminal reward of 1 upon task-completion.

4 LOAD: Learning Object Attention & Dynamics Agent

LOAD is a reinforcement learning agent composed of an object-centric relational policy, Attentive Object-DQN, and an Attentive Object-Model. LOAD uses 2 perceptual modules. The first, f o enc , takes in an observation x and produces objectencodings {zo,i }ni=1 for the n visible object-image-patches Xo = {xo,i }ni=1 , where zo,i ∈ Rdo . The second, f κ enc , takes in the egocentric observation and location xκ = [xego , xloc ] to produce the context for the objects zκ ∈ Rdκ . LOAD treats state as the union of these variables: s = {zo,i }∪{zκ }. Given object encodings, Attentive Object-DQN computes actionvalues Q(s, a = (b, xo,i )) for interacting with an object xo,i and leverages an attention module A to incorporate information about other objects xo,j̸=i into this computation (see §4.1).

To address the representation learning challenge induced by a sparse-reward signal, object-representations zo,i and objectattention A are trained to predict object-dynamics with an attentive object-model (see §4.2). See Figure 2 for an overview of the full architecture.

4.1 Attentive Object-DQN

Attentive Object-DQN uses ̂Q(s, a) to estimate the action-value function Qπ (s, a) = Eπ[∑∞t=0 γ t rt|St = s,At = a], which maps state-action pairs to the expected return on starting from that state-action pair and following policy π thereafter.

Leveraging Inter-object Attention During Action-value Estimation. In many tasks, an agent must integrate information about multiple objects when estimating Q-values. For example, in the "toast bread" task, the agent needs to integrate information about the toaster and the bread when deciding to turn on the toaster. To accomplish this, we exploit the objectcentric observations-space and employ attention [Vaswani et al., 2017] to incorporate inter-object attention into Q-value estimation. More formally, given an object-encoding zo,i , we can use attention to select relevant objects A(zo,i , Zo ) ∈ Rdo for estimating Q(s, a = (b, xo,i )). With a matrix of objectencodings, Zo = [zo,i ]i ∈ Rn×do , we can perform this computation efficiently for each object-image-patch via:

Here, ZoW qo projects each object-encoding to a "query" space and ZoW k projects each encoding to a "key" space, where their dot-product determines whether a key is selected for a query. The softmax acts as a soft selection-mechanism for selecting an object-encoding in Zo .

Estimating action-values. We can incorporate attention to estimate Q-values for selecting an interaction b ∈ I on an object xo,i as follows:

Importantly, this enables us to compute Q-values for a variable number of unlabeled objects. We can similarly incorporate attention to compute Q-values for navigation actions by replacing ZoW qo with (W qκ zκ )⊤ in equation 1. We estimate Q-values for navigation actions b ∈ AN as follows:

Learning. We estimate ̂Q(s, a) as a Deep Q-Network (DQN) by minimizing the following temporal difference objective:

where yt = rt + γ ̂Q(st+1, at+1; θold) is the target Q-value, and θold is an older copy of the parameters θ. To do so, we store trajectories containing transitions (st, at, rt, st+1) in a replay buffer that we sample from [Mnih et al., 2015]. To stabilize learning, we use Double-Q-learning [Van Hasselt et al., 2016] to choose the next action: at+1 = argmaxa ̂Q(st+1, a; θ).

4.2 Attentive Object-Dynamics Model

Consider the global set of objects {ogt,i }mi=1 , where m is the number of objects in the environment. At each time-step, each object-image-patch the agent observes corresponds to a 2D projection of ogt,i , ρ(ogt,i ) (or ρg,i t for short) and encodes it as zg,i t . Given, an object-image-patch encoding zg,i t and a performed interaction at, we can define an object-dynamics model D(Zo t , zg,i t , at) which produces the resultant encoding for ρg,i t+1 . We want D(Zo t , zg,i t , at) to be closer to zg,i t+1 than to encodings of other object-image-patches.

Classification Problem. We can formalize this by setting up a classification problem. For an object-image-patch encoding zg,i t , we define the prediction as the output of our object-dynamics model D(Zo t , zg,i t , at). We define the label as the encoding of a visible object-image-patch at the next time-step with the highest cosine similarity to the original encoding zg,i + = argmaxzg,j t+1 cos(zg,i t , zg,j t+1 ). We can then select K random object-encodings {zo k,-}Kk=1 as incorrect labels. Rewriting D(Zo t , zg,i t , at) as D, this leads to:

The set of indices corresponding to visible objects at time t is vt = {i : ρg,i t is visible at time t}. The set of observed objectimage-patch encodings is then Zo t = {zo,j t } = {zg,i t }i∈vt .

Bottom-panel: by measuring the % AUC achieved by each agent w.r.t to the agent with ground-truth information, we can measure how close each method is to the performance of an agent with ground-truth object-knowledge. We find LOAD (red), which learns an attentive object-model best closes the performance gap on 6/8. We hypothesize that this is due to our object-model's ability to capture oracle object-information about object-categories, object-properties, and object-relations. We show evidence for this in Table 2.

Assuming the probability of each object's next state is conditionally independent given the current set of objects and the action taken, we arrive at the following objective:

Our final objective becomes:

Leveraging Inter-object Attention for Improved Accuracy. Consider slicing an apple with a knife. When selecting "slice" on the apple patch, learning to attend to the knife patch both enables more accurate estimation of Q-values and higher model-prediction accuracy. We can accomplish this by incorporating A(zg,i , Zo ) into our object-model as follows:

To learn an action encoding za t for action at, following [Oh et al., 2015; Reed et al., 2014], we employ multiplicative interactions so our learned action representation za t compactly models the cartesian product of all base actions b and objectimage-patch selections oc as

where W o ∈ Rda×do , W b ∈ Rda×|AI | , and ⊙ is an elementwise hadamard product. In practice, fmodel is a small 1- or 2layer neural network making this method compact and simple to implement.

5 Experiments

The primary aim of our experiments is to study how different auxilliary tasks for learning object-representations enable sample complexity comparable to an agent with oracle objectknowledge. We additionally study the degree to which each auxilliary task enables object-representation learning that captures the ground-truth knowledge present in our oracle agent. We conclude this section with ablation experiments studying the importance of different forms of object-knowledge in task learning.

Evaluation Settings. The agent's spawning location is randomized from 81 grid positions. The agent receives a terminal reward of 1 if its task is completed successfully and 0 otherwise. It receives a time-step penalty of -0.04. Episodes have a time-limit of 500 time-steps. The agent has a budget of 500K samples to learn a task. This was the budget needed by a relational agent with oracle object-information.

Baseline Methods for Comparison. In order to study the effects of competing object representation learning methods, we compare combining Attentive Object-DQN with the Attentive Object-Model against four baseline methods:

1. Attentive Object-DQN. This baseline has no auxiliary task and lets us study how well an agent can learn from the sparse-reward signal alone.
2. Ground-Truth Object-Information. This baseline has no auxiliary task. Instead, we supply the agent with 14 groundtruth features from the simulator. They roughly describe an object's category (is it a toaster?), its properties (e.g., is it on/off/etc.?), and relevant object-containment (e.g., what object is this object inside of?). We found that this handdesigned object-representation enabled Attentive ObjectDQN to learn all our tasks within our sample-budget and it is our basis for comparing unsupervised object-representation learning methods. Please see §A.1 for detailed descriptions of these features.
3. OCN. The Object Contrastive Network [Pirk et al., 2019]. This method also employs a classification-like contrastive learning objective to cluster object-images across time-steps. However, it doesn't use an object-model or incorporate action-information. This enables us to study the importance of incoporating an object-model and action information.
4. COBRA Object-Model. This is the object-model employed by the COBRA RL agent [Watters et al., 2019]. They also targeted improved sample-efficiency-though in a simpler, fully-observable 2D environment with shapes that only needed differentiation by category. Their model had no mechanism for incorporating inter-object relations into its predictions.

To enable faster learning in a sparse-reward setting, all baselines sample training batches using a second self-imitation learning replay buffer of successful episodes [Oh et al., 2018].

5.1 Task Performance

Metrics. We evaluate agent performance by measuring the agent's success rate over 5K frames every 25K frames of experience. The success rate is the proportion of episodes that the agent completes. We compute the mean and standard error of these values across 5 seeds. To study sampleefficiency, we compare each method to "Ground-Truth ObjectInformation" by computing what percent of the Ground-Truth Object-Information mean success rate AUC each method achieved.

Performance

We present sample-efficiency bar plots for all 8 of our tasks in Figure 3. We found that using containment relationships and recognizing changing object-properties (Challenges C & D in

• §3) were most indicative of task difficulty. We only present learning curve results for 4 tasks which match this criteria. We present all learning curves in Figure 5 in appendix §C. We additionally present the maximum success rate achieved by each method in Table 6 in appendix §C.1.

We find that using Ground-Truth Object-Information is able to get the highest success rate on all tasks. Attentive Object-DQN performs below all methods besides OCN on 7/8 tasks. Surprisingly, Attentive Object-DQN outperforms OCN on 5/8 tasks. OCN doesn't incorporate action-information when learning to represent object-images across time-steps. We hypothesize that this leads it to learn degenerate objectrepresentations that cannot discriminate object-propertiess that change due to actions, something important for our tasks.

In terms of sample-efficiency, our Attentive Object-Model comes closest to Ground-Truth Object-Information on 6/8 tasks. For tasks that require using objects together, such as "Fill Cup with Water" where a cup must be used in a sink or "Toast Bread Slice" where bread must be cooked in a toaster, our Attentive Object-Model significantly improves over the COBRAObject-Model. Interestingly, sample-efficiency goes above 100% on 2 tasks. We suspect that this is because the object-model provides a learning signal for inter-object attention which is not provided by oracle information.

5.2 Analysis of Learned Object Representations

In Table 2, we explore our conjecture that the key to strong task-learning performance is an agent's ability to capture the information present in the oracle agent. To study this, we freeze the parameters of each encoding function, and add a linear layer to predict object-categories, object-properties, and containment relationships using a dataset of collected objectinteractions we construct (see Appendix B.3 for details on the dataset and training). We find that our object-model best captures the information present in the oracle agent.

5.3 Ablations

Importance of Object-properties and Object-relations. To verify that capturing object-properties and -relations is key, we train an agent with only oracle object-category information. We find that this agent is not able to learn tasks that require using objects together as object-properties change in our sample-budget (see Figure 4(a)).

Importance of Inter-object Attention for Policy. In order to verify the utility of using attention as an inductive bias for capturing object-relations, we ablate attention from both Attentive Object-DQN and our Attentive Object-Model. First, we look at two variants of Attentive Object-DQN without attention. The first is a regular DQN. In the second, we incorporate inter-object information by using the average of all present object-embeddings (DQN + Object Average). Neither learns our tasks in the sample-budget (see Figure 4).

(b) Ablation of inter-object attention in policy. Without this, DQN cannot learn these tasks in our sample-budget. See §5.3 for details.

Figure 4: Ablation Results.

Importance of Inter-object Attention for Model. Additionally, we look at performance where our policy can use inter-object attention but remove inter-object attention from our object-model. Without attention, we still get relatively good performance with 70% success rate; however, attention in the object-model helps increase this to 90%+ (see Figure 6 in our appendix for details).

6 Conclusion

We have shown that learning an attentive object-model can enable sample-efficient learning in high-fidelity, 3D, objectinteraction domains without access to expert demonstrations or ground-truth object-information. Further, when compared to strong unsupervised learning baselines, we have shown that our object-model best captures object-categories, objectproperties, and containmennt-relationships. We believe that LOAD is a promising steps towards agents that can efficiently learn complex object-interaction tasks.

Acknowledgements

This work was supported in part by a grant from DARPA's L2M program and by a NSF CAREER IIS 1453651 Grant. Any opinions, findings, conclusions, or recommendations expressed here are those of the authors and do not necessarily reflect the views of the sponsors. Additionally, WC was supported by an NSF and an RMF Fellowship.