---
title: "Proximal Curriculum with Task Correlations for Deep Reinforcement Learning"
authors:
  - "Georgios Tzannetos"
  - "Parameswaran Kamalaruban"
  - "Adish Singla"
doi: "10.24963/ijcai.2024/556"
url: "https://ijcai.org/proceedings/2024/556"
pdf: "https://ijcai.org/proceedings/2024/0556.pdf"
venue: "Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence"
year: 2024
pages: "5027-5036"
track: "Main Track"
keywords:
  - "Machine Learning: ML: Reinforcement learning"
  - "Planning and Scheduling: PS: Markov decisions processes"
---

# Proximal Curriculum with Task Correlations for Deep Reinforcement Learning

Georgios Tzannetos, Parameswaran Kamalaruban, Adish Singla

*Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence*, pages 5027-5036. DOI: [10.24963/ijcai.2024/556](https://doi.org/10.24963/ijcai.2024/556)

## Abstract

Curriculum design for reinforcement learning (RL) can speed up an agent's learning process and help it learn to perform well on complex tasks. However, existing techniques typically require domain-specific hyperparameter tuning, involve expensive optimization procedures for task selection, or are suitable only for specific learning objectives. In this work, we consider curriculum design in contextual multi-task settings where the agent's final performance is measured w.r.t. a target distribution over complex tasks. We base our curriculum design on the Zone of Proximal Development concept, which has proven to be effective in accelerating the learning process of RL agents for uniform distribution over all tasks. We propose a novel curriculum, ProCuRL-Target, that effectively balances the need for selecting tasks that are not too difficult for the agent while progressing the agent's learning toward the target distribution via leveraging task correlations. We theoretically justify the task selection strategy of ProCuRL-Target by analyzing a simple learning setting with REINFORCE learner model. Our experimental results across various domains with challenging target task distributions affirm the effectiveness of our curriculum strategy over state-of-the-art baselines in accelerating the training process of deep RL agents.

## Full text

## 1 Introduction

Deep reinforcement learning (RL) has shown remarkable success in various fields such as games, continuous control, and robotics, as evidenced by recent advances in the field [Mnih et al., 2015; Lillicrap et al., 2015; Silver et al., 2017; Levine et al., 2016]. However, despite these successes, the broader application of RL in real-world domains is often very limited. Specifically, training RL agents in complex environments, such as contextual multi-task settings and goal-based tasks with sparse rewards, still presents significant challenges [Kirk et al., 2021; Andrychowicz and others, 2017; Florensa et al., 2017; Riedmiller et al., 2018].

Curriculum learning has been extensively studied in the context of supervised learning [Weinshall et al., 2018; Zhou and Bilmes, 2018; Elman, 1993; Bengio et al., 2009]. Recent research has explored the benefits of using curriculum learning in sequential decision making settings, such as reinforcement learning and imitation learning [Florensa et al., 2017; Riedmiller et al., 2018; W¨ohlke et al., 2020; Florensa et al., 2018; Racani`ere et al., 2020; Klink et al., 2020; Eimer et al., 2021; Kamalaruban et al., 2019; Yengera et al., 2021]. The objective of curriculum design in RL is to speed up an agent's learning process and enable it to perform well on complex tasks by exposing it to a personalized sequence of tasks [Narvekar et al., 2020; Portelas et al., 2021]. To achieve this objective, several works have proposed different curriculum strategies based on different design principles, such as the Zone of Proximal Development (ZPD) [Vygotsky and Cole, 1978; Chaiklin, 2003], Self-Paced Learning (SPL) [Kumar et al., 2010; Jiang et al., 2015], and Unsupervised Environment Design (UED) [Dennis et al., 2020]. However, existing techniques typically require domain-specific hyperparameter tuning, involve expensive optimization procedures for task selection, or are suitable only for specific learning objectives, such as uniform performance objectives.

In this work, we investigate curriculum design in contextual multi-task settings with varying degrees of task similarity, where the agent's final performance is measured w.r.t. a target distribution over complex tasks. We base our curriculum design on the Zone of Proximal Development concept, which has proven to be effective in accelerating the learning process of RL agents for uniform distribution over all tasks [Florensa et al., 2017; W¨ohlke et al., 2020; Florensa et al., 2018; Tzannetos et al., 2023]. We propose a novel curriculum strategy, PROCURL-TARGET, that effectively balances the need for selecting tasks that are neither too hard nor too easy for the agent (according to the ZPD concept) while still progressing its learning toward the target distribution via leveraging task correlations. We have mathematically derived our curriculum strategy by analyzing a specific learning setting. The strengths of our curriculum strategy include its broad applicability to many domains with minimal hyperparameter tuning, computational and sample efficiency, easy integration with deep RL algorithms, and applicability to any target distribution over tasks, not just uniform distribution. Our main results and contributions are as follows:

I. We propose a curriculum strategy, PROCURL-

- TARGET, that effectively trades off the suitable task difficulty level for the agent and the progression towards the target tasks (Section 3).
- II. We mathematically derive PROCURL-TARGET for the single target task setting with a discrete pool of tasks by analyzing the effect of picking a task on the agent's learning progress (Section 3.1).
- III. We propose an extension of PROCURL-TARGET that can be applied to a wide range of task spaces and target distributions. This extension can be seamlessly integrated with deep RL frameworks, making it easy to use and apply in various scenarios (Section 3.2).
- IV. We empirically demonstrate that the curricula generated with PROCURL-TARGET significantly improve the training process of deep RL agents in various environments, matching or outperforming existing stateof-the-art baselines (Section 4). 1

## 1.1 RelatedWork

Curriculum strategies based on SPL concept. In the realm of supervised learning, curriculum strategies leveraging the SPL concept attempt to strike a balance between exposing the learner to all available training examples and selecting examples in which it currently performs well [Kumar et al., 2010; Jiang et al., 2015]. In the context of RL, the SPL concept has been adapted by researchers in SPDL [Klink et al., 2020; Klink et al., 2021], SPACE [Eimer et al., 2021], and CURROT [Klink et al., 2022] by controlling the intermediate task distribution with respect to the learner's current training progress. While both SPDL and CURROT involve a setting where the learner's performance is measured w.r.t. a target distribution over the task space (similar to our objective), SPACE operates in a setting where the learner's performance is measured w.r.t. a uniform distribution over the task space. The task selection mechanism varies across these methods. SPDL and CURROT operate by solving an optimization problem at each step to select the most relevant task [Klink et al., 2021; Klink et al., 2022]. On the other hand, SPACE relies on ranking tasks based on the magnitude of differences in current/previous critic values to choose the task for the next step [Eimer et al., 2021]. Furthermore, the work of CURROT [Klink et al., 2022] showcases issues about using KL divergence to measure the similarity between task distributions as used in SPDL - instead, they introduce an alternative approach by posing the curriculum design as a constrained optimal transport problem between task distributions.

Curriculum strategies based on UED concept. The UED problem setting involves automatically designing a distribution of environments that adapts to the learning agent [Dennis et al., 2020]. UED represents a self-supervised RL paradigm in which an environment generator evolves alongside a student policy to develop an adaptive curriculum learning approach. This approach can be utilized to create increasingly complex environments for training a policy, leading to the emergence of Unsupervised Curriculum Design.

[1 Github repository: https://github.com/machine-teaching-group/ ijcai2024-proximal-curriculum-target-rl](https://github.com/machine-teaching-group/ijcai2024-proximal-curriculum-target-rl)

PAIRED [Dennis et al., 2020] is an adversarial training technique that solves the problem of the adversary generating unsolvable environments by introducing an antagonist who works with the environment-generating adversary to design environments in which the protagonist receives a low reward. Furthermore, the connections between UED and another related method called PLR [Jiang et al., 2021b] have been explored in [Jiang et al., 2021a; Parker-Holder et al., 2022], resulting in demonstrated improvements over PAIRED. PLR, originally designed for procedural content generation based environments, samples tasks/levels by prioritizing those with higher estimated learning potential when revisited in the future. TD errors are used to estimate a task's future learning potential. Unlike [Jiang et al., 2021a; Parker-Holder et al., 2022], PLR does not assume control over the environment generation process, requiring only a black box generation process that returns a task given an identifier.

Curriculum strategies based on ZPD concept. Effective teaching provides tasks of moderate difficulty (neither too hard nor too easy) for the learner, as formalized by the ZPD concept [Vygotsky and Cole, 1978; Chaiklin, 2003; Oudeyer et al., 2007; Baranes and Oudeyer, 2013; Zou et al., 2019]. In the context of RL, several curriculum strategies are based on the ZPD concept, such as selecting the next task randomly from a set of tasks with success rates within a specific range [Florensa et al., 2017; Florensa et al., 2018]. However, the threshold values for success rates require tuning based on the learner's progress and domain. A unified framework for performance-based starting state curricula in RL is proposed by [W¨ohlke et al., 2020], while [Tzannetos et al., 2023] propose a broadly applicable ZPD-based curriculum strategy with minimal hyperparameter tuning and theoretical justifications. Nonetheless, these techniques are generally suitable only for settings where the learner's performance is evaluated using a uniform distribution over all tasks.

Curriculum strategies based on domain knowledge. In supervised learning, early works involve ordering examples by increasing difficulty [Elman, 1993; Bengio et al., 2009; Schmidhuber, 2013], which has been adapted in hand-crafted RL curriculum approaches [Wu and Tian, 2016]. Recent works on imitation learning have also utilized iterative machine teaching framework to design greedy curriculum strategies [Kamalaruban et al., 2019; Yengera et al., 2021; Liu et al., 2017; Zhu et al., 2018]. However, these approaches require domain-specific expert knowledge.

Other automatic curriculum strategies. Various automatic curriculum generation approaches exist, including: (i) formulating the curriculum design problem as a meta-level Markov Decision Process [Narvekar and Stone, 2019]; (ii) learning to generate training tasks similar to a teacher [Dendorfer et al., 2020; Matiisen et al., 2019; Turchetta et al., 2020]; (iii) using self-play for curriculum generation [Sukhbaatar et al., 2018]; (iv) leveraging disagreement between different agents trained on the same tasks [Zhang et al., 2020]; and (v) selecting starting states based on a single demonstration [Salimans and Chen, 2018]. Interested readers can refer to recent surveys on RL curriculum design [Narvekar et al., 2020; Portelas et al., 2021].

## Algorithm 1 RL Agent Training as Interaction between Teacher-Student Components

- 1: Input: RL agent's initial policy π1
- 2: for t = 1, 2, . . . do
- 3: Teacher component picks a task ct ∈ C.
- 5: Student component updates the policy to πt+1 using the rollout ξt.
- 4: Student component attempts the task via a trajectory rollout ξt using the policy πt inMct .

- 6: Output: RL agent's final policy πend ← πt+1.

## 2 Formal Setup

We formalize our problem setting based on prior work on teacher-student curriculum learning [Matiisen et al., 2019].

Multi-task RL. We consider a multi-task RL setting with a task/context space C, in which each task c ∈ C is associated with a learning environment modeled as a contextual Markov Decision Process (MDP), denoted by Mc := (S, A, γ, Tc, Rc, P 0 c ) [Hallak et al., 2015; Modi et al., 2018]. The state space S and action space A are shared by all tasks in C, as well as the discount factor γ. Each contextual MDP includes a contextual transition dynamics Tc : S × S × A → [0, 1], a contextual reward function Rc : S × A → [-Rmax,Rmax], where Rmax > 0, and a contextual initial state distribution P 0 c : S → [0, 1]. We denote the space of environments by M = {Mc : c ∈ C}. Moreover, we have a target distribution µ over C that is used for performance evaluation, as further discussed below.

RL agent and training process. We consider an RL agent acting in any environment Mc ∈ M via a contextual policy π : S × C × A → [0, 1] that is a contextual mapping from a state to a probability distribution over actions. Given a task c ∈ C, the agent attempts the task via a trajectory rollout obtained by executing its policy π in the MDP Mc. The trajectory rollout is denoted as ξ = {(s(τ) , a(τ) )}τ=0,1,... with s(0) ∼ P 0 c . The agent's performance on task c is measured by the value function V π (c) := E [∑∞τ=0 γ τ · Rc(s(τ) , a(τ) )∣∣π,Mc ]. The agent training corresponds to finding a policy that performs well w.r.t. the target distribution µ, i.e., maxπ V π µ where V π µ := Ec∼µ [V π (c)]. The training process of the agent involves an interaction between two components: a student component that is responsible for policy updates and a teacher component that is responsible for task selection. The interaction happens in discrete steps indexed by t = 1, 2, . . ., and is formally described in Algorithm 1. Let πend denote the agent's final policy at the end of teacher-student interaction. The training objective is to ensure that the performance of the policy πend is ϵ-nearoptimal, i.e., (maxπ V π µ -V πend µ ) ≤ ϵ.

Student component. We consider parametric policies of the form πθ : S × C × A → [0, 1], where θ ∈ Θ ⊆ Rd . The agent's policy at step t is given by πt := πθt . The student component updates the policy parameter based on the following quantities: the current parameter θt, the task ct picked by the teacher component, and the rollout ξt = {(s(τ) t , a(τ) t )}τ . For example, the policy parameter of the REINFORCE agent [Sutton et al., 1999] is updated as follows: θt+1 ← θt+ ηt · ∑∞τ=0 G(τ) t · [∇θ logπθ(a(τ) t |s(τ) t , ct)]θ=θt , where ηt is

<!-- formula-not-decoded -->

Teacher component. At step t, the teacher component selects a task ct for the student component to attempt via a trajectory rollout, as shown in line 3 in Algorithm 1. The sequence of tasks, also known as the curriculum, that is chosen by the teacher component has a significant impact on the performance improvement of the policy πt. The primary objective of this work is to develop a teacher component to achieve the training objective in a computationally efficient and sample-efficient manner.

## 3 Our Curriculum Strategy

In Section 3.1, we mathematically derive a curriculum strategy for the single target task setting with a discrete pool of tasks. Then, in Section 3.2, we present our final curriculum strategy that is applicable in general learning settings. The proofs are provided in the longer version of the paper.

## 3.1 Curriculum Strategy for Single Target Settings

In this section, we present our curriculum strategy for a setting where the task space C is a discrete set and the target distribution µ is a delta distribution concentrated on a single target task ctarg. To design our curriculum strategy, we investigate the effect of selecting a task ct at time step t on the agent's performance V πθt µ and its convergence towards the target performance V ∗ µ := maxπ V π µ . Therefore, we define the training objective improvement at step t and analyze this metric across both general and specific learning scenarios.

Expected improvement in the training objective. At step t, given the current policy parameter θt, the task ct picked by the teacher component, and the student component's rollout ξt, we define the improvement in the training objective as:

<!-- formula-not-decoded -->

Additionally, we define the expected improvement in the training objective at step t due to picking the task ct as follows [Weinshall et al., 2018; Kamalaruban et al., 2019; Yengera et al., 2021; Graves et al., 2017]:

<!-- formula-not-decoded -->

Based on the above measure, a natural greedy curriculum strategy for selecting the next task ct is given by:

<!-- formula-not-decoded -->

We aim to approximate such a curriculum strategy without computing the updated policy πθt+1 . To this end, we initially analyze the function It(·) for REINFORCE learner model within a general learning setting. Subsequently, we refine and simplify the findings by delving into a specific learning scenario. This analysis enables us to develop an intuitive curriculum strategy by effectively combining the following fundamental factors: (i) the learning potential inherent in the source and target tasks, and (ii) the transfer potential between the source and target tasks, i.e., their similarity.

Gradient alignment approximation of It(·). Here, we analyze the function It(·) within the context of the REINFORCE learner model operating under a general learning setting. We show that the natural greedy curriculum strategy can be approximated by a simple gradient alignment maximization strategy. Initially, through the application of the firstorder Taylor approximation of V πθt+1 (ctarg) at θt, we approximate the improvement in the training objective as follows:

<!-- formula-not-decoded -->

where gt(c) := [∇θV πθ (c)]θ=θt . Subsequently, by utilizing the parameter update form of the REINFORCE agent (i.e., Eξt|ct [θt+1] ← θt +ηt · gt(ct)), we approximate the expected improvement in the training objective as follows:

<!-- formula-not-decoded -->

Consequently, the natural greedy curriculum strategy in Eq. (1) can be effectively approximated by the following gradient-alignment-based curriculum strategy:

<!-- formula-not-decoded -->

In the following theorem, we demonstrate the effectiveness of employing the above curriculum strategy in accelerating the convergence of the REINFORCE agent.

Theorem 1. Consider Algorithm 1 with the REINFORCE learner model and the curriculum strategy defined in Eq. (2). Then, after t = O (log 1ϵ ) steps, we have:

<!-- formula-not-decoded -->

Subsequently, building on the curriculum strategy outlined in Eq. (2), we devise an intuitive curriculum strategy through an analysis of the curriculum objective ⟨gt(c), gt(ctarg)⟩ within the context of a contextual bandit setting.

Further simplification of gradient alignment. We consider the REINFORCE learner model with the following policy parameterization: given a feature mapping ϕ : S × C × A → Rd , for any θ ∈ Rd , we parameterize the policy as πθ(a|s, c) = exp(⟨θ,ϕ(s,c,a)⟩) ∑a′ exp(⟨θ,ϕ(s,c,a′ )⟩) , ∀s ∈ S, c ∈ C, a ∈ A. In the following, we consider a specific problem instance of contextual MDP setting. Let Mc be a contextual MDP with a singleton state space S = {s}, and an action space A = {a1, a2}. Any action a ∈ A taken from the initial state s ∈ S always leads to a terminal state. Let r : C → [0, 1] be a mapping from task/context space C to the interval [0, 1]. For any context c ∈ C, we denote the optimal and non-optimal actions for that context as aopt c and anon c , respectively. The contextual reward function is defined as follows: Rc(s, aopt c ) = 1, and Rc(s, anon c ) = 0, for all c ∈ C. Further, we define ψ : C → Rd as ψ(c) := (ϕ(s, c, aopt c ) - ϕ(s, c, anon c )). Subsequently, for the REINFORCE agent operating under the above setting, the following proposition quantifies the objective term of the curriculum strategy as per Eq. (2) at step t:

Proposition 1. For the REINFORCE agent with softmax policy parameterization under the contextual bandit setting described above, we have:

<!-- formula-not-decoded -->

where Zt(c) := V πθt (c) V ∗ (c) · (V ∗ (c) - V πθt (c)) denotes the agent's learning potential on task c at step t.

Our curriculum strategy. Inspired by the above analysis, we propose the following curriculum strategy:

<!-- formula-not-decoded -->

where ψ : C → Rd is a context representation mapping. At step t, the teacher component picks a task ct according to Eq. (3). The curriculum strategy involves the following quantities: A⃝ the agent's learning potential on task c, B⃝ the agent's learning potential on task ctarg, and C⃝ the correlation between the tasks c and ctarg. Term A⃝ enforces the selection of tasks that are neither too hard nor too easy for the current policy, aligning with the ZPD principle. The combined effect of terms B⃝and C⃝emphasizes the choice of tasks highly correlated with the target task (which has high learning potential). The curriculum strategy effectively balances these two objectives.

## 3.2 Curriculum Strategy for General Settings

In this section, we extend the curriculum strategy in Eq. (3) to practical settings of interest, i.e., a general task space C, a general target distribution µ, and V ∗ (c) values being unknown. We begin by constructing two large discrete sets, ̂ Cunif and ̂ Ctarg, which are subsets of the original task space C. ̂ Cunif is obtained by sampling contexts from C according to uniform distribution, while ̂ Ctarg is obtained by sampling contexts from C according to the target distribution µ. For the general setting, we consider the following curriculum strategy:

<!-- formula-not-decoded -->

Next, we replace V ∗ (·) with Vmax, i.e., the maximum possible value that can be achieved for any task in the task space - this value can typically be obtained for a given domain. Further, when training deep RL agents, allowing some stochasticity in task selection is useful. In particular, the argmax selection can be problematic in the presence of any approximation errors while computing V πθt (·) values. To make the selection more robust, we replace argmax selection with softmax selection and sample (cttarg , ct) from the distribution given below:

<!-- formula-not-decoded -->

(a) Complexity of environments

|              | PM-S:1T     | PM-S:2G              | SGR      | MINIG       | BW                        |
|--------------|-------------|----------------------|----------|-------------|---------------------------|
| Reward       | binary      | binary               | binary   | binary      | dense                     |
| Context      | R3          | R3                   | R3       | {0, 1}8     | R2                        |
| State        | R4          | R4                   | R4       | [0, 255]147 | R24                       |
| Action       | R2          | R2                   | R2       | 7           | R4                        |
| Target Dist. | Single Task | Double-Mode Gaussian | R2 Plane | Single Task | Uniformwith trivial tasks |

![Image](https://ijcai.org/proceedings/2024/fulltext/556/image_000000_2752a6a2fa8bb3a257d589ec0f7917071da7bff8a430c84ac986e10ed442a32c.png)

(b) Illustration of the environments

![Image](https://ijcai.org/proceedings/2024/fulltext/556/image_000001_2caef601b919c342056c9c10fddcebb62dc724de2947ea871522cd67a79ac116.png)

Figure 1: (a) provides a comprehensive overview of the complexity of the environments based on the reward signals, context space, state space, action space, and target distribution. (b) showcases the environments by providing an illustrative visualization of each environment (from left to right): PM-S, SGR, MINIG, and BW.

where β is a hyperparameter and V t (·) values are obtained from the critic network of the RL agent to estimate V πθt (·). Finally, the teacher component samples (cttarg , ct) from the above distribution and provides the task ct to the student component - we refer to this selection strategy as PROCURL-TARGET.

## 4 Experimental Evaluation

In this section, we validate the effectiveness of our curriculum strategy by conducting experiments in environments selected from the state-of-the-art works of [Klink et al., 2022] and [Romac et al., 2021]. We utilize the PPO method from the Stable-Baselines3 library for policy optimization [Schulman et al., 2017; Raffin et al., 2021]. The implementation details of different curriculum strategies are provided in the longer version of the paper.

## 4.1 Environments

In our evaluation, we examine four distinct environments detailed in the following paragraphs. These environments are selected to showcase the utility of our curriculum strategy in diverse settings, such as in procedural task generation or with image-based observations, and its effectiveness in handling target distributions with varying characteristics within the context space C. For the first environment, Point Mass Sparse (PM-S), we consider two settings. In one setting, the target is concentrated on a single context c ∈ C, a similar setting as analyzed in Section 3.1. In the second setting, the target distribution exhibits multiple modalities. The second environment, Sparse Goal Reaching (SGR), features target distributions with uniform coverage over specific dimensions of the context space and concentrated on one dimension. The third environment, MiniGrid (MINIG), uniquely features a discrete context space and additionally has an image-based state space. Lastly, a fourth environment, Bipedal Walker Stump tracks (BW), has a uniform target distribution spanning the entirety of the context space. Moreover, it shows the applicability of our technique in the procedural task generation domain. A summary and illustration of these environments are presented in Figure 1.

Point Mass Sparse (PM-S). Based on the work of [Klink et al., 2020], we consider a contextual PM-S environment where an agent navigates a point mass through a gate of a given size towards a goal in a two-dimensional space. To heighten the challenge, we replace the original dense reward function with a sparse one, a strategy also considered in [Tzannetos et al., 2023]. Specifically, in the PM-S environment, the agent operates within a goal-based reward setting where the reward is binary and sparse, i.e., the agent receives a reward of 1 only upon successfully moving the point mass to the goal position. The parameters governing this environment, such as the gate's position, width, and the ground's friction coefficient, are controlled by a contextual variable c ∈ C ⊆ R3 . This variable comprises CGatePosition, C-GateWidth, and C-Friction. Our experimental section explores two distinct PM-S environment settings. In the first setting, denoted as PM-S:2G, the target distribution µ takes the form of a bimodal Gaussian distribution. Here, the means of the contextual variables [C-GatePosition, C-GateWidth] are set to [-3.9, 0.5] and [3.9, 0.5] for the two modes, respectively. In the second setting, PM-S:1T, the target distribution µ is concentrated on a single context c ∈ C. More precisely, the contextual variables [CGatePosition, C-GateWidth, C-Friction] take on the following values: [0.9, 0.5, 3.5]. To construct our training tasks, we draw 20000 contexts from C using a uniform distribution, forming ̂ Cunif. The set ̂ Ctarg is created by sampling 400 contexts from C according to the target distribution µ. We employ a held-out set sampled from the target µ for evaluation.

![Image](https://ijcai.org/proceedings/2024/fulltext/556/image_000002_9e6e8742ea098045389cea0d1db0b8ffbce6827d0493f9b9799b56701586d30a.png)

![Image](https://ijcai.org/proceedings/2024/fulltext/556/image_000003_d8cd195b83c453af1cb23971205f00502fced589469c7e0eb653ad1529654c59.png)

Sparse Goal Reaching (SGR). Based on the work of [Klink et al., 2022], we consider a sparse-reward, goalreaching environment in which an agent needs to reach a desired position with high precision. Such environments have previously been studied by [Florensa et al., 2018]. Within this environment, the contexts, denoted as c ∈ C ⊆ R3 , encode both the desired 2D goal position and the acceptable tolerance for reaching that goal. Our primary objective centers around achieving as many goals as possible with high precision, indicated by a low tolerance threshold. In this regard, the target distribution µ takes the form of a uniform distribution, but it is restricted to a specific 2D region within C where the tolerance (C-Tolerance) for each context is set at a minimal value of 0.05. Additionally, the presence of walls within the environment renders many of the tasks specified by C infeasible, necessitating the identification of a feasible task subspace. We generate our training tasks by randomly selecting 9900 contexts from C using uniform distribution to create ̂ Cunif, and by selecting 100 contexts according to the target distribution µ to form ̂ Ctarg. For evaluation, we employ a separate held-out set sampled from the target distribution µ.

MiniGrid (MiniG). We establish the MINIG environment by assembling six diverse Minigrid environments from [Chevalier-Boisvert et al., 2023]: Crossing, Dynamic Obstacles, Four Rooms, Unlock, Unlock Pickup, and Blocked Unlock Pickup. Each environment presents a unique mission, demanding distinct skills such as navigation, goalreaching, lava avoidance, moving obstacle avoidance, key picking, door unlocking, object picking, and door unblocking. These skills define the discrete context space {0, 1}8 of MINIG, with each environment requiring a specific subset of skills for a successful resolution. The type Blocked Unlock Pickup is chosen as the target environment due to its inherent difficulty, making it challenging to solve without a curriculum. Additionally, MINIG includes environments like Crossing and Dynamic Obstacles, featuring skills not pertinent to the target mission. The state space comprises observed images of the grid world, and the action space is discrete. The reward is set at 1 for successful mission completion and 0 otherwise. For training tasks, we select 1000 instances from all six environment types. The first three types (Crossing, Dynamic Obstacles, and Four Rooms) collectively contribute to 75% of the training tasks, while the remaining three types, including samples from the target environment, equally constitute the remaining 25%.

Bipedal Walker Stump Tracks (BW). We conduct additional experiments within the TeachMyAgent benchmark for curriculum techniques, as introduced in [Romac et al., 2021]. In this context, we chose a bipedal agent tasked with walking in the Stump Tracks environment, which is an extension of the environment initially proposed in [Portelas et al., 2019]. The state space comprises lidar sensors, head position, and joint positions. The action space is continuous, and the goal is to learn a policy that controls the torque of the agent's motors. The walker is rewarded for going forward and penalized for torque usage. An episode lasts 2000 steps and is terminated if the agent reaches the end of the track or if its head collides with the environment (in which case a reward of -100 is received). Within this environment, the contextual variables c ∈ C ⊆ R2 control the height (C-StumpHeight) and spacing (C-StumpSpacing) of stumps placed along the track for each task. Our experimental setup is equivalent to the bipedal walker stump track environment with mostly trivial tasks, as described in [Romac et al., 2021]. In this setup, C-StumpHeight is constrained to the range [-3; 3], while C-StumpSpacing remains within [0; 6]. Notably, the environment enforces the clipping of negative values for CStumpHeight, setting them to 0. Consequently, half of the tasks have a mean stump height of 0, introducing a significant proportion of trivial tasks (50%). To address the procedural task generation, we randomly draw 1000 tasks from C to construct the training task set, denoted as ̂ Cunif. Additionally, every four epochs, we resample 1000 tasks and update the training set ̂ Cunif. The set ̂ Ctarg is obtained by sampling 500 tasks from C according to the target distribution µ, which is uniform in C.

## 4.2 Curriculum Strategies Evaluated

Variants of our curriculum strategy. We consider two curriculum strategies as described next. First, PROCURLTARGET is based on Eq. (4). Throughout all the experiments, we use the following choice to compute the similarity between ψ(s) and ψ(ctarg): exp(-∥c - ctarg∥2 ). Second, PROCURL-UNIF is a variant of it that does not take into account the target distribution µ and hence ignores the correlations. Specifically, PROCURL-UNIF drops the target task-related terms B⃝ and C⃝ derived in Eq. (3), and selects the next task according to the following distribution: P[ct = c] ∝ exp (β · V t (c) Vmax · (Vmax - V t (c))). We note that this strategy is similar to a ZPD-based curriculum strategy proposed in [Tzannetos et al., 2023] for uniform performance objectives.

State-of-the-art baselines. SPDL [Klink et al., 2020], CURROT [Klink et al., 2022], PLR [Jiang et al., 2021b], and GRADIENT [Huang et al., 2022] are state-of-the-art curriculum strategies for contextual RL. We adapt the implementation of an improved version of SPDL, presented in [Klink et al., 2021], to work with a discrete pool of contextual tasks. PLR [Jiang et al., 2021b] was originally designed for procedurally generated content settings, but we have adapted its implementation for the contextual RL setting operating on a discrete pool of tasks.

Prototypical baselines. We consider two prototypical baselines: IID and TARGET. The IID strategy samples the next task from C with a uniform distribution, while the TARGET strategy samples according to the target distribution µ.

## 4.3 Results

Convergence behavior. As illustrated in Figure 2, the RL agents trained using our curriculum strategy, PROCURLTARGET, perform competitively w.r.t. those trained with state-of-the-art and prototypical baselines. For PM-S:1T, in Figure 2a, we observe that PROCURL-TARGET quickly succeeds in the single target task compared to the other techniques. Although CURROT and GRADIENT converge slower, they finally perform similarly to the proposed technique. The results for PM-S:2G are presented in Figure 2b, where we can observe that PROCURL-TARGET, CURROT and GRADIENT outperform the other strategies. PROCURL-TARGET demonstrates success in handling bimodal target distributions by alternating the selection between the modes of the target distribution. Although it initially has a slower performance than PROCURL-UNIF and CURROT, it quickly matches and surpasses their performance. Despite PROCURL-UNIF not explicitly considering the target distribution in its formulation, it progressively selects more challenging contexts and effectively encompasses the tasks from the target distribution in this scenario. In Figure 2c for SGR, PROCURL-TARGET outperforms all the other techniques. PROCURL-TARGET selects tasks that are neither too hard nor too easy for the agent's current policy and are also correlated with the target distribution. CURROT stands out among other strategies due to its ability to gradually choose tasks from the target distribution. Importantly, solely selecting target contexts for training is inadequate, as evidenced by the underperformance of TARGET compared to all other techniques. Similarly, in Figure 2d for MINIG, PROCURL-TARGET outperforms all the other techniques by a large margin. For BW, Figure 2e, where the target distribution is uniform, PROCURL-TARGET and PROCURLUNIF achieve the best performance. Although, PROCURLUNIF, by definition, considers a uniform performance objective, PROCURL-TARGET is capable of succesfully handling a uniform target distribution.

Figure 2: Performance comparison of RL agents trained using different curriculum strategies. The performance is measured as the mean return (±1 standard error) on the test pool of tasks. The results are averaged over 25 random seeds for PM-S:1T, 25 random seeds for PM-S:2G, 10 random seeds for SGR, 20 random seeds for MINIG, and 10 random seeds for BW. The plots are smoothed across 2 evaluation snapshots that occur over 25000 training steps.

![Image](https://ijcai.org/proceedings/2024/fulltext/556/image_000004_b9da1a989a7f7918cba570142051e81d50504e4507d9c917816133535c63983f.png)

Curriculum plots. Figures 3a and 3b display the average distance between the target and the contexts selected from PROCURL-TARGET, CURROT, IID, and TARGET. We observe that PROCURL-TARGET and CURROT manage to reduce the average context distance below that of IID, indicating that both techniques gradually prioritize tasks that align with the target. However, it is noteworthy that CURROT continues to decrease the context values to reach the target. Whereas PROCURL-TARGET, after succeeding on the target, returns closer to IID sampling. Figure 3c provides a visual representation of the two-dimensional context space for the PM-S:2G setting. The curriculum initially starts from larger C-GateWidth values and centered C-GatePosition values, gradually shifting towards the two modes of the target in the later stages of training. In Figure 3d, we display the average C-Tolerance of selected tasks in SGR. Our findings indicate a consistent trend with Figures 3a and 3b. Both PROCURL-TARGET and CURROT reduce the average C-Tolerance below that of IID. However, PROCURLTARGET does not necessarily converge to the target. Conversely, CURROT persists in reducing the context values to attain convergence with the target. In Figure 3e, we depict the two-dimensional context space for the BW setting. Despite the uniformity of the target distribution of contexts, we observe that in the later stages of training, PROCURL-TARGET disregards trivial tasks characterized by C-StumpHeight values smaller than 0. Instead, it focuses on tasks from the remaining task space.

Figure 3: (a-b) present the average distance between the selected contexts C-GatePosition and C-GateWidth and the target distribution for PM-S:2G. (c) presents the two-dimensional context space of PM-S:2G. The target distribution is depicted as a black x and encodes the two gates with C-GateWidth = 0.5 at C-GatePosition = {-3.9, 3.9}. Each colored dot represents the context/task selected by PROCURLTARGET during training, where brighter colors indicate later training stages. (d) presents the average C-Tolerance of the selected tasks during different curriculum strategies for SGR. (e) presents the two-dimensional context space of BW. The target distribution is uniform. Each colored dot represents the context/task selected by PROCURL-TARGET during training, where brighter colors indicate later training stages.

![Image](https://ijcai.org/proceedings/2024/fulltext/556/image_000005_589ed31b7d22bdaf726b1f71b25cfb76d86134ba88269deec6f2da81c03c44f6.png)

## 5 Concluding Discussions

We proposed a novel curriculum strategy that strikes a balance between selecting tasks that are neither too hard nor too easy for the agent while also progressing the agent's learning toward the target distribution by utilizing task correlation. We mathematically derived our curriculum strategy through an analysis of a specific learning scenario and demonstrated its effectiveness in various environments through empirical evaluations. Here, we discuss a few limitations of our work and outline a plan on how to address them in future work. First, it would be interesting to extend our curriculum strategy to high-dimensional context spaces in sparse reward environments. However, sampling new tasks in such environments poses a significant challenge due to the estimation of the value of all tasks in the discrete sets ̂ Cunif and ̂ Ctarg. Second, while our curriculum strategy uses a simple distance measure to capture task correlation, it would be worthwhile to investigate the effects of employing different distance metrics over the context space on curriculum design.

## Acknowledgments

Parameswaran Kamalaruban acknowledges support from The Alan Turing Institute. Funded/Co-funded by the European Union (ERC, TOPS, 101039090). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.

## References

1. [Andrychowicz and others, 2017] Marcin Andrychowicz et al. Hindsight Experience Replay. In NeurIPS, 2017.
2. [Baranes and Oudeyer, 2013] Adrien Baranes and Pierre- Yves Oudeyer. Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots. Robotics and Autonomous Systems, 61(1):49-73, 2013.
3. [Bengio et al., 2009] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum Learn- ing. In ICML, 2009.
4. [Chaiklin, 2003] Seth Chaiklin. The Zone of Proximal De- velopment in Vygotsky's Analysis of Learning and In- struction. Vygotsky's Educational Theory in Cultural Con- text, page 39, 2003.
5. [Chevalier-Boisvert et al., 2023] Maxime Chevalier- Boisvert et al. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR, abs/2306.13831, 2023.
6. [Dendorfer et al., 2020] Patrick Dendorfer, Aljosa Osep, and Laura Leal-Taixé. Goal-GAN: Multimodal Trajectory Pre- diction based on Goal Position Estimation. In ACCV, 2020.
7. [Dennis et al., 2020] Michael Dennis, Natasha Jaques, Eu- gene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design. In NeurIPS, 2020.
8. [Eimer et al., 2021] Theresa Eimer, André Biedenkapp, Frank Hutter, and Marius Lindauer. Self-Paced Context Evaluation for Contextual Reinforcement Learning. In ICML, 2021.
9. [Elman, 1993] Jeffrey L Elman. Learning and Development in Neural Networks: The Importance of Starting Small. Cognition, 48(1):71-99, 1993.
10. [Florensa et al., 2017] Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse Curriculum Generation for Reinforcement Learning. In CORL, 2017.
11. [Florensa et al., 2018] Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic Goal Generation for Reinforcement Learning Agents. In ICML, 2018.
12. [Graves et al., 2017] Alex Graves, Marc G Bellemare, Ja- cob Menick, Rémi Munos, and Koray Kavukcuoglu. Au- tomated Curriculum Learning for Neural Networks. In ICML, 2017.
13. [Hallak et al., 2015] Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual Markov Decision Processes. CoRR, abs/1502.02259, 2015.
14. [Huang et al., 2022] Peide Huang, Mengdi Xu, Jiacheng Zhu, Laixi Shi, Fei Fang, and Ding Zhao. Curriculum Re- inforcement Learning using Optimal Transport via Grad- ual Domain Adaptation. In NeurIPS, 2022. Jiang et al., 2015]
15. Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander G Hauptmann. Self-Paced Curriculum Learning. In AAAI, 2015.
16. [Jiang et al., 2021a] Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Replay-Guided Adversarial Environ- ment Design. In NeurIPS, 2021.
17. [Jiang et al., 2021b] Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized Level Replay. In ICML, 2021.
18. [Kamalaruban et al., 2019] Parameswaran Kamalaruban, Rati Devidze, Volkan Cevher, and Adish Singla. Inter- active Teaching Algorithms for Inverse Reinforcement Learning. In IJCAI, 2019.
19. [Kirk et al., 2021] Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A Survey of Gen- eralisation in Deep Reinforcement Learning. CoRR, abs/2111.09794, 2021.
20. [Klink et al., 2020] Pascal Klink, Carlo D'Eramo, Jan R Pe- ters, and Joni Pajarinen. Self-Paced Deep Reinforcement Learning. In NeurIPS, 2020.
21. [Klink et al., 2021] Pascal Klink, Hany Abdulsamad, Boris Belousov, Carlo D'Eramo, Jan Peters, and Joni Pajarinen. A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement Learning. JMLR, 22:182- 1, 2021.
22. [Klink et al., 2022] Pascal Klink, Haoyi Yang, Carlo D'Eramo, Jan Peters, and Joni Pajarinen. Curriculum Re- inforcement Learning via Constrained Optimal Transport. In ICML, 2022.
23. [Kumar et al., 2010] M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-Paced Learning for Latent Variable Models. In NeurIPS, 2010.
24. [Levine et al., 2016] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end Training of Deep Visuomotor Policies. JMLR, 17(1):1334-1373, 2016.
25. [Lillicrap et al., 2015] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous Control with Deep Reinforcement Learning. CoRR, abs/1509.02971, 2015.
26. [Liu et al., 2017] Weiyang Liu, Bo Dai, Ahmad Humayun, Charlene Tay, Chen Yu, Linda B Smith, James M Rehg, and Le Song. Iterative Machine Teaching. In ICML, 2017.
27. [Matiisen et al., 2019] Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher-Student Curriculum Learning. IEEE Transactions on Neural Networks and Learning Systems, 31(9):3732-3740, 2019.
28. [Mnih et al., 2015] Volodymyr Mnih et al. Human-Level Control Through Deep Reinforcement Learning. Nature, 518(7540):529-533, 2015.
29. [Modi et al., 2018] Aditya Modi, Nan Jiang, Satinder Singh, and Ambuj Tewari. Markov Decision Processes with Con- tinuous Side Information. In ALT, 2018.
30. [Narvekar and Stone, 2019] Sanmit Narvekar and Peter Stone. Learning Curriculum Policies for Reinforcement Learning. In AAMAS, 2019.
31. [Narvekar et al., 2020] Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum Learning for Reinforcement Learn- ing Domains: A Framework and Survey. JMLR, 21:1-50, 2020.
32. [Oudeyer et al., 2007] Pierre-Yves Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic Motivation Systems for Autonomous Mental Development. IEEE Transactions on Evolutionary Computation, 11(2):265-286, 2007.
33. [Parker-Holder et al., 2022] Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foer- ster, Edward Grefenstette, and Tim Rocktäschel. Evolving Curricula with Regret-Based Environment Design. CoRR, abs/2203.01302, 2022.
34. [Portelas et al., 2019] Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yves Oudeyer. Teacher algorithms for curriculum learning of Deep RL in continuously pa- rameterized environments. In CoRL, 2019.
35. [Portelas et al., 2021] Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer. Auto- matic Curriculum Learning for Deep RL: A Short Survey. In IJCAI, 2021.
36. [Racanière et al., 2020] Sébastien Racanière, Andrew K Lampinen, Adam Santoro, David P Reichert, Vlad Firoiu, and Timothy P Lillicrap. Automated Curricula Through Setter-Solver Interactions. In ICLR, 2020.
37. [Raffin et al., 2021] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-Baselines3: Reliable Reinforcement Learning Implementations. JMLR, 22(268):1-8, 2021.
38. [Riedmiller et al., 2018] Martin A Riedmiller et al. Learning by Playing Solving Sparse Reward Tasks from Scratch. In ICML, 2018.
39. [Romac et al., 2021] Clément Romac, Rémy Portelas, Katja Hofmann, and Pierre-Yves Oudeyer. Teachmyagent: a Benchmark for Automatic Curriculum Learning in Deep RL. In ICML, 2021.
40. [Salimans and Chen, 2018] Tim Salimans and Richard Chen. Learning Montezuma's Revenge from a Single Demonstration. CoRR, abs/1812.03381, 2018.
41. [Schmidhuber, 2013] Jürgen Schmidhuber. Powerplay: Training an Increasingly General Problem Solver by Con- tinually Searching for the Simplest Still Unsolvable Prob- lem. Frontiers in Psychology, 4:313, 2013.
42. [Schulman et al., 2017] John Schulman, Filip Wolski, Pra- fulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. CoRR, abs/1707.06347, 2017.
43. [Silver et al., 2017] David Silver et al. Mastering the Game of Go Without Human Knowledge. Nature, 550(7676):354-359, 2017. Sukhbaatar et al., 2018] Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Robert Fergus. Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play. In ICLR, 2018.
44. [Sutton et al., 1999] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy Gradient Methods for Reinforcement Learning with Function Ap- proximation. In NeurIPS, 1999.
45. [Turchetta et al., 2020] Matteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause, and Alekh Agarwal. Safe Reinforcement Learning via Curriculum Induction. In NeurIPS, 2020.
46. [Tzannetos et al., 2023] Georgios Tzannetos, Bar- bara Gomes Ribeiro, Parameswaran Kamalaruban, and Adish Singla. Proximal Curriculum for Reinforcement Learning Agents. TMLR, 2023.
47. [Vygotsky and Cole, 1978] Lev Semenovich Vygotsky and Michael Cole. Mind in Society: Development of Higher Psychological Processes. Harvard University Press, 1978.
48. [Weinshall et al., 2018] Daphna Weinshall, Gad Cohen, and Dan Amir. Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks. In ICML, 2018.
49. [Wöhlke et al., 2020] Jan Wöhlke, Felix Schmitt, and Herke van Hoof. A Performance-Based Start State Curriculum Framework for Reinforcement Learning. In AAMAS, 2020.
50. [Wu and Tian, 2016] Yuxin Wu and Yuandong Tian. Train- ing Agent for First-Person Shooter Game with Actor- Critic Curriculum Learning. In ICLR, 2016.
51. [Yengera et al., 2021] Gaurav Raju Yengera, Rati Devidze, Parameswaran Kamalaruban, and Adish Singla. Curricu- lum Design for Teaching via Demonstrations: Theory and Applications. In NeurIPS, 2021.
52. [Zhang et al., 2020] Yunzhi Zhang, Pieter Abbeel, and Ler- rel Pinto. Automatic Curriculum Learning Through Value Disagreement. In NeurIPS, 2020.
53. [Zhou and Bilmes, 2018] Tianyi Zhou and Jeff Bilmes. Min- imax Curriculum Learning: Machine Teaching with Desir- able Difficulties and Scheduled Diversity. In ICLR, 2018.
54. [Zhu et al., 2018] Xiaojin Zhu, Adish Singla, Sandra Zilles, and Anna N. Rafferty. An Overview of Machine Teaching. CoRR, abs/1801.05927, 2018.
55. [Zou et al., 2019] Xiaotian Zou, Wei Ma, Zhenjun Ma, and Ryan S Baker. Towards Helping Teachers Select Optimal Content for Students. In AIED, 2019.