Active Learning within Constrained Environments through Imitation of an Expert Questioner

1 Introduction

Active learning (AL) agents are intended to learn from an oracle, often assumed to be human, but typically not designed for more realistic human environments. Understanding environmental context however is especially important for robotic agents, generally assumed to be colocated in the environment with the oracle or teacher. Within the robotics community, there has been AL work aimed at understanding [Cakmak et al., 2010; Knox et al., 2013; Gonzalez-Pacheco et al., 2018; Bullard et al., 2018a], modeling [Rosenthal and Veloso, 2011; Racca and Kyrki, 2018], and improving [Chao et al., 2010] interaction with a human partner. An important aspect of the interactive learning problem, this body of work focuses on interaction with the teacher, but there still remains the open question of how the learner should integrate reasoning about the environment in which it is situated.

Specifically, external constraints imposed on the learner may have direct implications for solving the learning problem. For example, a teacher has only a limited time frame of availability or limited cognitive resources that can be devoted to answering the learner's questions. This information may need to influence the learner's questioning policy. However, the problem of trading off learning goals with environmental constraints is relatively unexplored within AL literature, particularly when considering dynamic environments. Yet this problem is important for learning in realistic human settings.

∗Contact Author

In this work, we investigate the question of how to enable an active learner to reason about its learning objectives within a dynamically changing environment while concurrently considering time and resource constraints provided for solving the learning problem. We use a decision-theoretic approach to active learning, whereby the individual decision criteria (or decision features) within the objective function are hand designed and include both task-centric and environment-centric features. Nonetheless, since the learning agent must consider multiple and diverse decision criteria, it becomes difficult to manually tune the individual objectives. Thus we propose imitation of an expert questioner for learning to weight the decision features. Our approach employs Inverse Reinforcement Learning (IRL) for inferring weights of the objective function from demonstrations of an expert policy.

In the experiments conducted, the agent is given a concept learning problem that it must use active learning to solve, under different environmentally constrained conditions. This work makes the following contributions:

• first AL work to reason about environmental constraints within the objective function of the learner
• first AL work to use imitation learning for mimicking the policy of an expert questioner

We evaluate efficacy using two separate task datasets and show that environmentally-aware reasoning allows our algorithm to significantly outperform an established AL baseline of uncertainty sampling and task-centric questioning strategies examined.

2 RelatedWork

Active Learning encompasses an extensive body of literature, spanning across several problem domains. We focus here on the most relevant work within the broader space, active learning for robots and embodied artificial agents. Most literature in AL for robots solves learning problems directly relevant to robotics domains: learning an expert policy to derive desirable robot behavior [Chernova and Veloso, 2009; Lopes et al., 2009; Kroemer et al., 2010; Cakmak and Thomaz, 2012; Daniel et al., 2014; Basu et al., 2018], inferring sequencing constraints on actions in a task [Hayes and Scassellati, 2014], and grounding task-relevant symbols or descriptions [Chao et al., 2010; Kulick et al., 2013; Thomason et al., 2017; Bullard et al., 2018b]. A key limitation however is current approaches only reason about the learning problem the agent must solve, but not time and resource constraints imposed by the environment in which the agent is situated. In other words, in prior literature, the learner typically uses an objective function to reason about its learning goals but does not additionally consider environmental constraints.

The most relevant prior work explored autonomous arbitration between multiple types of active learning queries, acquiring both feature and instance input from the teacher, and situated in dynamic environments [Bullard et al., 2018b]. A primary contribution of this work was an algorithm for arbitration between diverse query types that could also adapt the frequency of its questioning to the rate at which objects changed in its environment. Nonetheless, a key limitation is the inability to reason about constraints imposed by the teacher or learning environment. Thus, while able to adapt its questioning strategy to the rate of environmental change, the learner has no mechanism for adapting its strategy to the query budget given or amount of learning time allocated.

3 ProblemFormulation and Approach

Symbol (or concept) grounding is the problem of mapping symbolic representations to constructs in the physical world [Harnad, 1990]. Specifically, the agent must solve a tasksituated concept grounding problem, whereby it is given abstract task-relevant concepts to be perceptually grounded in its environment, in a way appropriate for the task. For example, when learning concepts for the serve breakfast task, eggs scrambled or sunny-side up would be a more appropriate grounding than a dozen cartoned eggs.

We formalize the problem as follows: Given a set of objects X from a scene in the agent's purview taken at time t, each object instance x ∈ X is represented by a feature vector xt =< f t 1 ... f t m >. We assume the agent has both exteroceptive and proprioceptive sensors for perceiving its external environment and internal state. Each object instance then is modeled by the superset of features F extracted from the agent's sensors at t (e.g. object height or color, position of robot base or end effector). A set of binary classifiers, one for each symbol y ∈ Y, the set of object symbols, each take as input an instance x and produce a degree of confidence p(y|x) = [0, 1] that x has label y. For each symbol, a Gaussian Process Classifier was trained. This representation was selected because it both probabilistically models agent uncertainty and learns well from sparse data.

In dynamic environments, groundings also change over time and concept models must be refined accordingly. This may include change in the physical object state (e.g. eggs going from being in a shell to scrambled) or objects being replaced within the same category (e.g. breakfast beverage being served one day as coffee in a mug and another as orange juice in a glass). Since it is unreasonable to expect a human partner to track the agent's knowledge over time, in a changing environment, we take an active concept grounding approach.

3.1 Active Learning for Concept Grounding

Active Learning enables a learner to query an oracle or teacher for information about which is has uncertainty. It typically assumes a query will be made at every turn and seeks to equip the learner with a utility function for selecting an optimal query [Settles, 2012]. However, real world environments often do not allow the learner unlimited queries or time for querying, and simultaneously change over time. This means it is not always the best use of time and resources to make a query at every time step, until the query budget is depleted. Thus, employing a traditional AL strategy may not maximize learning in dynamically changing, constrained environments, as shown by [Bullard et al., 2018b].

We present a decision-theoretic AL approach which extends prior work intended for dynamic environments [Bullard et al., 2018b]. In that work, the authors contributed a decision-theoretic framework for arbitrating between multiple types of AL queries, acquiring both informative features and representative training instances from a teacher. Building upon that framework, our approach contributes a model that is able to reason about both the agent's concept learning goals and external time and resource constraints imposed on the agent. Specifically, the objective function of the learner is expanded to include decision criteria which reason about environmental context. Equation 1 shows the learner's objective function used at each turn t to assess the expected utility (EU) of an action a, given the current learning state st .

The learning state at t includes: {estimate of posterior probability distributions of y ∈ Y for all x ∈ X, interaction history, query budget, and teaching time allocation}. The set of candidate actions At consist of demonstration queries for each of the task-relevant concepts [DQ(y) ∀y ∈Y], label queries for each object in the current scene [LQ(x) ∀x ∈X], a feature subset query [FSQ] to identify relevant features for discriminating between task concepts, and a no query action [NQ]. Thus, there are |At | = |Y|+|X|+2 candidate actions from which the agent can choose at each turn t. Additionally, each of the query types is associated with a cost, given a priori. The learner selects an optimal action a∗ as

where

The set of decision features f ∈ F used in computing U(s) comprise the representation for the agent's objective (decision) function and is primarily what distinguishes prior work from the approaches introduced in this work. U(s) is represented as a function of decision features f : S → [0, 1]k , where k is number of decision criteria or individual objectives for which the agent is optimizing.

Baseline Approaches

We employ two AL models from prior literature, as baselines for comparison: a standard uncertainty sampling approach (U-sampling) and a state-of-the-art decision-theoretic approach for arbitrating between diverse query actions (DTiros). Uncertainty sampling algorithms are possibly the most commonly employed class of AL strategies in the literature [Settles, 2012; Fu et al., 2013]. They assume a single hypothesis q and utilize the posterior probability distribution over labels y ∈Y given unlabeled instance x, pq(Y|x), in order to detect outliers or instances closest to a decision boundary. Like other standard AL approaches, they query at every turn, each time requesting a label for a maximally informative instance, based upon predetermined selection criteria. A commonly used metric for uncertainty sampling is prediction entropy: -�y∈Y pq(y|x)log pq(y|x). We employ this as our standard AL baseline (U-sampling).

Decision-theoretic approaches to active learning simulate all possible outcomes of each candidate query action and optimize with respect to future expected utility. This work builds from prior work employing decision theory to arbitrate between diverse types of learning queries, including a supplemental no-query action [Bullard et al., 2018b]. The set of decision features investigated were average classifier discriminability and class distribution uniformity.

Given a set of instances in the agent's purview X and a task-relevant concept y, the classifier discriminability metric assesses the range of probabilities over the set of instances: pq (y|xmax)- pq (y|xmin), where xmax and xmin are the model's prediction of the most and least probable examples of class y, respectively. Range is a standardized metric of statistical dispersion; an average is taken over all y ∈Y. Class distribution uniformity assesses selection bias in the training sample, due to an unrepresentative class distribution. It is a useful decision feature in sparse data environments, as has traditionally been the assumption in Learning from Demonstration settings, where the learner does not have sufficient evidence to confidently infer the underlying distribution of classes. This metric incentivizes the learner to minimize sample selection bias. Given this work also seeks to arbitrate between all action types, we employ this previously published decisiontheoretic objective function, where the number of decision features k = 2, as our state-of-the-art baseline (DT-iros).

Experimental Approaches

We introduce two experimental questioning policies: a learning-centric model intended to improve the state-of-theart (DT-task) and an environmentally-aware active learner (DT-task-env). For the learning-centric model, we propose two additional decision features that we believe improves the performance of the originally published DT-iros algorithm, even before consideration of environmental context: instance variation and label prediction margin, defined by Equations 3 and 4 respectively.

Instance variation is a standardized measure of statistical dispersion. Given a class y and a set of scene instances X, it is a measure of relative standard deviation of the class conditional distribution pq(X|y). Intuitively, it attempts to assess each classifier's ability to recognize variation amongst the set of unlabelled instances.

In the context of concept learning, the class-conditional distribution p(X|y) can be thought of as the likelihood of each unlabelled instance x ∈ X being selected as an example of class y. Given that multiple, diverse instances within a scene may serve as positive examples of a given class, it seems useful to employ decision features which approximate the learner's ability to recognize diversity amongst the set of unlabelled instances in its purview. Because of this, both classifier discriminability and instance variation are measures of statistical dispersion, but along different dimensions. Whereas, classifier discriminability is a measure of statistical dispersion over the likelihood of instances belonging to a class, instance variation quantifies the statistical dispersion over the features values of instances. The former rewards the learner for differentiating between the most prototypical and improbable examples of each class; the latter rewards the learner for recognizing greater variation between instances. Both decision features incentivize the selection of queries which increase the learner's recognition of the underlying diversity that exists within the pool of unlabelled instances.

Given an unlabelled instance x and a distribution over class labels p(Y|x), label prediction margin measures the difference between what the learner predicts to be the most probable label y1 and second most probable label y2. Previously employed in AL literature [Settles, 2012], it is a measure of uncertainty; as the margin increases, the learner is more confident about its prediction. It is computed for all scene instances, then averaged. This decision feature incentivizes accuracy in the class prediction for each unlabelled instance.

Thus, the first decision function proposed in this work (DTtask) subsumes the set of decision features considered by DTiros, considering four learning-centric criteria that each optimize for different aspects of the concept learning problem.

The primary contribution of this work however is in the addition of environmental context into the AL agent's objective function. We introduce the following environmental features:

• query budget consumption - measures the proportion of query budget consumed at turn t, given the query history
• remaining time usage - measures the proportion of allocated time remaining after turn t
• non-query time passed - measures the proportion of consecutive turns no query was made within a sliding time window Tw; here the size of the time window is proportional to the rate of environmental change; it is computed as tNQ = nNQ |Tw| ; tNQ →0 when the learner has just queried and tNQ →1 when the learner has not queried throughout the entire duration of the time window; intuitively this metric is intended to penalize the agent for being too passive, in a dynamically changing environment

The environmentally-aware agent's objective function (DT-task-env) is then composed of a linear combination of seven decision features, a subset of which roughly attempt to estimate progress towards learning goals (i.e. learning task centric) and the remaining features intended to incentivize wise time and resource management (i.e. environment centric). All decision-theoretic learners described can arbitrate between all communicative action types.

Given the different types of decision features being considered however, it is challenging to decipher how to trade them off (e.g. budget consumption versus prediction margin). One key observation is humans can often intuitively reason about decision criteria that are difficult to compare quantitatively; thus, we propose to observe the strategy of a human expert questioner, given the same learning problem, and infer how the expert trades off the given decision criteria.

3.2 Imitating an Expert Questioning Strategy

Imitation learning seeks to efficiently learn desired behavior by mimicking a domain expert [Osa et al., 2018]. Within imitation learning literature, Inverse Reinforcement Learning (IRL) aims to recover the expert's reward (objective) function from demonstrations of a policy [Ng et al., 2000; Abbeel and Ng, 2004]. We employ a state of the art IRL algorithm, maximum entropy IRL [Ziebart et al., 2008], to infer the weights w ∈W of Equation 2, for the active learner's decision features, ∀f ∈ F, as wielded by an expert.

The maximum entropy loss function LME maximizes entropy of distributions over paths followed by the expert, while satisfying the constraint that the learner's decision feature counts should ideally match those of the expert. The problem is formulated as follows: Given fE(t) ∀t ∈ T demo , find an optimal weight vector w such that

subject to the constraint

where fE(t) and fL(t) represent the feature counts of the expert and learner respectively, for a trajectory t. In our problem domain, a trajectory is a sequence of learning states visited and communicative actions taken {s1, a1, s2, a2, ...sT , aT } at each time step t <= T , the maximum number of iterations allowed in a learning episode.

Optimization using the maximum entropy loss LME(w) is equivalent to maximizing the log likelihood of the expert demonstrations [Ziebart et al., 2008; Osa et al., 2018]:

Using this formulation, the gradient of the IRL loss, shown in Equation 7, is the difference between the empirical feature counts (demonstrated by the expert) and the expected feature counts, computed from sample trajectories generated with w.

We used an empirically determined maximum number of iterations as stopping criteria for the IRL algorithm. Weights for the environmentally-aware active learner's objective function were learned offline and tested for generalization in AL episodes under different environmental conditions.

3.3 Learning Episode

Figure 1 shows the high-level flow for the learning system. For each active learning episode conducted, the taskrelevant concepts and questioning strategy are given as input. Within an episode, at each turn t, the agent perceives all objects in its purview, computes its estimate of the posterior probability distributions p(y|x) ∀x,y to update learning state st , determines the set of candidate actions At , computes EU(a|st ) ∀a ∈ At , then takes an optimal action a∗ . The learning episode concludes once t = T .

4 Evaluation

This work explores an AL strategy designed to optimize for environmental constraints and proposes an imitation learning approach for accomplishing this. Toward this end, we test two hypotheses: (1) Reasoning additionally about environmental context can enable an AL agent to adapt its questioning strategy and improve its learning performance under constrained conditions, and (2) Imitation Learning can be used to infer an expert's strategy for managing time and resources allocated to solve a given learning problem, then generalized to other constrained environments. As illustrated in Figure 1, each of the questioning strategies conduct their own learning episodes, during which binary classifiers are trained for all task-relevant classes, based upon information gathered. We use recognition accuracy on hold-out test sets for assessing the concept models learned, given each questioning strategy.

4.1 Experimental Design

In evaluating the AL approaches, we focus experiments on a concept grounding task in a dynamic environment, under different environmentally constrained conditions. We also examine performance on another task for generalization of the learned decision feature weights across tasks. Both concept grounding tasks are given the same four abstract concepts to ground (main dish, snack, fruit, and beverage), but are generated from different object RGB-D datasets and represent different properties of dynamic change.

The prepare-lunch task, the most difficult of the two learning problems, is our focus; it places emphasis on the same objects changing state, as one might expect over the course of the task (e.g. pasta going from being in a box in the pantry to being cooked in a pot to being served in bowl for lunch.) It was extracted from a local RGB-D object dataset focused on state-change. Figure 2 shows an example of the type of dynamic change the learner may expect to see in this task. In the second pack lunchbox task, objects do not change state, but have greater within-category diversity. For example, the fruit class contains apples, oranges, peaches, and pears, and the beverage class contains varieties of both soda and water. This task was extracted from the University of Washington RGBD dataset of common household objects [Lai et al., 2011]. As both tasks are from prior literature, details regarding data collection can be found in [Bullard et al., 2018b].

Since RGB-D datasets were being used for evaluation, in order to create a learning environment that more closely approximates real-world settings, we simulated multi-modal features, representing features extracted from a robot's other sensors 1 . Gaussian noise was added to all simulated features since robot sensor data is typically noisy. We also simulated dynamic change in the environment by sampling a new set of object images at a predetermined rate, to represent the scene changing. At each turn t, O contains only one observation (image) of each object in the scene. To simulate environmental change, the perceptual system generates a new set of observations. Else, it outputs the set of observations from t -1. In the prepare-lunch task, objects can change state.

1 object's location relative to interest points in the environment (e.g. counter top, stove, refrigerator, pantry), the object's location relative to the robot base, absolute location of robot's base in the

Using the complete RGB-D object datasets, we generated five smaller task training data samples and one disjoint holdout test sample for each task. Since the UWdataset is several orders of magnitude larger than the local dataset created, the data sample sizes vary by task. Each of the training and test task samples are 80 images and 40 images respectively for the prepare-lunch task and 3200 images and 800 images respectively for the pack-lunchbox task.

There are four AL algorithms being evaluated on each task: the uncertainty sampling baseline (U-sampling), two task-centric decision-theoretic learners (DT-iros and DTtask), and our experimental environmentally-aware decisiontheoretic approach (DT-task-env), which reasons about both learning objectives and environmental constraints.

For training of DT-task-env decision feature weights, an expert questioner was given a very constrained query budget (15) and time allocation (30 turns) to ground the preparelunch task concepts. During training, the environmental scene changed every 10 turns, and major object state changes took place with each scene change; thus spreading the query budget out over the allocated time period affords the opportunity to acquire a more diverse and representative training sample. This was a key part of the strategy employed by the expert used, which was one of the authors of this paper.

The expert provided three demonstrations of questioning sessions (learning episodes) in the training scenario. As part of the strategy demonstrated, the expert also always requested relevant features for discriminating between concepts (FSQ) early in the learning episode (within the first five turns), focused most queries on the least costly query type, and focused on quickly acquiring representative training examples for each class. During IRL training, the maximum number of iterations was set to 100, and we selected the set of weights w∗ that performed best on the validation set. Qualitatively examining the rollouts associated with w∗ under the training conditions, the behavior of the imitation learner was able to closely match the expert's questioning strategy. Environmental conditions were held constant across demonstrations and IRL training. Values changed for testing were time and budget allocation and frequency of environmental change.

4.2 Results

To test our hypotheses, we first investigated generalization to different environmentally constrained conditions. The goal was to vary time on one axis and resources (query budget) on the other axis. In order to have a feasible stopping point, we

environment, location of the robot's base with respect to the counter top, the robot's joint positions for each arm, pose of the robot's hands, robot's hand states (open vs closed), weight of the object, and max/min/average volume of noise in the environment over duration of learning episode

(c) CONDITION 1: Constrained Time & Budget

(d) CONDITION 2: Constrained Budget

(a) CONDITION 1: Constrained Time & Budget

Figure 4: Pack Lunchbox Task. Shows performance (test accuracy with standard error) on a separate task, under the most constrained experimental condition: both time and query budget constrained.

could not truly allow unlimited time or query budget; however, as defined here, the constrained versus unconstrained parameters denote an order of magnitude difference in allocation. All action types were assigned an a-priori cost, which is 2 for demo queries and feature subset queries, 1 for label queries, and 0 for no query. This can be assigned in any way desired, but for our purposes, was intended to map roughly with the cognitive load required by the teacher in answering a particular type of question. Figure 3 shows learning curves for each combination of time and resource parameters. For each task, learning curves are averaged over 5 runs, each run sampling from a different pre-generated task training data sample. Examining the subfigures: from left to right, time allocation is increased (from 40 to 150) and from bottom to top, query budget is increased (from 25 to 500). Thus fig 3c is the most constrained (budget 25, time 40) and most similar to the training scenario (budget 15, time 30). Overall, the left half roughly corresponds to the agent being given approximately 20 questions, assuming the most costly queries are minimized, whereas the right half corresponds to the agent being given unlimited queries during the time allocated. In testing, once a strategy has exceeded its query budget, it is no longer allowed to make queries, representing complete resource consumption. Thus for the remainder of the episode, it must select the no-query action at no cost. It should also be stated that we assume at most one query per turn.

We used the Mann-Whitney U-test to perform pairwise statistical comparisons of our experimental approach (DT-taskenv) with each of the baseline approaches at the end of the allocated learning time, for all four experimental conditions. A one-tailed test was conducted, as the goal was to understand if the experimental environmentally-aware approach improves performance over the baseline task-centric approaches to AL. In conditions 1, 3, and 4, DT-task-env statistically outperforms all other approaches by the end of the learning time. In condition 2, it statistically outperforms all approaches except DT-task, compared to which it performs equivalently. Using the second (pack-lunchbox) task, we were able to replicate our results in another domain. We only show the most constrained condition (limited time and query budget) in Figure 4. Overall, there is a clear pattern. DT-task-env always performs at least as well as the task-centric baselines but moreover dominates task-centric approaches under most of the environmental conditions examined. This confirms our second hypothesis that a questioning strategy learned through imitation of an expert in one environment can be used to generalize to other constrained environments. Our first hypothesis however is not supported by findings from Condition 2.

Figure 5: Questioning Behavior of each Strategy in Prepare-Lunch Task for one training sample. Dots indicate when a query is made.

To better understand what behavior leads to these findings, we analyze learning episodes from one training data sample under two different experimental conditions in Figure 5. The dots represent points where queries were made for the given strategy. We find that given a limited query budget and ample time (figure 5b), DT-task and DT-task-env employ very similar conservative strategies. Both use most of their budget closer to the beginning of the episode, as they attempt to build initial concept models, but also attempt to modulate budget consumption with rate of environmental change. However, when the agent is allowed to ask unlimited queries given the same time frame (figure 5a), these two strategies behave very differently. Whereas the task-centric learners employ exactly the same strategy (because they have no ability to reason about environmental constraints), DT-task-env employs a very liberal strategy. In fact, it makes at least an order of magnitude more queries both than DT-task and DT-iros (134 versus 22 and 9), largely accounting for its complete domination over the other strategies. Also notably, U-sampling asks a question at every turn until it exceeds its budget, since it is able to modulate neither for environmental change nor for external constraints. It also does not have the capability to autonomously select a feature subset query, so it must rely upon computational feature selection for solving its learning problem. The decision-theoretic approaches, by contrast, are able to reason about and request an FSQ early in the episode, making them significantly more sample efficient.

The key implication of all of the experimental findings is that the DT-task-env strategy has the ability to effectively adapt its questioning behavior both to the rate of environmental change (like DT-iros and DT-task) and to time and resource constraints imposed externally. In more realistic environments, this is compelling as it gives human partners the capability to specify their own time and cognitive load constraints, with the understanding that the agent can integrate this knowledge into its reasoning about the learning task.

5 Discussion

This work contributes a new cognitive capability for active learners in more realistic contexts, by enabling them with a policy to trade off learning objectives with environmental constraints. Yet, it is not without its limitations. The properties of environmental change captured by the object datasets used, serve only as a proxy for partial observability encountered in the real world. In realistic environments, scenes are often cluttered and scene change happens continuously.

Additionally, the active learning models contributed are intended for use by any type of artificial agent and thus agnostic to an agent's embodiment. However, what an agent decides to ask cannot always be decoupled from how it must use its embodiment to execute the query. For example, query cost may increase if a query requires fine-grained manipulation that is difficult for the agent to maneuver or simply takes a longer time to generate than other queries. Ideally, this should be incorporated into the agent's decision function.

Future work could explore how to adapt these methods, given more complex perceptual data or agent embodiment as input for the questioning framework.

6 Conclusion

This work contributed a first exploration and novel computational approach for solving the problem of active learning under externally imposed time and resource constraints. Imitation of an expert questioner's policy was used to learn to weight the diverse set of decision criteria for an environmentally-aware active learner. Experiments were conducted under various environmentally constrained conditions and on two concept learning tasks. Key findings show the experimental approach presented statistically outperformed a standard uncertainty sampling baseline and the strictly taskcentric active learners under most environmental conditions; thus representing a promising alternative for active learners in more realistic human environments.