Interaction Considerations in Learning from Humans

1 Introduction

Intelligent agents such as self-driving cars, recommendation engines, and assistive devices are becoming fixtures in everyday society due to their growing ability to learn from large-scale data and to personalize based on data from individuals. Approaches to collecting data from people include asking for annotations on video and images [Real et al., 2017], ratings of behavior [Daniel et al., 2015], task demonstrations [Abbeel and Ng, 2004], critiques or corrections of proposed trajectories [Cui and Niekum, 2018; Bajcsy et al., 2018], and preferences between options [Sadigh et al., 2017]. Distinctions between these techniques have led to a growing body of work on understanding them relative to each other. Different interaction types can be used in combination to accelerate learning [Palan et al., 2019], better leverage people as teachers [Bullard et al., 2018], and differ in the amount of implicit information they encode [Jeon et al., 2020]. Learning interactions are often selected based on how informative they are for a learner, without examining how human teachers may differently perceive and respond to those interactions. However, people's experiences with different interactions can affect the data they provide.

∗Contact Author

In order to collect high-quality data, whether via active learning or curated training sets, i.e. passive learning, it is necessary to leverage data collection processes that accommodate people's limitations. People provide noisy data, are biased towards providing positive rewards, and get fatigued [Amershi et al., 2014]. Several of the shortcomings in people's teaching capabilities may relate to the fact that as the cognitive load (i.e. the portion of working memory being utilized) on an individual increases, they grow more easily distracted and have worse task performance [Sweller, 1988]. Interaction design can be used to modulate cognitive load in human learners [Chandler and Sweller, 1991]; that is, the way a task is presented can affect how burdensome it is and may ultimately affect the quality of the data it produces.

Our key insights are twofold. First, that we can formalize interactions as a Markov Decision Process (MDP) and taxonomize them based on how data is provided to and acted upon by human teachers. This taxonomy allows us to analyze groups of similar interactions. It also enables us to empirically evaluate our second insight: these groups of interaction types result in differences in human factors such as cognitive load. Such factors are important to characterize, as they may lead to downstream data quality effects.

We present a model-agnostic MDP to formalize interactions, and a taxonomy of interaction types for human-agent learning using four features: the amount of data the user is asked to respond to, the amount of the data the user provides in their response, the granularity of the user's response, and the responses the user can choose from. We analyze current paradigms for learning from people, and find four distinct interaction archetypes: Showing, Categorizing, Sorting, and Evaluating. We also present results from a user study designed to identify differences between interaction types in terms of human factors related to data quality, such as cognitive load, confidence, and subjective usability. We find that Evaluating interaction types, where people identify good behavior, are the most cognitively loading and least usable in both of the study's task domains. Categorizing (i.e., assigning a positive or negative reward) and Showing (i.e., giving demonstrations) are less cognitively loading and more usable.

2 RelatedWork

Quality control research for data collection investigates how users, often crowd workers, can provide good data via incentives and task design [Lease, 2011; Kittur et al., 2013]. Our research differs in that we study how human factors, such as cognitive load and usability, differ between various interaction types. The correlations between cognitive load, usability, and task performance (e.g. providing good data) have been observed and studied throughout cognitive psychology [Sweller, 1988], human-computer [Longo, 2018] and humanrobot interaction [Prewett et al., 2010].

Researchers categorized the questions people ask while learning into three interaction types: labels, demonstrations, and feature queries [Cakmak and Thomaz, 2012]. Additional research into developing learning agents that emulate the rich learning interactions people use has involved combining multiple interaction types [Bullard et al., 2018] to better leverage human teachers, leveraging trade-offs between interactions [Palan et al., 2019], and exploring explicit and implicit information transfer [Jeon et al., 2020].

Our research extends this body of work by providing a general taxonomy of interaction types as well as a principled framework for representing model-agnostic interactions. Based on our taxonomy (justified in Section 3), we discuss four categories of interactions.

Showing. Inverse reinforcement learning (IRL) is a learning from demonstration technique for recovering a reward function from which to train a policy [Ng et al., 2000; Abbeel and Ng, 2004]. Behavioral cloning learns a policy directly [Bain and Sammut, 1995; Schaal, 1999]. While demonstrations can be highly informative, people are limited in the number of examples they can provide and by their expertise.

Categorizing. This type of learning is commonly used for classification and regression. For example, computer vision leverages popular large-scale labeled datasets [Deng et al., 2009; Everingham et al., 2010; Lin et al., 2014]. Labels also include the assignment of rewards to actions, as in bandit problems and reinforcement learning with human feedback [Kober et al., 2013; Daniel et al., 2015]. The informativeness of labels is limited by the size of the label set, and people are known to give both overly positive rewards [Amershi et al., 2014] and shifting ratings [O'Connor and Cheema, 2018].

Sorting. Preference elicitation is an active area of research, especially in recommendation engines [F¨urnkranz and H¨ullermeier, 2011]. Comparison and ranking-based approaches for learning reward functions are increasingly common [Sadigh et al., 2017; Bıyık et al., 2020; Wirth et al., 2017]. These interactions are precise, and thought to be low user effort. The technique is good at fine-tuning, but the information that can be gained from each query is limited.

Evaluating. Corrections are feedback on a proposed set of actions either during or after task execution. These can be physical or simulated [Bajcsy et al., 2018; Jain et al., 2015]. Users can also mark good or bad regions of a trajectory via critiques [Cui and Niekum, 2018]. Credit-assignment interactions, such as the one posed in [Jeon et al., 2020], can be construed as a form of critique where the user is limited to identifying only one good region. Off-switch games can be considered as another special case of critiques where trajectories are segmented into a singular allowed section preceding a singular disallowed section [Hadfield-Menell et al., 2016].

3 Taxonomy of Interaction Types

Many interaction types share fundamental similarities in how people interface with them. By taxonomizing the space of interaction types along those lines, we can more tractably compare clusters of interactions. We construct a novel taxonomy that characterizes interaction types along four dimensions: the action batch size of the learner's queries, the action batch size of the user's responses, the number of intervention opportunities available to the user per query, and the number of response choices available for a user to select from per query. We identify four interaction archetypes, termed: Showing, Categorizing, Sorting, and Evaluating.

3.1 RepresentingModel-Agnostic Interactions

In order to represent interactions, we must first introduce some terminology. Let a query q refer to data that the user is prompted to respond to. An annotation is the feedback that the user gives in response to a query. An interaction type is the format of a query (i.e., "Showing", "Categorizing", "Sorting", or "Evaluating"). An interaction instance is a specific query, e.g. "Should the label be 'library' or 'bookshop'?" Finally, an interaction session is a series of interaction instances of a particular interaction type, and associated annotations.

We choose to model interaction sessions as Markov Decision Processes (MDPs) for several reasons. MDPs provide a sequential decision-making paradigm that captures how, in active and passive learning, people provide a series of annotations over the course of an interaction session. Furthermore, with this paradigm, we can treat human teachers as agents making decisions over their own action and state spaces, rather than as oracles in possession of data that is always equally accessible. This allows us to account for imperfect decision-making due to human factors (i.e. cognitive load, usability). Finally, this enables us to analyze interaction types separately from any underlying learning models. The interaction type is a means to obtain data, and the learning model (e.g. Gaussian process, neural network, Q-learning) consumes that data. This distinction enables us to discuss interaction types in terms of the user's actions and the learner's actions, to analyze both passive and active data collection, and to assess interactions regardless of learning objectives.

We subsequently define this MDP in further detail. First, we define a user U as an agent interacting with a learner L via queries. The interactions between U and L can be situated in passive or active learning contexts. Let AL define the set of actions available to the learner L and at ∈ AL the action taken at time t. For example, in an autonomous driving task, AL consists of available steering controls. Let st ∈ SL denote the state at time t (e.g. the position of the car). A trajectory is a series of state-action pairs, ξ = (st, at)Tt=0 , where T is some finite task horizon. This notation holds for oneshot tasks such as accepting or rejecting an image annotation: s0 is the image, and a0 is the suggested annotation.

Now, we describe an interaction session I as an MDP := (SU,AU, T ,R). Let AU define the set of actions available to the user, e.g. the feedback a user can give in a particular interaction type. For example, in a reward-punishment interaction, AU = {-1, +1}. Note that AU and AL need not be distinct: for demonstrations, the user may have the same action space as the learner. This is sufficient notation for discussing the curation of training sets for passive learning. For active learning, we define the state σi ∈ SU as the parameterization of L at the ith query, as made visible to the user via means such as model weights. This is distinct from the state of the underlying learner's environment st, as discussed previously. The transition T is a property of L (e.g., gradient descent if the model is a neural network), and can be opaque to the user. The reward R : SU ↦→ R minimizes the difference between the desired and true output of L.

3.2 Features of the Taxonomy

Query Size (Actions). The batch size of a query is determined by the number of actions a ∈ AL it contains, and is given by N · T . A query q consists of one or more trajectories ξ with finite time horizon T such that q = {ξ0, .., ξN-1}, where N is the number of options presented to the user. In the reward-punishment interaction we have been referring to, the query presented to the user is a single (N = 1) example of a trajectory: q = {ξ0}. If we were to instead use a preference interaction, the user would select one of two trajectories presented to them, such that N = 2 and q = {ξ0, ξ1}.

Response Size (Actions). The number of actions a ∈ AU that a user provides in response to a query q is variable in size. In the reward-punishment case, the user provides one action: either a reward, or a punishment. If we were to use a preference interaction instead, the user would provide two actions by returning a total ordering over the two options presented to them.

Intervention Options. This quantifies the user's granularity in providing feedback. In both the reward-punishment and preference interactions we have used as examples, the expectation is that the user must respond to the entirety of the query q. Therefore, the space of their intervention choices is 0; it is a coarse response. However, some interactions, such as corrections [Bajcsy et al., 2018] allow the user to select subsets of q = {ξ0} to modify; in the lunar lander case, a user could select the entirety of ξ0 as good and make no modifications, or adjust some chunk of i < T actions. The user's intervention choice is 2T because they have the opportunity to intervene at each time step.

Response Choice Space. This is the number of possible responses that a user can provide, given a query q, and is related to AU. In the reward-punishment example, the user can give |AU | = |{-1, +1}| = 2 possible responses. More generally, a user has as many response options as they have potential rewards or labels to assign. On the other hand, if we were to use a preference interaction, then |AU| = 2; more generally, in ranking N options, users have N! orderings to choose from.

4 User Study

We designed a mixed-design user study to find empirical differences in cognitive load and usability between interaction types. Our within-subjects independent variable, interaction type, had four levels: Showing, Categorizing, Sorting, and Evaluating. Our between-subjects independent variable, task domain, had two levels: Sequential Decision Making (henceforth SDM), and Classification.

To enable comparisons between interaction types, we selected similarly complex examples from each cluster and minimized presentation differences. We made the assumption that salient differences in user attitudes manifest even between low-complexity interactions (e.g. differences would be present between reward-punishment and 2-way preference comparisons, not just rating scales and N-way rankings). We chose the lowest-complexity, non-trivial examples of each interaction type: demonstrations with a manageable |AL| = 4 for Showing, reward & punishment for Categorizing, preference comparisons for Sorting, and credit assignment (a subcategory of critiques) for Evaluating (Figure 1). We also standardized the interaction interface (e.g. the number of buttons, duration of tasks, available controls) as much as possible to minimize their impact on user attitudes.

The SDM task involved piloting a lunar lander to land upright between flag posts. Participants supplied or responded to a trajectory. We manually created trajectories to show to participants, and ensured an equal distribution of successful and failed trajectories. For Showing, participants used keyboard inputs (|AL| = 4) to provide example trajectories. For Categorizing, participants labeled a video of a potential lunar lander trajectory with a thumbs-up or thumbs-down. For Sorting, participants were shown two videos of potential trajectories for a lunar lander, and chose the better one. Finally for Evaluating, participants were given one video of a potential lunar lander trajectory, and used a double-ended slider to select the best portion of the trajectory.

The Classification task consisted of a series of images to be annotated. Users provided or responded to one-word captions. We used 20 images from Pascal VOC 2012 [Everingham et al., 2010] by randomly selecting one image from each of its classes. Captions to be evaluated were generated by a Keras InceptionV3 [Szegedy et al., 2016] model trained on ImageNet [Deng et al., 2009]. For Showing, participants were given an image and typed a caption of their own choosing into a textbox. For Categorizing, participants labeled an image-caption pair as thumbs-up or thumbs-down. For Sorting participants were shown two potential captions for a given image, and chose the one they felt was better. Finally, for Evaluating, participants were given one image-caption pair, and used a grid to select the parts of the image that best justified the proposed caption.

Hypotheses

We hypothesize that interaction types are not interchangeable with respect to their human factors:

• H1 Cognitive load differs between interaction types
• H2 Task completion times differ between interaction types
• H3 User confidence varies between interaction types
• H4 Subjective usability differs between interaction types
• H5 Preferred interaction types differ between tasks

Our study is designed to identify significant differences, not to find causal relationships, but we expect that as Response Choice Space and Response Size increase, cognitive load will increase, while usability and performance suffer.

4.1 Measures

We collected metrics on cognitive load (M1, M2), performance (M3, M4), and usability (M5-M9), as well as participants' responses, and any button toggles or video replays. Participants had the opportunity to provide additional feedback, and were asked to report their age and gender.

M1 - Secondary task performance. During each interaction, participants pressed a key every time a color-changing circle turned pink (Figure 2). The longer the participants' reaction time, the greater the cognitive load [DeLeeuw and Mayer, 2008].

• M2 - Paas subjective rating scale. After each interaction section, participants responded to the prompt "How much mental effort did this interaction type demand?" using a 9point Likert scale [Paas, 1992] .

M3 - Primary question response time. We recorded the time between when the participant was given the stimulus (e.g. began the lunar lander game, or was first presented with an image to label) and when they submitted their response.

M4 - Self-reported confidence per query. For each query, participants responded to the prompt "How confident are you in your answer to the primary question (not the colorchanging circle) above?" with a 4-point Likert scale. We did not include a neutral option.

• M5 - Frustration. After each interaction section, participants responded to the NASA TLX [Hart and Staveland, 1988] prompt "How insecure, discouraged, irritated, stressed, and annoyed were you?" on a 9-point Likert scale.

M6 - Complexity. After each interaction section, participants responded to the System Usability Scale (SUS) [Brooke, 1996] prompt: "I found this interaction type unnecessarily complex" with a 5-point Likert scale.

• M7 - Ease of Use. After each interaction section, participants responded to the SUS prompt: "I thought this interaction type was easy to use" with a 5-point Likert scale.
• M8 - Overall Confidence. After each interaction section, participants responded to the SUS prompt: "I felt very confident using this interaction type" with a 5-point Likert scale.
• M9 - ForcedRanking. At the study's conclusion, users answered "My nth choice interaction type would be . . . " for their first, second, third, and fourth choice interactions.

4.2 Procedure

Participants were fully counterbalanced between all orderings of interaction types within a task domain. Participants were given instructions describing the study. They then practiced responding to the secondary task. At the beginning of each interaction type's section, participants were presented with instructions describing the interaction, and an example of a good response. Participants then practiced the interaction, including the secondary task and confidence assessment. Each interaction type's section comprised five questions presented in a sequential, but randomized, order.

5 Results

We collected data from 150 Prolific workers over the age of 18 and with approval ratings ≥ 98%. Partial or duplicate task completions were discarded, leaving us with 144 participants, 72 per task domain. 61.1% of the participants self-identified as male, 37.5% as female, and 1.39% as non-binary. Their ages ranged from 18 to 70 (M = 26.71, SD = 9.45). This study and recruitment procedure was approved by our Institutional Review Board. We analyzed the effects of interaction types in each domain separately, and opted not to evaluate interaction effects between domains for two reasons. First, we use Likert-type scale data which is subject to interpersonal variance and cannot be reliably compared between separate populations. Second, our goal is not to identify differences between these specific domains, but to show that task domain can affect participants' preferences.

We analyzed all ordinal data using a Friedman Test followed by a post-hoc Wilcoxon signed-rank test (Bonferonni correction α = 0.0083). Numerical data was analyzed with a one-way repeated measures ANOVA and post-hoc pairwise Tukey analyses. We used α = 0.05 for our analyses. Because we used only portions of NASA-TLX and SUS to avoid participant fatigue, we treat each question as an individual item.

H1: Cognitive load differs between interaction types. A one-way repeated measures ANOVA revealed a statistically significant difference in secondary task reaction times (M1, Figure 3a) between interaction types (F(3, 213) = 6.57, p < 0.001 in SDM, and F(3, 213) = 20.04, p < 0.001 in Classification). In SDM, secondary reaction time was significantly longer in Showing as compared to Categorizing (p < 0.05). In Classification, secondary task reaction time during Evaluating was significantly less than in Showing, Sorting or Categorizing (p < 0.01). For completeness, we repeated this analysis after performing outlier rejection for samples more than three standard deviations from the mean: no differences were found in our results.

Differences were also found in participants' subjective assessments of the mental effort each interaction type required (M2, Figure 3b) in both domains (χ2 (3) = 1.3 × 10-13 , p < 0.001 in SDM, and χ2 (3) = 4.44 × 10-15 , p < 0.001 in Classification). In SDM, participants felt that Sorting was significantly harder than Categorizing (p < 0.006), and that Evaluating was significantly harder than Showing, Sorting, and Categorizing (p < 0.001 in all cases). In Classification, participants rated Evaluating as significantly harder than Showing, Sorting, and Categorizing (p < 0.001 in all cases). The data supports H1.

H2: Task completion times differ between interaction types. Statistically significant differences were found in response times (M3, Figure 4) between interaction types in both domains (F(3, 213) = 28.79, p < 0.001 in SDM, F(3, 213) = 166.29, p < 0.001 in Classification). In SDM, response times were significantly greater in Sorting as compared to Showing and Categorizing, and in Evaluating as compared to any other interaction type (p < 0.01 for both). In Classification, participants' response times to Evaluating were significantly greater than to any other interaction type (p < 0.01 in all cases). Figure 4 demonstrates these findings visually. When we repeated this analysis after performing outlier rejection for samples more than three standard deviations from the mean, our results largely stayed the same: we additionally found that Showing interactions in the Classification domain took significantly longer than Categorizing (α < 0.05). The data supports H2.

Figure 3: Cognitive load (H1) was measured both objectively (secondary task reaction time) and subjectively.

H3: User confidence varies between interaction types. Median per-trial confidence scores (M4, Figure 5) were significantly different between interaction types in both domains (χ2 (3) = 49.24, p < 0.001 in SDM, and χ2 (3) = 102.91, p < 0.001 in Classification). In SDM, participants were significantly more confident in their responses to Showing and Categorizing than Sorting or Evaluating (p < 0.01 in all cases). In Classification, participants were more confident in their responses to Showing than to any other interaction (p < 0.01 in all cases). They were also more confident in their responses to Categorizing than to Sorting and Evaluating (p < 0.01 in both cases). The data supports H3.

Figure 5: Per-trial confidence in response quality (H3).

H4: Subjective usability differs between interaction types. Significant differences were found in participants' ratings of frustration (χ2 (3) = 27.07, p < 0.001 in SDM, χ2 (3) = 50.94, p < 0.001 in Classification), perceptions of complexity (χ2 (3) = 41.68, p < 0.001 in SDM, and χ2 (3) = 69.30, p < 0.001 in Classification), ease of use (χ2 (3) = 33.14, p < 0.001 in SDM, and χ2 (3) = 54.19, p < 0.001 in Classification), and confidence with the interaction type (χ2 (3) = 28.66, p < 0.001 in SDM, and χ2 (3) = 55.63, p < 0.001 in Classification). These results, corresponding to M5 through M8, are shown in Figures 6 through 9.

Participants felt more frustrated by Evaluating than any other interaction in both task domains (p < 0.00145 in all cases). They perceived Evaluating as more unnecessarily complex than any other interaction in both task domains as well (p < 0.001 in all cases); in Classification, they also perceived Sorting as unnecessarily more complex than Categorizing (p < 0.00785). Correspondingly, participants found Evaluating to be less easy to use than any other interaction type in both task domains (p < 0.001 in all cases). In SDM, participants were more confident with Showing, Sorting, and Categorizing over Evaluating (p < 0.001 in all cases). In Classification, participants felt more confident using Showing than Sorting (p < 0.00230) or Evaluating (p < 0.001). They also felt more confident using either Sorting or Categorizing over Evaluating (p < 0.001 in both cases). The data supports H4.

Figure 7: Responses to subjective measures of complexity (H4). Darker colors denote higher perceived complexity.

H5: Preferred interaction types differ between tasks. We tallied participants' preferred interaction types (M9) using a Condorcet method. We found that in SDM, participants preferred Showing, Sorting, Categorizing, and then Evaluating. In Classification, they preferred Categorizing, Sorting, Showing, and then Evaluating. The data supports H5.

6 Discussion

Our results show that interaction types are differently cognitively loading and usable, and may variably impact performance as estimated via task completion times and selfassessed confidence. Participants rated Evaluating interactions as requiring the most cognitive effort, being the most frustrating, the most unnecessarily complex, least easy to use, and inspiring the least confidence. Objectively, they also took the longest time to complete Evaluating tasks. In SDM, Sorting took longer than Showing and Categorizing. In Classification, participants felt Sorting was more unnecessarily complex than Categorizing and were less confident using it than Showing. This suggests that Categorizing is preferable to Sorting, which is preferable to Evaluating. This corresponds to our expectation that as an interaction's Response Choice Space and Response Size increases, its usability decreases.

Figure 9: Responses to subjective measures of confidence (H4). Darker colors denote greater perceived confidence.

Unexpectedly, Showing, has the largest Response Choice Space and was among the easiest to use. This may be due to cognitive shortcuts: when guiding the lunar lander, users may not be processing all possible trajectories, nor are they thinking of every word they know in order to caption images. Thus, our big-O estimates may have been too coarse to capture the nuances of a user's perceived response space.

We also found a disagreement between participants' subjective assessment of mental effort and their objective secondary task performance, as in prior work [DeLeeuw and Mayer, 2008]. This could indicate that there are additional, unknown factors that affect perceived mental effort. We did also observe a relationship between participants' primary task reaction times and subjective assessments of cognitive load, indicating that they took longer on cognitively loading tasks. This may have given them more opportunities to respond to the secondary task, influencing their reaction times.

Pre-existing notions that interaction types such as Evaluating and Sorting might be more user-friendly than others (particularly Showing), because they require fewer inputs from a user, were not supported in the two domains we evaluated. Furthermore, differences existed in participants' preferred interactions between the task domains, despite our standardization of interaction types within and between them. Future work is required to understand how properties of a task domain influence interactions.

This work is one step towards developing a principled understanding of the algorithmic and human-factors components of learning interactions. In particular, it is a necessary step towards understanding the trade-off between the expected informativeness of a learning interaction, and a user's ability to provide high quality feedback. As data-gathering needs increase in scale and across domains, understanding this relationship will expand our ability to design learning interactions that not only accommodate the needs of learning agents, but also leverage the capabilities of human teachers.

Acknowledgements

We thank our anonymous reviewers, as well as Aditya Dhawale, Tesca Fitzgerald, Misha Khodak, Ada Taylor and the HARP Lab at CMU for their feedback and advice. We also thank Meghna Behari for helping with coding the user study during her internship. This work is supported in part by the Office of Naval Research (N00014-18-1-2503). The Classification image in Figure 1 is "Inside Whitehaven library" by librariesteam and is licensed with CC BY 2.0 [Creative Commons, 2004].