Bonus or Not? Learn to Reward in Crowdsourcing

1 Introduction

Recent advances in crowdsourcing present great potentials in harnessing the dispersed crowd intelligence for various purposes, such as data collection [Wah et al., 2011], scientific discovery [Khatib et al., 2011] and disaster relief [Zook et al., 2010]. A crucial factor that affects the success of a crowdsourcing attempt is the incentive design. Many domainspecific crowdsourcing systems are designed smartly to engage participants with their intrinsic motivations such as enjoyment in game playing and curiosity for new knowledge [Von Ahn, 2006; Savage, 2012]. However, on most general crowdsourcing platforms such as Amazon Mechanical Turk (MTurk), the primary type of incentive remains to be extrinsic motivation, that is, crowd workers complete tasks in exchange for monetary compensations.

Many studies have been conducted to understand the effects of financial incentives in crowdsourcing settings. Initial exploration suggested that financial incentives affect only the quantity of tasks that workers complete but not the work quality [Mason and Watts, 2009]. However, recent experiments showed that when a worker's reward is dependent

Yiling Chen

Harvard University Cambridge MA, USA yiling@seas.harvard.edu

on her performance in the task, financial incentives can influence the quality of work. For example, Harris [2011] found that adding performance-contingent bonus on top of the performance-independent base payment in each task leads to higher work quality and Ho et al. [2015] further identified that performance based payment can improve work quality when the task is "effort-responsive", that is, workers can produce higher quality work by exerting more efforts in the task. Moreover, it was also observed in previous studies that changing the magnitude of performance-contingent bonus in a task sequence leads to fluctuating work quality [Yin et al., 2013a; 2013b].

While all such evidence suggests that performancecontingent rewards can affect the quality of crowd work, it is not necessarily always beneficial to provide such rewards in a crowdsourcing working session (i.e. a sequence of tasks) as the potentially improved quality comes with an increase in cost. Thus, a key challenge for a crowdsourcing system designer (i.e. requester) to address is whether and when to offer such rewards in a working session to maximize the overall utility he derives from the session. Currently, a common practice among requesters is to follow a fixed or a random scheme to offer a performance-contingent bonus on none of the tasks, all of the tasks or a number of randomly selected tasks in a working session, and each worker is awarded in the same way. Can we improve requester utility by dynamically adjusting the placement of bonuses in a working session?

We provide an initial answer to this question by presenting an algorithmic approach to control bonuses in crowdsourcing working sessions based on a probabilistic model. In particular, our algorithmic approach helps a requester to dynamically decide for each task in a working session, whether or not he should offer an extra performance-contingent bonus to a worker given his observation on the worker's past performance. We make the following contributions:

• We propose to train an input-output hidden Markov model (IOHMM) to characterize the impact of performancecontingent bonuses on work quality in a working session.
• We augment the learned IOHMMwith a requester's utility function and turn the bonus placement problem into a problem of utility maximization under uncertainty. Three heuristic algorithms are provided to solve for a utilitymaximizing, dynamic policy of bonus placement.

• We design and conduct a randomized experiment on Amazon Mechanical Turk (MTurk) to evaluate our approach of dynamic bonus placement against three other baseline bonus schemes. Our approach improves the requester utility by as much as 27.22% beyond what is achieved by the best-performing baseline bonus scheme. To the best of our knowledge, this is the first time that the effectiveness of algorithmically-controlled bonus schemes is shown with real crowd workers.
• We further validate the robustness of our approach through simulations on two synthesized data sets that are generated according to two different worker behavior models respectively. Simulation results show that our approach consistently leads to improved requester utility for both models of worker behavior and different compositions of worker population.

1.1 RelatedWork

The lack of quality assurance hence the potentially low requester utility is a big concern in crowdsourcing. To tackle this problem, researchers have proposed different solutions from various aspects. For example, to boost the accuracy of labeling tasks based on redundant annotations, a large number of algorithms are designed to infer task difficulty, worker reliability and the underlying true labels simultaneously [Whitehill et al., 2009; Raykar et al., 2009]. Researchers also take real-world constraints such as the limited budget into consideration and investigate how to optimally allocate the budget among tasks and workers to elicit high quality labels [Chen et al., 2013]. In addition, machine learning and decisiontheoretic techniques have been adopted to guarantee the requester's utility by dynamically controlling the crowdsourcing process through decisions on worker recruitment [Kamar et al., 2012], task assignment [Dai et al., 2010] and workflow switches [Lin et al., 2012].

One common assumption of these previous studies is that worker performance is decided by the worker's inherent capability level, which is independent of her working environment, such as how much she gets paid in each of the completed tasks. The empirical observations that we have mentioned before, however, indicate that worker performance can be dependent on the provided incentives in tasks, and thus lead to an interesting question of how to reward workers in an optimal way such that the requester utility is maximized. To this end, Wang and Ipeirotis [2013] proposed a "payment with reimbursement" scheme based on the conjectures that the fluctuation of payments in task sequences is undesirable as workers may interpret a decrease of financial incentives as a punishment. Ho et al. [2014] used the classical principalagent model to characterize how a worker makes strategic decisions when provided with a performance-contingent payment and studied how to adaptively adjust such payment over time. Different from these studies, in this paper, we propose to first learn a model from the data to empirically characterize worker behavior in reaction to financial incentives, and then use this model to make decisions on the placement of bonuses. Huang et al. [2010] adopted a similar approach to optimize crowdsourcing task environment. However, they focused on design variables such as how many HITs (i.e. Human Intelligence Tasks on MTurk) to post and how many tasks to be bundled in one HIT, and followed a fixed rate of pay for each task in the HIT.

Finally, our work is different from several pricing mechanisms proposed to elicit more (or faster) work from rational workers given a fixed budget [Singer and Mittal, 2013; Singla and Krause, 2013; Gao and Parameswaran, 2014] - our goal is to elicit high-quality work and we have no assumption on the rationality of workers.

2 An Algorithmic Approach

Our algorithmic approach for bonus control in a crowdsourcing working session can be decomposed into two steps: First, we collect a training data set and learn an input-output hidden Markov model to characterize how the presence of performance-contingent bonuses affect the work quality in a working session; second, we incorporate utilities with the learned IOHMM and dynamically determine whether to place a bonus on each task for a new incoming worker by solving a utility maximization problem.

2.1 Characterizing the Impact of Bonuses with an IOHMM

Suppose we have collected a data set on worker behavior in a working session, which is a sequence of tasks that a worker completes without interruption. For each task in the session, we record the provided bonus level and the worker performance in it. To enable the algorithmic bonus control for future workers, we first aim at leveraging this training data set to quantitatively model the impact of bonuses on work quality in a working session.

A natural model choice is the input-output hidden Markov model, which is a generative probabilistic model for sequential data. In our context, consider that time step t in IOHMM corresponds to the t-th task in a session. For each t, there is an input at and an output xt, which represents the level of bonus and the observed work quality in task t, respectively. Furthermore, there is a hidden state zt, which comes from one of the K states and can capture the unobserved properties of a crowd worker in the t-th task. Worker behavior is controlled by the transition and emission probabilities: Ptr(zt|zt-1, at) gives a worker's chance of transiting to state zt in the t-th task conditioned on that her state in the (t -1)-th task is zt-1 and the input for the t-th task is at, and Pe(xt|zt, at) represents how likely a worker will produce output xt in the t-th task given her current state zt and input at.

For simplicity, we illustrate our algorithmic approach in the rest of the paper assuming that the work quality for a task is binary (i.e. incorrect or correct, low-quality or high-quality, etc.) and the requester can choose one of two bonus levels in a task - either place a bonus of fixed magnitude or not provide any bonus at all. Suppose the number of tasks bundled in a working session is T. The IOHMM model details for this simplified problem are as follows:

• Inputs: at ∈ {0, 1}, t = 1, 2, · · · , T, with 0 representing bonus is not placed on the task.
• Outputs: xt ∈ {0, 1}, t = 1, 2, · · · , T, with 0 representing an incorrect (or low-quality) answer for the task.

• Hidden States: zt ∈ {1, 2, · · · ,K}, t = 1, 2, · · · , T.
• Transition probability: Ptr(zt|zt-1, at); e.g., Ptr(k′ |k, 0) gives the probability of a worker transiting from hidden state k in task t - 1 to hidden state k′ in task t given that there is no extra bonus in task t.
• Emission probability: Pe(xt|zt, at); e.g., Pe(1|k, 0) gives the probability of a worker submitting a correct (or high-quality) answer to task t when the current state is k and no extra bonus is offered on task t.

With the training data set that is collected from a group of workers, we can learn the IOHMM for this worker population through an expectation-maximization algorithm, which is similar to the Baum-Welch algorithm for learning hidden Markov models [Bengio and Frasconi, 1996].

2.2 DecisionMaking with the learned IOHMM

Given the learned IOHMM M to describe the impact of bonuses on work quality in a working session, the next question is to decide for a new incoming worker whether or not the requester places a bonus on each task in her working session.

To quantify the requester's tradeoff between work quality and cost, we assume that the requester obtains a utility of wh (or wl) when he gets a high-quality (or low-quality) answer, while the economic cost for paying a performance-contingent bonus is c. We further assume that the requester has a quasilinear utility function U = whNHQ + wlNLQ - cNB, where NHQ (or NLQ) and NB represents the number of highquality (or low-quality) answers elicited and the number of times a performance-contingent bonus is incurred, respectively. As a requester can continuously make observations on worker performance over time, we are interested in dynamically controlling the placement of bonus in a working session in an online fashion. That is, we keep making decisions on whether the requester should provide a bonus to a worker in her next task given the history of inputs and outputs in all tasks that the worker has completed so far in the session.

In particular, for a worker who has completed tc tasks, we estimate the distribution of her current state as b(tc) (i.e. the "state belief") based on M. We define EUmax(b, a, l) as the maximum expected utility a requester can obtain in the next l tasks given that the current state belief is b = (b(1), · · · , b(K)), the input level for the next task is a, and input levels for later tasks follow the optimal policy. Thus, the optimal input level for the next task is atc+1 = argmaxa∈{0,1} EUmax(b(tc), a, T - tc) - when atc+1 = 1, we offer a bonus on the next task; otherwise, we don't.

EUmax(b, a, l) can be calculated recursively. Specifically, we denote R(b, a) = ∑Ki=1 b(i)(∑Kj=1 Ptr(j|i, a) · (Pe(0|j, a)wl + Pe(1|j, a)(wh - I(a = 1)c))) as the requester's expected utility in the next task when the current state belief is b and the next input is a. When there is only one task left in the session, that is, l = 1, EUmax(b, a, l) = R(b, a); otherwise, we have:

where V (b, l) = maxa∈{0,1} EUmax(b, a, l) and b′a,x is the updated state belief if the input for the next task is a and the observed output is x, which can be computed as follows:

And finally, in preparation for the decision making in future tasks, we update the belief state after implementing the input level atc+1 and observing the output level xtc+1, that is, b(tc + 1) = b′atc+1,xtc+1 .

Heuristic Solutions

The decision making problem above is essentially equivalent to solve a finite-horizon partially observable Markov decision process (POMDP), with "action" corresponding to "input" inMand the reward of taking action a in state k being R(k, a) = ∑Ki=1 Ptr(i|k, a)(Pe(0|i, a)wl + Pe(1|i, a)(wh - I(a = 1)c)). In practice, finding exact solutions for POMDPs are often computationally intractable [Papadimitriou and Tsitsiklis, 1987]. Therefore, we list a few heuristic algorithms to solve the problem approximately:

Algorithm 1 (n-step look-ahead) When making decisions for whether to place an extra bonus on the next task, we look ahead for at most n tasks. That is, atc+1 = argmaxa∈{0,1} EUmax(b(tc), a, n′ ), where n′ = min(n, T -tc). A similar strategy is used in [Dai et al., 2010] for optimal control of crowdsourcing workflows.

Note that if a worker's hidden state ztc after completing tc tasks as well as her states in the future tasks can be accurately identified, the finite horizon POMDP degenerates into a finite horizon MDP, for which we can calculate the Q-functions and optimal policies efficiently. We thus consider two algorithms that leverage this advantage:

Algorithm 2 (MLS-MDP) We infer the most likely sequence (MLS) of hidden states up to the current task (i.e. ˆ Ztc 1 = ( ˆ z1, · · · , ˆ ztc )) using the Viterbi algorithm [Viterbi, 1967] and estimate ztc as ˆ ztc . The input level for the next task atc+1 is then set to be πT-tc (ztc ), that is, the optimal MDP policy on state ztc when the length of horizon is T -tc.

Algorithm 3 (Q-MDP) We first calculate QT-tc (k, a) for the underlying MDP, which is the Q-function value for taking action a on state k with T - tc steps to go. Then, atc+1 = argmaxa∈{0,1} ∑Kk=1 pkQT-tc (k, a), with pk being the k-th element of b(tc) [Littman et al., 1995].

3 MTurk Experiment

To examine whether our algorithmic approach of placing bonuses can effectively improve requester utility in a real crowdsourcing working session, we design and conduct an online experiment on MTurk.

3.1 Task

We use a word puzzle game as our task in the experiment: In each task, a worker will see a 12×12 board filled with capital letters and a "target" word on the screen. This target word can be placed on the board horizontally, vertically or diagonally and for multiple times. The worker is asked to identify as many appearance of the target word on the board as possible and specify the location of each identified appearance. Each worker completes 9 tasks in a working session (i.e. one HIT). A similar task is previously used by Mason and Watts [2009] to study the effects of financial incentives on the performance of crowds.

A worker earns a performance-independent reward of 5 cents in each task, that is, the base payment for the HIT is 45 cents. In addition, we also inform the worker that some tasks in the session are "bonus tasks" (specified with a bonus icon), in which she may earn an extra bonus of 5 cents if she submits a high-quality answer to it by pointing out more than 80% of all appearances of the target word. By design, each board in our experiment contains the target word 11 times (workers are not aware of this fact, however), which means that a worker can only earn the extra reward in a bonus task if she identifies the target word for at least 9 times.

3.2 Procedure

Corresponding to the two steps in our algorithmic approach, we divide our experiment into two phases.

In the first phase, we collect a training data set by recruiting 50 MTurk workers to participate in our experiment. For each of the 9 tasks that a worker completes in the HIT, we randomly set it as a bonus task with a 20% chance; whether that task is a bonus task and whether the worker submits a high-quality answer to it (i.e. finds out the target word at least 9 times) is recorded. We then learn an IOHMM to understand the impact of bonuses on worker performance in the word puzzle game sequences using the collected data set. Specifically, we run the expectation-maximization algorithm with 100000 random restarts, and each run is terminated after convergence or 500 iterations, whichever is reached earlier. In searching for a parsimonious model, we experiment on a range of values for the number of hidden states (K = 1 ∼ 7) to train different IOHMMs, and the IOHMM with the maximized Bayesian information criterion (BIC) score [Schwarz and others, 1978] is selected to be used in the second phase. In our experiment, K = 2 for the selected IOHMM.

The second phase of our experiment is the testing phase, in which we have 6 experimental treatments and each treatment corresponds to one bonus scheme. In particular, we include 3 dynamic bonus schemes that are designed according to our algorithmic approach using different heuristics (i.e. 2-step look-ahead 1 , MLS-MDP and Q-MDP). When these schemes are used in treatments, we keep track of a worker's performance in the session and use the learned IOHMM to strategically make a decision on whether to offer an extra bonus to the worker on the next task. As a comparison, we also consider 3 fixed or random baseline bonus schemes: not placing bonus in any task (No Bonus), always placing bonuses in all tasks (All Bonus) and randomly choosing 50% of the tasks to place bonuses (50% Bonus). The utility parameters we use in the experiment are wh = 0.15, wl = 0 and c = 0.052 .

1 We set n = 2 to balance between performance and efficiency based on simulations for workers who indeed behave according to the learned IOHMM.

To make sure the IOHMM learned from the training phase is useful for the testing phase, we recruit workers from the same pool by running our second phase experiment exactly 2 weeks after the first phase experiment around the same time. Each worker is randomly assigned to one treatment and 50 workers are recruited for each treatment. All workers in a treatment are paid according to the bonus scheme of that treatment. We again collect data on the presence of bonus and work quality for each task and each worker.

Our experiment is limited to U.S. workers and each worker is allowed to take the HIT only once.

3.3 Results

Figure 1 compares the overall utility a requester derives from all 50 workers in the working session across the 6 treatments of our second phase experiment. As we can see in the figure, among the 3 baseline bonus schemes, the best scheme is to pay no bonus at all. Yet, following our algorithmic approach, the 3 dynamic bonus schemes (i.e. 2-step lookahead, MLS-MDP and Q-MDP) lead to an increase of 3.11%, 27.22% and 12.89% in the requester's utility, respectively, compared to the No Bonus scheme. To see whether the utility improvement is material, we decompose the overall requester utility in each treatment into the number of high-quality answers the requester elicits (hence the economic benefits) and the cost the requester pays to encourage better performance (see the table in Figure 1). Results suggest that the requester can elicit more high-quality work with lower cost by applying our dynamic bonus schemes. For example, compared to the 50% Bonus scheme, a requester using the 2-step lookahead scheme obtains a similar number of high-quality answers with a 77.9% saving in money; while a requester following the MLS-MDP (or Q-MDP) scheme elicits 19.3% (or 11.2%) more high-quality answers with roughly a quarter (or a half) of the cost.

The statistical significance of the improvement in utility brought by our algorithmic approach is further examined through Wilcoxon rank-sum tests. Specifically, we compute the utility a requester obtains from each worker in every treatment. Thus, in total, we have 6 samples of requester utility with 50 data points in each sample. We conduct pairwise comparisons to test whether a pair of samples have the same mean value, and the p-values of the tests are reported in Table 1. Almost all pairwise comparisons are statistically significant at the p = 0.05 level, which suggests that our approach helps the requester to improve his utility.

2 c is set to be 0.05 as the bonus magnitude used in the experiment is $0.05. We set wh = 0.15 and wl = 0 to ensure that a dynamic bonus scheme doesn't become a No Bonus or All Bonus scheme.

Finally, to get a qualitative intuition of how our dynamic bonus schemes work, we pick a few exemplary workers from the treatment in which the MLS-MDP bonus scheme is used and take a close look at how they are awarded in the working session. Figure 2 displays for each of the 4 selected workers, whether a bonus is provided and whether her submitted answer is of high quality for each task in the working session. The comparison between worker A and worker B first suggests that our algorithmic approach can effectively differentiate "diligent" workers from "lazy" workers and reward them differently: For a diligent worker (worker A) who always submits high-quality answers, there is no need for the requester to place extra bonuses in the working session; however, for a lazy worker who can be responsive to financial incentives (worker B), the requester keeps offering bonuses in hope of increasing the work quality through providing additional motivation to the worker 3 . In fact, the MLS-MDP bonus scheme strategically focuses on incentivizing lazy workers: On average, the requester offers a bonus on 6 tasks to a worker who performs well in at most half of the tasks in the working session, while for a worker who performs well in more than half of the tasks, the requester offers a bonus on only 0.49 tasks. Furthermore, our algorithmic approach also seems to offer bonuses at the right timing. On the one hand, for a worker who starts a session with unsatisfying performance (worker C), the requester keeps placing bonus on each task to incentivize better performance until the worker stabilizes in submitting high-quality answers; on the other hand, for a worker who slacks off from her initial good performance (worker D), the requester provides extra incentives in time to bring back hard working from the worker.

4 Simulation

The performance of the dynamic bonus schemes in our MTurk experiment suggests the promise of algorithmically controlling the provision of financial incentives in crowdsourcing. However, one may wonder that following our algorithmic approach, whether the high requester utility can always be obtained in various worker populations where workers potentially behave in different ways. To understand the robustness of our approach, we further run simulations on two synthesized data sets and each data set is generated according to a predefined worker behavior model.

3 Offering bonus per se is not costly; the cost will only be incurred when the work quality in a bonus task meets the predefined standard.

Specifically, given a particular worker behavior model, in the training phase, we generate a data set of 3000 workers, while each worker completes a session of 50 tasks (20% of them are randomly selected as bonus tasks) and decides her performance in each task probabilistically according to the given model. In the testing phase, we consider the same 6 bonus schemes as those in our MTurk experiment. Six groups of testing data are thus generated: Each group is assigned to a unique bonus scheme and is composed of 100 workers; each worker completes a session of 10 tasks and is paid according to the bonus scheme of her group, while her performance in each task is controlled by the same behavior model as that used for generating the training data set. The requester's utility under a specific bonus scheme is calculated as the sum of utilities that the requester derives from all 100 workers of the corresponding group, with wl and wh set to be 0 and 1, respectively. We repeat the simulation 30 times and report the mean value of the requester's utility for each scheme.

4.1 Model 1: Workers with Two Capability Levels

In our first worker behavior model, we assume that when a performance-contingent bonus is placed (or not placed) in a task, the probability for worker i to submit a high-quality answer in a task is acchi (or accli ). This model is consistent with previous empirical observations that the existence of performance-based bonuses can affect the worker performance [Harris, 2011; Ho et al., 2015].

Figure 3: The requester's utility when workers have two capability levels in reaction to the placement of bonus. Error bars are omitted as they are too small.

We construct two different worker populations based on this worker behavior model. To test the performance of different bonus schemes when a requester's willingness to reward workers differs, we also vary the economic costs (i.e. magnitude) of the performance-contingent bonus c from 0.1 to 0.5 in the simulation. Figure 3(a) demonstrates the simulation results for a population that is composed of 2 types of workers: For the first type, accli = 0.5 and acchi = 0.9; while for the second type, accli = 0.8 and acchi = 0.9. Each worker in the population is drawn uniformly randomly from the two types. Similarly, Figure 3(b) corresponds to another population where each worker draws her accuracy from Beta distributions: accli ∼ Beta(2, 2) and acchi ∼ Beta(6, 2). As the figures suggest, for both populations, when the magnitude of bonus is small (or large) enough such that adding a bonus is always beneficial (or too costly), the dynamic bonus schemes lead to similar requester utility as the All Bonus (or No Bonus) scheme does. However, when the bonus magnitude is moderate, the dynamic bonus schemes robustly result in higher requester utility compared to any single baseline bonus scheme.

4.2 Model 2: Workers Influenced by Reference Payment Levels

Previous literature suggests another possible worker behavior in reaction to financial incentives in a working session. That is, a worker maintains and updates a reference point of "appropriate" payment level when she completes tasks in a working session and decides her performance in each task by comparing the provided payment with the reference of that time [Yin et al., 2013a; 2013b; Popescu and Wu, 2007; Akerlof and Yellen, 1990]. To see how our algorithmic approach performs when workers indeed behave in this way, we define the second worker behavior model. In particular, we assume that each worker behaves according to an IOHMM, with the hidden state zt corresponding to the reference payment level rzt in the worker's mind in the t-th task 4 . Each worker i is further characterized by her skill level αi and her responsiveness to financial incentives βi, and the emission probability in task t is parameterized as Pe(1|zt, at) = 1 1+e-αi-βi(at-rzt ) . Obviously, the larger αi or βi is, the worker is more skilled or more responsive to rewards hence more likely to produce high-quality answers.

For manageability, we assume in the simulation that each worker has K = 3 hidden states (i.e. 3 discrete reference payment levels), that is, r = (r1, r2, r3) = (0.2, 0.6, 1.2). Each worker updates her reference payment level in the working session according the transition probability matrices:

4 The reference payment level is defined relative to the magnitude of the bonus, e.g. rk = 0.5 means that in state k, the worker considers a half of the current bonus as an appropriate payment.

where the (i, j)-th element of matrix T a represents the probability for a worker to update her reference payment level from ri to rj given the current input level a, i.e. Ptr(j|i, a).

Furthermore, we assume that there are 3 possible skill levels for a worker, i.e. αi ∈ {0, 1, 3}, and each worker i draws her skill level according to the categorical distribution pα = (p1α, p2α, p3α), where p1α + p2α + p3α = 1 and p1α (or p2α, p3α) represents the probability that αi = 0 (or 1, 3). Similarly, we also consider 3 levels of responsiveness to financial incentives, that is, βi ∈ {0, 1, 3}, and each worker draws her βi according to another categorical distribution pβ = (p1β , p2β , p3β ). Varying pα and pβ thus provides us the flexibility to construct a number of populations in which various types of workers are mixed in different proportions. For each population and 4 selected bonus magnitude (i.e. c = 0.05, 0.1, 0.15, 0.2), Figure 4 displays the utility difference (in percentage) a requester obtains by following a dynamic bonus scheme (averaged over the 3 dynamic schemes) over following the best-performing baseline scheme to control bonuses in the session: On the one hand, the best-performing baseline scheme differs across various populations and magnitude of bonus, suggesting that following a single baseline bonus scheme can't guarantee a high requester utility in all conditions; on the other hand, following our dynamic bonus schemes, the requester can consistently obtain similar or higher utility than what he could have obtained by following the best-performing baseline scheme, which again implies that the improved performance of our approach is robust.

5 Conclusions and FutureWork

In this paper, we take an algorithmic approach to dynamically control whether and when to place a bonus in a crowdsourcing working session. Our MTurk experiment and simulation results suggest that our approach can robustly lead to significant improvement in requester utility than fixed and random bonus schemes do. There are many interesting future directions for this work. For example, our current approach leads to a policy that provides more bonus opportunities for workers with lower accuracy. This may drive high accuracy workers away in the long term. It will be interesting to understand the long-term impact of our algorithmic approach.

Acknowledgements

We thank the support of the National Science Foundation under grant CCF-1301976 and the Xerox Foundation on this work. Any opinions, findings, conclusions, or recommendations expressed here are those of the authors alone.