{"type":"ScholarlyArticle","title":"Enhancing Crowdworkers' Vigilance","authors":[{"name":"Avshalom Elmalech","slug":"avshalom-elmalech","given":"Avshalom","family":"Elmalech","affiliation":"Harvard University , USA"},{"name":"David Sarne","slug":"david-sarne","given":"David","family":"Sarne","affiliation":"Bar-Ilan University , Israel"},{"name":"Esther David","slug":"esther-david","given":"Esther","family":"David","affiliation":"Ashkelon Academic College , Israel"},{"name":"Chen Hajaj","slug":"chen-hajaj","given":"Chen","family":"Hajaj","affiliation":"Vanderbilt University , USA"}],"abstract":"This paper presents methods for improving the attention span of workers in tasks that heavily rely on their attention to the occurrence of rare events.  The underlying idea in our approach is to dynamically augment the task with some dummy (artificial) events at different times throughout the task, rewarding the worker upon identifying and reporting them.  The proposed approach is an alternative to the traditional approach of exclusively relying on rewarding the worker for successfully identifying the event of interest itself.  We propose three methods for timing the dummy events throughout the task. Two of these methods are static and determine the timing of the dummy events at random or uniformly throughout the task. The third method is dynamic and uses the identification (or misidentification) of dummy events as a signal for the worker's attention to the task, adjusting the rate of dummy events generation accordingly.","doi":"10.24963/ijcai.2017/675","url":"https://ijcai.org/proceedings/2017/675","pdf":"https://ijcai.org/proceedings/2017/0675.pdf","markdown":"https://ijcai.org/proceedings/2017/675.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/675","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Best Sister Conferences","track":"Best Sister Conferences","number":675,"pages":{"first":4826,"last":4830},"keywords":["Artificial Intelligence: human computer interaction","Artificial Intelligence: artificial intelligence"],"videos":[],"references":[{"position":1,"raw":"[Aarabi, 2007] Parham Aarabi. The art of lecturing: a prac- tical guide to successful university lectures and business presentations. Cambridge University Press, 2007.","title":"The art of lecturing: a practical guide to successful university lectures and business presentations","authors":"Parham Aarabi","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Akkaya et al., 2010] Cem Akkaya, Alexander Conrad, Janyce Wiebe, and Rada Mihalcea. Amazon mechanical turk for subjectivity word sense disambiguation. In Pro- ceedings of the NAACL HLT 2010 Workshop on Creat- ing Speech and Language Data with Amazon's Mechan- ical Turk, pages 195-203. Association for Computational Linguistics, 2010.","title":"Amazon mechanical turk for subjectivity word sense disambiguation","authors":"Akkaya","year":"2010","venue":"Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Elmalech et al., 2015a] Avshalom Elmalech, David Sarne, and Barbara J Grosz. Problem restructuring for better deci- sion making in recurring decision situations. Autonomous Agents and Multi-Agent Systems, 29(1):1-39, 2015.","title":"Problem restructuring for better decision making in recurring decision situations","authors":"Elmalech","year":"2015","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Elmalech et al., 2015b] Avshalom Elmalech, David Sarne, Avi Rosenfeld, and Eden Shalom Erez. When suboptimal rules. In Proc. of AAAI, pages 1313-1319, 2015.","title":"When suboptimal rules","authors":"Elmalech","year":"2015","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Elmalech et al., 2016a] Avshalom Elmalech, David Sarne, and Noa Agmon. Agent development as a strategy shaper. Autonomous Agents and Multi-Agent Systems, 30(3):506- 525, 2016.","title":"Agent development as a strategy shaper","authors":"Elmalech","year":"2016","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Elmalech et al., 2016b] Avshalom Elmalech, David Sarne, Esther David, and Chen Hajaj. Extending workers' atten- tion span through dummy events. In Fourth AAAI Confer- ence on Human Computation and Crowdsourcing, 2016.","title":"Extending workers' attention span through dummy events","authors":"Elmalech","year":"2016","venue":"Fourth AAAI Conference on Human Computation and Crowdsourcing","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Feng et al., 2014] Zhenni Feng, Yanmin Zhu, Qian Zhang, Lionel M Ni, and Athanasios V Vasilakos. Trac: Truth- ful auction for location-aware collaborative sensing in mo- bile crowdsourcing. In Proceedings of INFOCOM, pages 1231-1239, 2014.","title":"Trac: Truthful auction for location-aware collaborative sensing in mobile crowdsourcing","authors":"Feng","year":"2014","venue":"Proceedings of INFOCOM","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gao et al., 2012] Xi Alice Gao, Yoram Bachrach, Peter Key, and Thore Graepel. Quality expectation-variance tradeoffs in crowdsourcing contests. In Proc. of AAAI, 2012.","title":"Quality expectation-variance tradeoffs in crowdsourcing contests","authors":"Alice Xi, Yoram Gao, Peter Bachrach, Thore Key, Graepel","year":"2012","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hajaj et al., 2015] Chen Hajaj, Noam Hazon, and David Sarne. Improving comparison shopping agents' compe- tence through selective price disclosure. Electronic Com- merce Research and Applications, 14(6):563-581, 2015.","title":"Improving comparison shopping agents' competence through selective price disclosure","authors":"Chen Hajaj, Noam Hazon, David Sarne","year":"2015","venue":"Electronic Commerce Research and Applications","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hajaj et al., 2016] C. Hajaj, N. Hazon, and D. Sarne. En- hancing comparison shopping agents through ordering and gradual information disclosure. to appear in JAAMAS, 2016.","title":"Enhancing comparison shopping agents through ordering and gradual information disclosure","authors":"C Hajaj, N Hazon, D Sarne","year":"2016","venue":"JAAMAS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kaufmann et al., 2011] Nicolas Kaufmann, Thimo Schulze, and Daniel Veit. More than fun and money. worker moti- vation in crowdsourcing-a study on mechanical turk. In AMCIS, volume 11, pages 1-11, 2011.","title":"More than fun and money. worker motivation in crowdsourcing-a study on mechanical turk","authors":"Nicolas Kaufmann, Thimo Schulze, Daniel Veit","year":"2011","venue":"AMCIS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Law et al., 2016] Edith Law, Ming Yin, Joslin Goh, Kevin Chen, Michael A Terry, and Krzysztof Z Gajos. Curiosity killed the cat, but makes crowdwork better. In Proceed- ings of the 2016 CHI Conference on Human Factors in Computing Systems, pages 4098-4110. ACM, 2016.","title":"Curiosity killed the cat, but makes crowdwork better","authors":"Edith Law, Ming Yin, Joslin Goh, Kevin Chen, Michael A Terry, Krzysztof Z Gajos","year":"2016","venue":"Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Laws et al., 2011] Florian Laws, Christian Scheible, and Hinrich Schütze. Active learning with amazon mechanical turk. In Proceedings of the conference on empirical meth- ods in natural language processing, pages 1546-1556. As- sociation for Computational Linguistics, 2011.","title":"Active learning with amazon mechanical turk","authors":"Laws","year":"2011","venue":"Proceedings of the conference on empirical methods in natural language processing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mason and Watts, 2010] Winter Mason and Duncan J Watts. Financial incentives and the performance of crowds. ACM SigKDD Explorations Newsletter, 11(2):100-108, 2010.","title":"Winter Mason and Duncan J Watts. Financial incentives and the performance of crowds","authors":"Watts Mason","year":"2010","venue":"ACM SigKDD Explorations Newsletter","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rahman, 2012] David Rahman. But who will monitor the monitor? The American Economic Review, pages 2767- 2797, 2012.","title":"But who will monitor the monitor? The American Economic Review","authors":"David Rahman","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Yin and Chen, 2015] Ming Yin and Yiling Chen. Bonus or not? learn to reward in crowdsourcing. In Proc. of IJCAI, pages 201-207, 2015.","title":"Bonus or not? learn to reward in crowdsourcing","authors":"Chen ; Yin, Ming Yin, Yiling Chen","year":"2015","venue":"Proc. of IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/35"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}