{"type":"ScholarlyArticle","title":"The Many Benefits of Annotator Rationales for Relevance Judgments","authors":[{"name":"Tyler McDonnell","slug":"tyler-mcdonnell","given":"Tyler","family":"McDonnell","affiliation":"Dept. of Computer Science , University of Texas at Austin , USA"},{"name":"Mucahid Kutlu","slug":"mucahid-kutlu","given":"Mucahid","family":"Kutlu","affiliation":"Dept. of Computer Science , University of Texas at Austin , USA; School of Information , University of Texas at Austin , USA"},{"name":"Tamer Elsayed","slug":"tamer-elsayed","given":"Tamer","family":"Elsayed","affiliation":""},{"name":"Matthew Lease","slug":"matthew-lease","given":"Matthew","family":"Lease","affiliation":"‡ Dept. of Computer Science and Engineering , Qatar University , Qatar"}],"abstract":"When collecting subjective human ratings of items, it can be difficult to measure and enforce data quality due to task subjectivity and lack of insight into how judges arrive at each rating decision. To address this, we propose requiring judges to provide a specific type of rationale underlying each rating decision. We evaluate this approach in the domain of Information Retrieval, where human judges rate the relevance of Webpages. Cost-benefit analysis over 10,000 judgments collected on Mechanical Turk suggests a win-win: experienced crowd workers provide rationales with no increase in task completion time while providing further benefits, including more reliable judgments and greater transparency.","doi":"10.24963/ijcai.2017/692","url":"https://ijcai.org/proceedings/2017/692","pdf":"https://ijcai.org/proceedings/2017/0692.pdf","markdown":"https://ijcai.org/proceedings/2017/692.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/692","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Best Sister Conferences","track":"Best Sister Conferences","number":692,"pages":{"first":4909,"last":4913},"keywords":["Artificial Intelligence: human computer interaction","Artificial Intelligence: other","Artificial Intelligence: machine learning"],"videos":[],"references":[{"position":1,"raw":"[Alonso, 2009] Omar Alonso. Guidelines for designing crowdsourcing-based relevance experiments, 2009. CiteSeerX DOI 10.1.1.149.6649.","title":"Guidelines for designing crowdsourcing-based relevance experiments","authors":"Omar Alonso","year":"2009","venue":"CiteSeerX","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Artstein and Poesio, 2008] Ron Artstein and Massimo Poesio. Inter-coder agreement for computational linguistics. Computa- tional Linguistics, 34(4):555-596, 2008.","title":"Ron Artstein and Massimo Poesio. Inter-coder agreement for computational linguistics","authors":"Poesio Artstein","year":"2008","venue":"Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bailey et al., 2008] Peter Bailey, Nick Craswell, Ian Soboroff, Paul Thomas, Arjen P de Vries, and Emine Yilmaz. Relevance assessment: are judges exchangeable and does it matter. In SI- GIR, pages 667-674. ACM, 2008.","title":"Relevance assessment: are judges exchangeable and does it matter","authors":"Peter Bailey, Nick Craswell, Ian Soboroff, Paul Thomas, Arjen P De Vries, Emine Yilmaz","year":"2008","venue":"SI-GIR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bernstein et al., 2010] Michael S Bernstein, Greg Little, Robert C Miller, Björn Hartmann, Mark S Ackerman, David R Karger, David Crowell, and Katrina Panovich. Soylent: a word processor with a crowd inside. In UIST, pages 313-322. ACM, 2010.","title":"Soylent: a word processor with a crowd inside","authors":"Greg Michael S Bernstein, Robert C Little, Björn Miller, Hartmann, David R Mark S Ackerman, David Karger, Katrina Crowell, Panovich","year":"2010","venue":"UIST","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Blanco et al., 2011] Roi Blanco, Harry Halpin, Daniel M Herzig, Peter Mika, Jeffrey Pound, Henry S Thompson, and Thanh Tran Duc. Repeatable and reliable search system evaluation using crowdsourcing. In SIGIR, pages 923-932. ACM, 2011.","title":"Repeatable and reliable search system evaluation using crowdsourcing","authors":"Roi Blanco, Harry Halpin, Daniel M Herzig, Peter Mika, Jeffrey Pound, Thanh Tran Henry S Thompson, Duc","year":"2011","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Carletta, 1996] Jean Carletta. Assessing agreement on classi- fication tasks: the kappa statistic. Computational linguistics, 22(2):249-254, 1996.","title":"Assessing agreement on classification tasks: the kappa statistic","authors":"Carletta ;, Jean Carletta","year":"1996","venue":"Computational linguistics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Clarke et al., 2010] Charles L Clarke, Nick Craswell, and Ian Soboroff. Overview of the TREC 2009 Web Track. In Proceed- ings of NIST TREC, 2010.","title":"Overview of the TREC 2009 Web Track","authors":"Nick Charles L Clarke, Ian Craswell, Soboroff","year":"2010","venue":"Proceedings of NIST TREC","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hosseini et al., 2012] Mehdi Hosseini, Ingemar J Cox, Nataša Milić-Frayling, Gabriella Kazai, and Vishwa Vinay. On aggre- gating labels from multiple crowd workers to infer relevance of documents. In ECIR, pages 182-194. Springer, 2012.","title":"On aggregating labels from multiple crowd workers to infer relevance of documents","authors":"Mehdi Hosseini, Ingemar J Cox, Nataša Milić-Frayling, Gabriella Kazai, Vishwa Vinay","year":"2012","venue":"ECIR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kittur et al., 2013] Aniket Kittur, Jeffrey V Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. The Future of Crowd Work. In CSCW, pages 1301-1318. ACM, 2013.","title":"The Future of Crowd Work","authors":"Aniket Kittur, Michael Jeffrey V Nickerson, Elizabeth Bernstein, Aaron Gerber, John Shaw, Matt Zimmerman, John Lease, Horton","year":"2013","venue":"CSCW","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[McDonnell et al., 2016] Tyler McDonnell, Matthew Lease, Muc- ahid Kutlu, and Tamer Elsayed. Why Is That Relevant? Collect- ing Annotator Rationales for Relevance Judgments. In Proc. of the 4th AAAI Conference on Human Computation and Crowd- sourcing (HCOMP), pages 139-148, 2016. Best Paper Award.","title":"Why Is That Relevant? Collecting Annotator Rationales for Relevance Judgments","authors":"Tyler Mcdonnell, Matthew Lease, Mucahid Kutlu, Tamer Elsayed","year":"2016","venue":"Proc. of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Sheshadri and Lease, 2013] Aashish Sheshadri and Matthew Lease. SQUARE: A Benchmark for Research on Computing Crowd Consensus. In Proceedings of the AAAI Conference on Human Computation (HCOMP), pages 156-164, 2013.","title":"SQUARE: A Benchmark for Research on Computing Crowd Consensus","authors":"Aashish Sheshadri, Matthew Lease","year":"2013","venue":"Proceedings of the AAAI Conference on Human Computation (HCOMP)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Voorhees et al., 2005] Ellen M Voorhees, Donna K Harman, et al. TREC: Experiment and evaluation in information retrieval. The MIT Press, 2005.","title":"TREC: Experiment and evaluation in information retrieval","authors":"Ellen M Voorhees, Donna K Harman","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Voorhees, 2001] Ellen M Voorhees. The philosophy of informa- tion retrieval evaluation. In Evaluation of cross-language infor- mation retrieval systems, pages 355-370. Springer, 2001.","title":"The philosophy of information retrieval evaluation","authors":"Ellen M Voorhees","year":"2001","venue":"Evaluation of cross-language information retrieval systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Zaidan et al., 2007] Omar F Zaidan, Jason Eisner, and Christine D Piatko. Using annotator rationales to improve machine learning for text categorization. In HLT-NAACL, pages 260-267, 2007.","title":"Using annotator rationales to improve machine learning for text categorization","authors":"Jason Omar F Zaidan, Christine D Eisner, Piatko","year":"2007","venue":"HLT-NAACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}