{"type":"ScholarlyArticle","title":"Imprecise Oracles Impose Limits to Predictability in Supervised Learning (Extended Abstract)","authors":[{"name":"Anjali Sifar","slug":"anjali-sifar","given":"Anjali","family":"Sifar","affiliation":""},{"name":"Nisheeth Srivastava","slug":"nisheeth-srivastava","given":"Nisheeth","family":"Srivastava","affiliation":""}],"abstract":"Supervised learning operates on the premise that labels unambiguously represent ground truth. This premise is reasonable in domains wherein a high degree of consensus is easily possible for any given data record, e.g. in agreeing on whether an image contains an elephant or not. However, there are several domains wherein people disagree with each other on the appropriate label to assign to a record, e.g. whether a tweet is toxic. We argue that data labeling must be understood as a process with some degree of domain-dependent noise and that any claims of predictive prowess must be sensitive to the degree of this noise. We present a method for quantifying labeling noise in a particular domain wherein people are seen to disagree with their own past selves on the appropriate label to assign to a record: choices under prospect uncertainty. Our results indicate that `state-of-the-art' choice models of decisions from description, by failing to consider the intrinsic variability of human choice behavior, find themselves in the odd position of predicting humans' choices better than the same humans' own previous choices for the same problem. We conclude with observations on how the predicament we empirically demonstrate in our work could be handled in the practice of supervised learning.","doi":"10.24963/ijcai.2021/661","url":"https://ijcai.org/proceedings/2021/661","pdf":"https://ijcai.org/proceedings/2021/0661.pdf","markdown":"https://ijcai.org/proceedings/2021/661.md","bibtex":"https://ijcai.org/proceedings/2021/bibtex/661","volume":{"slug":"2021","title":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2021","url":"https://ijcai.org/proceedings/2021/","year":2021,"location":"Montreal","dates":"19-27 August 2021","editors":"Prof. Zhi-Hua Zhou","isbn":"978-0-9992411-9-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2021"},"section":"Sister Conferences Best Papers","track":"Sister Conferences Best Papers","number":661,"pages":{"first":4834,"last":4838},"keywords":["Machine Learning: Classification","Uncertainty in AI: Uncertainty Representations","Machine Learning Applications: Applications of Supervised Learning","Humans and AI: Cognitive Systems"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}