{"type":"ScholarlyArticle","title":"An Interactive Human-Machine Learning Interface for Collecting and Learning from Complex Annotations","authors":[{"name":"Jonathan Erskine","slug":"jonathan-erskine","given":"Jonathan","family":"Erskine","affiliation":"University of Bristol"},{"name":"Matt Clifford","slug":"matt-clifford","given":"Matt","family":"Clifford","affiliation":"University of Bristol"},{"name":"Alexander Hepburn","slug":"alexander-hepburn","given":"Alexander","family":"Hepburn","affiliation":"University of Bristol"},{"name":"Raul Santos Rodriguez","slug":"raul-santos-rodriguez","given":"Raul","family":"Santos Rodriguez","affiliation":"University of Bristol"}],"abstract":"Human-Computer Interaction has been shown to lead to improvements in machine learning systems by boosting model performance, accelerating learning and building user confidence. In this work, we aim to alleviate the expectation that human annotators adapt to the constraints imposed by traditional labels by allowing for extra flexibility in the form that supervision information is collected. For this, we propose a human-machine learning interface for binary classification tasks which enables human annotators to utilise counterfactual examples to complement standard binary labels as annotations for a dataset. Finally we discuss the challenges in future extensions of this work.","doi":"10.24963/ijcai.2024/999","url":"https://ijcai.org/proceedings/2024/999","pdf":"https://ijcai.org/proceedings/2024/0999.pdf","markdown":"https://ijcai.org/proceedings/2024/999.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/999","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Demo Track","track":"Demo Track","number":999,"pages":{"first":8644,"last":8647},"keywords":["Humans and AI: HAI: Human-AI collaboration","Humans and AI: HAI: Human-computer interaction","Humans and AI: HAI: Intelligent user interfaces"],"videos":[],"references":[{"position":1,"raw":"[Collins et al., 2022] Katherine M. Collins, Catherine Wong, Jiahai Feng, Megan Wei, and Joshua B. Tenenbaum. Struc- tured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks, 2022.","title":"Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks","authors":"Katherine M Collins, Catherine Wong, Jiahai Feng, Megan Wei, Joshua B Tenenbaum","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapo- lis, Minnesota, June 2019. Association for Computational Linguistics.","title":"BERT: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Goodfellow et al., 2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org.","title":"Deep Learning","authors":"Ian Goodfellow, Yoshua Bengio, Aaron Courville","year":"2016","venue":null,"doi":null,"url":"http://www.deeplearningbook.org","internal_url":null},{"position":4,"raw":"[Hastie et al., 2009] Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.","title":"The elements of statistical learning: data mining, inference, and prediction","authors":"Trevor Hastie, Robert Tibshirani, Jerome H Friedman, Jerome H Friedman","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Kaushik et al., 2020] Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. Learning the difference that makes a dif- ference with counterfactually-augmented data. In Interna- tional Conference on Learning Representations, 2020.","title":"Learning the difference that makes a difference with counterfactually-augmented data","authors":"Kaushik","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kurzendorfer et al., 2017] Tanja Kurzendorfer, Peter Fis- cher, Negar Mirshahzadeh, Thomas Pohl, Alexander Brost, Stefan Steidl, and Andreas Maier. Rapid interactive and intuitive segmentation of 3d medical images using ra- dial basis function interpolation. Journal of Imaging, 3(4), 2017.","title":"Rapid interactive and intuitive segmentation of 3d medical images using radial basis function interpolation","authors":"Tanja Kurzendorfer, Peter Fischer, Negar Mirshahzadeh, Thomas Pohl, Alexander Brost, Stefan Steidl, Andreas Maier","year":"2017","venue":"Journal of Imaging","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Mosqueira-Rey et al., 2021] Eduardo Mosqueira-Rey, David Alonso-Ríos, and Andrés Baamonde-Lozano. Inte- grating iterative machine teaching and active learning into the machine learning loop. Procedia Computer Science, 192:553-562, 2021. Knowledge-Based and Intelligent Information & Engineering Systems: Proceedings of the 25th International Conference KES2021.","title":"Integrating iterative machine teaching and active learning into the machine learning loop","authors":"Eduardo Mosqueira-Rey, David Alonso-Ríos, Andrés Baamonde-Lozano","year":"2021","venue":"Knowledge-Based and Intelligent Information & Engineering Systems: Proceedings of the 25th International Conference KES2021","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Prince, 2004] Michael Prince. Does active learning work? a review of the research. Journal of engineering education, 93(3):223-231, 2004.","title":"Does active learning work? a review of the research","authors":"Michael Prince","year":"2004","venue":"Journal of engineering education","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ren et al., 2022] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. A survey of deep active learning. ACM Comput. Surv., 54(9):1-40, December 2022.","title":"A survey of deep active learning","authors":"Ren","year":"2022","venue":"ACM Comput. Surv","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Wang et al., 2021] Zijie J. Wang, Dongjin Choi, Shenyu Xu, and Diyi Yang. Putting humans in the natural lan- guage processing loop: A survey, 2021.","title":"Putting humans in the natural language processing loop: A survey","authors":"Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Wei et al., 2022] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022.","title":"Emergent abilities of large language models","authors":"Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, William Fedus","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Zhu, 2015] Xiaojin Zhu. Machine teaching: An inverse problem to machine learning and an approach toward op- timal education. Proceedings of the AAAI Conference on Artificial Intelligence, 29(1), March 2015.","title":"Machine teaching: An inverse problem to machine learning and an approach toward optimal education","authors":"Xiaojin Zhu","year":"2015","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}