{"type":"ScholarlyArticle","title":"Are They the Same Picture? Adapting Concept Bottleneck Models for Human-AI Collaboration in Image Retrieval","authors":[{"name":"Vaibhav Balloli","slug":"vaibhav-balloli","given":"Vaibhav","family":"Balloli","affiliation":"University of Michigan"},{"name":"Sara Beery","slug":"sara-beery","given":"Sara","family":"Beery","affiliation":"Massachusetts Institute of Technology"},{"name":"Elizabeth Bondi-Kelly","slug":"elizabeth-bondi-kelly","given":"Elizabeth","family":"Bondi-Kelly","affiliation":"University of Michigan"}],"abstract":"Image retrieval plays a pivotal role in applications from wildlife conservation to healthcare, for finding individual animals or relevant images to aid diagnosis. Although deep learning techniques for image retrieval have advanced significantly, their imperfect real-world performance often necessitates including human expertise. Human-in-the-loop approaches typically rely on humans completing the task independently and then combining their opinions with an AI model in various ways, as these models offer very little interpretability or correctability. To allow humans to intervene in the AI model instead, thereby saving human time and effort, we adapt the Concept Bottleneck Model (CBM) and propose CHAIR. CHAIR (a) enables humans to correct intermediate concepts, which helps \\textit{improve}  embeddings generated, and (b) allows for flexible levels of intervention that accommodate varying levels of human expertise for better retrieval. To show the efficacy of \\methodname, we demonstrate that our method performs better than similar models on image retrieval metrics without any external intervention. Furthermore, we also showcase how human intervention helps further improve retrieval performance, thereby achieving human-AI complementarity.","doi":"10.24963/ijcai.2024/866","url":"https://ijcai.org/proceedings/2024/866","pdf":"https://ijcai.org/proceedings/2024/0866.pdf","markdown":"https://ijcai.org/proceedings/2024/866.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/866","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Human-Centred AI","track":"Human-Centred AI","number":866,"pages":{"first":7824,"last":7832},"keywords":["Computer Vision: CV: Image and video retrieval","Humans and AI: HAI: Human-AI collaboration"],"videos":[],"references":[{"position":1,"raw":"[Beede et al., 2020] Emma Beede, Elizabeth Baylor, Fred Hersch, Anna Iurchenko, Lauren Wilcox, Paisan Ruamvi- boonsuk, and Laura M Vardoulakis. A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy. In Proceedings of the 2020 CHI conference on human factors in computing systems, pages 1-12, 2020.","title":"A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy","authors":"Emma Beede, Elizabeth Baylor, Fred Hersch, Anna Iurchenko, Lauren Wilcox, Paisan Ruamviboonsuk, Laura M Vardoulakis","year":"2020","venue":"Proceedings of the 2020 CHI conference on human factors in computing systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Beery et al., 2019] Sara Beery, Dan Morris, and Siyu Yang. Efficient pipeline for camera trap image review. arXiv preprint arXiv:1907.06772, 2019.","title":"Efficient pipeline for camera trap image review","authors":"Sara Beery, Dan Morris, Siyu Yang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bondi et al., 2022] Elizabeth Bondi, Raphael Koster, Han- nah Sheahan, Martin Chadwick, Yoram Bachrach, Tay- lan Cemgil, Ulrich Paquet, and Krishnamurthy Dvijotham. Role of human-ai interaction in selective prediction. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 36, pages 5286-5294, 2022.","title":"Role of human-ai interaction in selective prediction","authors":"Elizabeth Bondi, Raphael Koster, Hannah Sheahan, Martin Chadwick, Yoram Bachrach, Taylan Cemgil, Ulrich Paquet, Krishnamurthy Dvijotham","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[De-Arteaga et al., 2020] Maria De-Arteaga, Riccardo Fogliato, and Alexandra Chouldechova. A case for humans-in-the-loop: Decisions in the presence of erro- neous algorithmic scores. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1-12, 2020.","title":"A case for humans-in-the-loop: Decisions in the presence of erroneous algorithmic scores","authors":"Maria De-Arteaga, Riccardo Fogliato, Alexandra Chouldechova","year":"2020","venue":"Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Espinosa Zarlenga et al., 2022] Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, Pietro Lio, and Mateja Jamnik. Concept embed- ding models: Beyond the accuracy-explainability trade- off. Advances in Neural Information Processing Systems, 35, 2022.","title":"Concept embedding models: Beyond the accuracy-explainability tradeoff","authors":"Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, Pietro Lio, Mateja Jamnik","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Havasi et al., 2022] Marton Havasi, Sonali Parbhoo, and Fi- nale Doshi-Velez. Addressing leakage in concept bottle- neck models. Advances in Neural Information Processing Systems, 35:23386-23397, 2022.","title":"Addressing leakage in concept bottleneck models","authors":"Marton Havasi, Sonali Parbhoo, Finale Doshi-Velez","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kim et al., 2023] Eunji Kim, Dahuin Jung, Sangha Park, Si- won Kim, and Sungroh Yoon. Probabilistic concept bot- tleneck models. arXiv preprint arXiv:2306.01574, 2023.","title":"Probabilistic concept bottleneck models","authors":"Eunji Kim, Dahuin Jung, Sangha Park, Siwon Kim, Sungroh Yoon","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Koh et al., 2020] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In Interna- tional conference on machine learning, pages 5338-5348.","title":"Concept bottleneck models","authors":"Pang Wei Koh, Thao Nguyen, Siang Yew, Stephen Tang, Emma Mussmann, Been Pierson, Percy Kim, Liang","year":null,"venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"PMLR, 2020.","title":null,"authors":"Pmlr","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kulits et al., 2021] Peter Kulits, Jake Wall, Anka Bedetti, Michelle Henley, and Sara Beery. Elephantbook: A semi-automated human-in-the-loop system for elephant re-identification. In ACM SIGCAS Conference on Com- puting and Sustainable Societies, pages 88-98, 2021.","title":"Elephantbook: A semi-automated human-in-the-loop system for elephant re-identification","authors":"Peter Kulits, Jake Wall, Anka Bedetti, Michelle Henley, Sara Beery","year":"2021","venue":"ACM SIGCAS Conference on Computing and Sustainable Societies","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2021] Wanhua Li, Xiaoke Huang, Jiwen Lu, Jian- jiang Feng, and Jie Zhou. Learning probabilistic ordinal embeddings for uncertainty-aware regression. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13896-13905, 2021.","title":"Learning probabilistic ordinal embeddings for uncertainty-aware regression","authors":"Wanhua Li, Xiaoke Huang, Jiwen Lu, Jianjiang Feng, Jie Zhou","year":"2021","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.","title":"Deep learning face attributes in the wild","authors":"Ziwei Liu, Ping Luo, Xiaogang Wang, Xiaoou Tang","year":"2015","venue":"Proceedings of International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2020] Yishu Liu, Liwang Ding, Conghui Chen, and Yingbin Liu. Similarity-based unsupervised deep transfer learning for remote sensing image retrieval. IEEE Transactions on Geoscience and Remote Sensing, 58(11):7872-7889, 2020.","title":"Similarity-based unsupervised deep transfer learning for remote sensing image retrieval","authors":"Yishu Liu, Liwang Ding, Conghui Chen, Yingbin Liu","year":"2020","venue":"IEEE Transactions on Geoscience and Remote Sensing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mahinpei et al., 2021] Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. arXiv preprint arXiv:2106.13314, 2021.","title":"Promises and pitfalls of black-box concept learning models","authors":"Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Velez, Weiwei Pan","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Marcinkevičs et al., 2024] Ričards Marcinkevičs, Sonia La- guna, Moritz Vandenhirtz, and Julia E Vogt. Beyond con- cept bottleneck models: How to make black boxes inter- venable? arXiv preprint arXiv:2401.13544, 2024.","title":"Ričards Marcinkevičs, Sonia Laguna","authors":"Marcinkevičs","year":"2024","venue":"Beyond concept bottleneck models: How to make black boxes intervenable","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Marconato et al., 2022] Emanuele Marconato, Andrea Passerini, and Stefano Teso. Glancenets: Interpretable, leak-proof concept-based models. Advances in Neural Information Processing Systems, 35:21212-21227, 2022.","title":"Glancenets: Interpretable, leak-proof concept-based models","authors":"Emanuele Marconato, Andrea Passerini, Stefano Teso","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mendes et al., 2022] Ethan Mendes, Yang Chen, Wei Xu, and Alan Ritter. Human-in-the-loop evaluation for early misinformation detection: A case study of covid-19 treat- ments. arXiv preprint arXiv:2212.09683, 2022.","title":"Human-in-the-loop evaluation for early misinformation detection: A case study of covid-19 treatments","authors":"Ethan Mendes, Yang Chen, Wei Xu, Alan Ritter","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Miao et al., 2021] Zhongqi Miao, Ziwei Liu, Kaitlyn M Gaynor, Meredith S Palmer, Stella X Yu, and Wayne M Getz. Iterative human and automated identification of wildlife images. Nature Machine Intelligence, 3(10):885- 895, 2021.","title":"Iterative human and automated identification of wildlife images","authors":"Miao","year":"2021","venue":"Nature Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mozannar and Sontag, 2020] Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning, pages 7076-7087. PMLR, 2020.","title":"Consistent estimators for learning to defer to an expert","authors":"Sontag Mozannar, Hussein Mozannar, David Sontag","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nguyen et al., 2018] An T Nguyen, Aditya Kharosekar, Saumyaa Krishnan, Siddhesh Krishnan, Elizabeth Tate, Byron C Wallace, and Matthew Lease. Believe it or not: Designing a human-ai partnership for mixed-initiative fact-checking. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, pages 189-199, 2018.","title":"Believe it or not: Designing a human-ai partnership for mixed-initiative fact-checking","authors":"Nguyen","year":"2018","venue":"Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Oikarinen et al., 2023] Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free con- cept bottleneck models. arXiv preprint arXiv:2304.06129, 2023.","title":"Label-free concept bottleneck models","authors":"Tuomas Oikarinen, Subhro Das, Tsui-Wei Lam M Nguyen, Weng","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Paszke et al., 2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural in- formation processing systems, 32, 2019.","title":"Pytorch: An imperative style, highperformance deep learning library","authors":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Peiffer-Smadja et al., 2020] Nathan Peiffer-Smadja, Timo- thy Miles Rawson, Raheelah Ahmad, Albert Buchard, P Georgiou, F-X Lescure, Gabriel Birgand, and Alison He- len Holmes. Machine learning for clinical decision support in infectious diseases: a narrative review of current appli- cations. Clinical Microbiology and Infection, 26(5):584- 595, 2020.","title":"Machine learning for clinical decision support in infectious diseases: a narrative review of current applications","authors":"Nathan Peiffer-Smadja, Timothy Miles Rawson, Raheelah Ahmad, Albert Buchard, P Georgiou, F-X Lescure, Gabriel Birgand, Alison Helen Holmes","year":"2020","venue":"Clinical Microbiology and Infection","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Rajpurkar et al., 2020] Pranav Rajpurkar, Chloe O'Connell, Amit Schechter, Nishit Asnani, Jason Li, Amirhossein Kiani, Robyn L Ball, Marc Mendelson, Gary Maartens, Daniël J van Hoving, et al. Chexaid: deep learning assis- tance for physician diagnosis of tuberculosis using chest x-rays in patients with hiv. NPJ digital medicine, 3(1):115, 2020.","title":"Chexaid: deep learning assistance for physician diagnosis of tuberculosis using chest x-rays in patients with hiv","authors":"Pranav Rajpurkar, Chloe O' Connell, Amit Schechter, Nishit Asnani, Jason Li, Amirhossein Kiani, Robyn L Ball, Marc Mendelson, Gary Maartens, J Daniël, Van Hoving","year":"2020","venue":"NPJ digital medicine","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Shankar et al., 2017] Devashish Shankar, Sujay Naru- manchi, HA Ananya, Pramod Kompalli, and Krishnendu Chaudhury. Deep learning based large scale visual rec- ommendation and search for e-commerce. arXiv preprint arXiv:1703.02344, 2017.","title":"Deep learning based large scale visual recommendation and search for e-commerce","authors":"Devashish Shankar, Sujay Narumanchi, Pramod Ha Ananya, Krishnendu Kompalli, Chaudhury","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Sharif Razavian et al., 2014] Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn features off-the-shelf: an astounding baseline for recogni- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 806-813, 2014.","title":"Cnn features off-the-shelf: an astounding baseline for recognition","authors":"Sharif Razavian","year":"2014","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition workshops","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sheth and Ebrahimi Kahou, 2024] Ivaxi Sheth and Samira Ebrahimi Kahou. Auxiliary losses for learning general- izable concept-based models. Advances in Neural Infor- mation Processing Systems, 36, 2024.","title":"Auxiliary losses for learning generalizable concept-based models","authors":"Ebrahimi Sheth, Ivaxi Kahou, Samira Ebrahimi Sheth, Kahou","year":"2024","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wah et al., 2011] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech- UCSD Birds-200-2011 Dataset. Jul 2011.","title":"The Caltech-UCSD Birds-200-2011 Dataset","authors":"Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, Serge Belongie","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wan et al., 2014] Ji Wan, Dayong Wang, Steven Chu Hong Hoi, Pengcheng Wu, Jianke Zhu, Yongdong Zhang, and Jintao Li. Deep learning for content-based image retrieval: A comprehensive study. In Proceedings of the 22nd ACM international conference on Multimedia, pages 157-166, 2014.","title":"Deep learning for content-based image retrieval: A comprehensive study","authors":"Ji Wan, Dayong Wang, Steven Chu, Hong Hoi, Pengcheng Wu, Jianke Zhu, Yongdong Zhang, Jintao Li","year":"2014","venue":"Proceedings of the 22nd ACM international conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wang et al., 2020] Zheng Wang, Xin Yuan, Toshihiko Ya- masaki, Yutian Lin, Xin Xu, and Wenjun Zeng. Re- identification= retrieval+ verification: Back to essence and forward with a new metric. arXiv preprint arXiv:2011.11506, 2020.","title":"Re-identification= retrieval+ verification: Back to essence and forward with a new metric","authors":"Zheng Wang, Xin Yuan, Toshihiko Yamasaki, Yutian Lin, Xin Xu, Wenjun Zeng","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zarlenga et al., 2023] Mateo Espinosa Zarlenga, Kather- ine M Collins, Krishnamurthy Dvijotham, Adrian Weller, Zohreh Shams, and Mateja Jamnik. Learning to re- ceive help: Intervention-aware concept embedding mod- els. arXiv preprint arXiv:2309.16928, 2023.","title":"Learning to receive help: Intervention-aware concept embedding models","authors":"Mateo Espinosa Zarlenga, Katherine M Collins, Krishnamurthy Dvijotham, Adrian Weller, Zohreh Shams, Mateja Jamnik","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}