{"type":"ScholarlyArticle","title":"Probabilistic Feature Matching for Fast Scalable Visual Prompting","authors":[{"name":"Thomas Frick","slug":"thomas-frick","given":"Thomas","family":"Frick","affiliation":"IBM Research"},{"name":"Cezary Skura","slug":"cezary-skura","given":"Cezary","family":"Skura","affiliation":"IBM Research"},{"name":"Filip M. Janicki","slug":"filip-m-janicki","given":"Filip M.","family":"Janicki","affiliation":"IBM Research"},{"name":"Roy Assaf","slug":"roy-assaf","given":"Roy","family":"Assaf","affiliation":"IBM Research"},{"name":"Niccolo Avogaro","slug":"niccolo-avogaro","given":"Niccolo","family":"Avogaro","affiliation":"IBM Research"},{"name":"Daniel Caraballo","slug":"daniel-caraballo","given":"Daniel","family":"Caraballo","affiliation":"IBM Research"},{"name":"Yagmur G. Cinar","slug":"yagmur-g-cinar","given":"Yagmur G.","family":"Cinar","affiliation":"IBM Research"},{"name":"Brown Ebouky","slug":"brown-ebouky","given":"Brown","family":"Ebouky","affiliation":""},{"name":"Ioana Giurgiu","slug":"ioana-giurgiu","given":"Ioana","family":"Giurgiu","affiliation":"IBM Research"},{"name":"Takayuki Katsuki","slug":"takayuki-katsuki","given":"Takayuki","family":"Katsuki","affiliation":"IBM Research"},{"name":"Piotr Kluska","slug":"piotr-kluska","given":"Piotr","family":"Kluska","affiliation":"IBM Research"},{"name":"Cristiano Malossi","slug":"cristiano-malossi","given":"Cristiano","family":"Malossi","affiliation":"IBM Research"},{"name":"Haoxiang Qiu","slug":"haoxiang-qiu","given":"Haoxiang","family":"Qiu","affiliation":"IBM Research"},{"name":"Tomoya Sakai","slug":"tomoya-sakai","given":"Tomoya","family":"Sakai","affiliation":"IBM Research"},{"name":"Florian Scheidegger","slug":"florian-scheidegger","given":"Florian","family":"Scheidegger","affiliation":"IBM Research"},{"name":"Andrej Simeski","slug":"andrej-simeski","given":"Andrej","family":"Simeski","affiliation":"IBM Research"},{"name":"Daniel Yang","slug":"daniel-yang","given":"Daniel","family":"Yang","affiliation":"IBM Research"},{"name":"Andrea Bartezzaghi","slug":"andrea-bartezzaghi","given":"Andrea","family":"Bartezzaghi","affiliation":"IBM Research"},{"name":"Mattia Rigotti","slug":"mattia-rigotti","given":"Mattia","family":"Rigotti","affiliation":"IBM Research"}],"abstract":"In this work, we propose a novel framework for image segmentation guided by visual prompting which leverages the power of vision foundation models.\r\nInspired by recent advancements in computer vision, our approach integrates multiple large-scale pretrained models to address the challenges of segmentation tasks with limited and sparsely annotated data interactively provided by a user.\r\nOur method combines a frozen feature extraction backbone with a scalable and efficient probabilistic feature correspondence (soft matching) procedure derived from Optimal Transport to couple pixels between reference and target images.\r\nMoreover, a pretrained segmentation model is harnessed to translate user scribbles into reference masks and matched target pixels into output target segmentation masks.\r\nThis results in a framework that we name Softmatcher, a versatile and fast training-free architecture for image segmentation by visual prompting.\r\nWe demonstrate the efficiency and scalability of Softmatcher for real-time interactive image segmentation by visual prompting and showcase it in diverse visual domains including technical visual inspection use cases.","doi":"10.24963/ijcai.2024/1000","url":"https://ijcai.org/proceedings/2024/1000","pdf":"https://ijcai.org/proceedings/2024/1000.pdf","markdown":"https://ijcai.org/proceedings/2024/1000.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/1000","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Demo Track","track":"Demo Track","number":1000,"pages":{"first":8648,"last":8652},"keywords":["Humans and AI: HAI: Human-computer interaction","Computer Vision: CV: Applications","Computer Vision: CV: Segmentation","Computer Vision: CV: Machine learning for vision"],"videos":[],"references":[{"position":1,"raw":"[Bar et al., 2022] Amir Bar, Yossi Gandelsman, Trevor Dar- rell, Amir Globerson, and Alexei A. Efros. Visual Prompt- ing via Image Inpainting, September 2022.","title":"Visual Prompting via Image Inpainting","authors":"Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, Alexei A Efros","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bommasani et al., 2022] Rishi Bommasani, Drew A. Hud- son, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Dur- mus, Stefano Ermon, John Etchemendy, Kawin Etha- yarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Gross- man, Neel Guha, Tatsunori Hashimoto, Peter Hender- son, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Ku- mar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christo- pher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav San- thanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the Opportunities and Risks of Foun- dation Models, July 2022.","title":"On the Opportunities and Risks of Foundation Models","authors":"Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Michael S Sydney Von Arx, Jeannette Bernstein, Antoine Bohg, Emma Bosselut, Erik Brunskill, Shyamal Brynjolfsson, Dallas Buch, Rodrigo Card, Niladri Castellon, Annie Chatterji, Kathleen Chen, Jared Quincy Creel, Dora Davis, Chris Demszky, Moussa Donahue, Esin Doumbouya, Stefano Durmus, John Ermon, Kawin Etchemendy, Li Ethayarajh, Chelsea Fei-Fei, Trevor Finn, Lauren Gale, Karan Gillespie, Noah Goel, Shelby Goodman, Neel Grossman, Tatsunori Guha, Peter Hashimoto, John Henderson, Daniel E Hewitt, Jenny Ho, Kyle Hong, Jing Hsu, Thomas Huang, Saahil Icard, Dan Jain, Pratyusha Jurafsky, Siddharth Kalluri, Geoff Karamcheti, Fereshte Keeling, Omar Khani, Pang Khattab, Mark Wei Koh, Ranjay Krass, Rohith Krishna, Ananya Kuditipudi, Faisal Kumar, Mina Ladhak, Tony Lee, Jure Lee, Isabelle Leskovec, Levent, Lisa Xiang, Xuechen Li, Tengyu Li, Ali Ma, Christopher D Malik, Suvir Manning, Eric Mirchandani, Zanele Mitchell, Suraj Munyikwa, Avanika Nair, Deepak Narayan, Ben Narayanan, Allen Newman, Juan Carlos Nie, Hamed Niebles, Julian Nilforoshan, Giray Nyarko, Laurel Ogut, Isabel Orr, Papadimitriou, Sung Joon, Chris Park, Eva Piech, Christopher Portelance, Aditi Potts, Rob Raghunathan, Hongyu Reich, Frieda Ren, Yusuf Rong, Camilo Roohani, Jack Ruiz, Christopher Ryan, Dorsa Ré, Shiori Sadigh, Keshav Sagawa, Andy Santhanam, Krishnan Shih, Alex Srinivasan, Rohan Tamkin, Armin W Taori, Florian Thomas, Rose E Tramèr, William Wang, Bohan Wang, Jiajun Wu, Yuhuai Wu, Sang Wu, Michihiro Michael Xie, Jiaxuan Yasunaga, Matei You, Michael Zaharia, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zhang, Kaitlyn Zheng, Percy Zhou, Liang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Caron et al., 2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers. arXiv:2104.14294 [cs], April 2021.","title":"Emerging Properties in Self-Supervised Vision Transformers","authors":"Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Choromanski et al., 2020] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Col- well, and Adrian Weller. Rethinking Attention with Per- formers. arXiv:2009.14794 [cs, stat], September 2020.","title":"Rethinking Attention with Performers","authors":"Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, Adrian Weller","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Crouse, 2016] David F. Crouse. On implementing 2D rect- angular assignment algorithms. IEEE Transactions on Aerospace and Electronic Systems, 52(4):1679-1696, Au- gust 2016.","title":"On implementing 2D rectangular assignment algorithms","authors":"F David, Crouse","year":"2016","venue":"IEEE Transactions on Aerospace and Electronic Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dognin et al., 2019] Pierre Dognin, Igor Melnyk, Youssef Mroueh, Jerret Ross, Cicero Dos Santos, and Tom Sercu. Wasserstein Barycenter Model Ensembling, Febru- ary 2019.","title":"Wasserstein Barycenter Model Ensembling","authors":"Dognin","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gupta and Kembhavi, 2022] Tanmay Gupta and Aniruddha Kembhavi. Visual Programming: Compositional visual reasoning without training, November 2022.","title":"Visual Programming: Compositional visual reasoning without training","authors":"Kembhavi ; Gupta, Aniruddha Gupta, Kembhavi","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ito et al., 2022] T. Ito, T. Klinger, D.H. Schultz, J.D. Mur- ray, M.W. Cole, and M. Rigotti. Compositional general- ization through abstract representations in human and arti- ficial neural networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2022.","title":"Compositional generalization through abstract representations in human and artificial neural networks","authors":"T Ito, T Klinger, D H Schultz, J D Murray, M W Cole, M Rigotti","year":"2022","venue":"Advances in Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Janouskova et al., 2023] K. Janouskova, M. Rigotti, I. Giurgiu, and C. Malossi. Model-Assisted Labeling via Explainability for Visual Inspection of Civil Infrastruc- tures. In Computer Vision-ECCV 2022 Workshops: Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VII, pages 244-257. Springer, 2023.","title":"Model-Assisted Labeling via Explainability for Visual Inspection of Civil Infrastructures","authors":"K Janouskova, M Rigotti, I Giurgiu, C Malossi","year":"2022","venue":"Computer Vision-ECCV 2022 Workshops","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kirillov et al., 2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment Anything, April 2023.","title":"Segment Anything","authors":"Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, Piotr Dollár, Ross Girshick","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2020] Xiang Li, Tianhan Wei, Yau Pun Chen, Yu- Wing Tai, and Chi-Keung Tang. FSS-1000: A 1000-Class Dataset for Few-Shot Segmentation. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2866-2875, June 2020.","title":"FSS-1000: A 1000-Class Dataset for Few-Shot Segmentation","authors":"Xiang Li, Tianhan Wei, Yu-Wing Yau Pun Chen, Chi-Keung Tai, Tang","year":"2020","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2023] Yang Liu, Muzhi Zhu, Hengtao Li, Hao Chen, Xinlong Wang, and Chunhua Shen. Matcher: Seg- ment Anything with One Shot Using All-Purpose Feature Matching, May 2023.","title":"Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching","authors":"Yang Liu, Muzhi Zhu, Hengtao Li, Hao Chen, Xinlong Wang, Chunhua Shen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Oquab et al., 2023] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khali- dov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning Robust Visual Features without Super- vision, April 2023.","title":"DINOv2: Learning Robust Visual Features without Supervision","authors":"Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, Piotr Bojanowski","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Su- pervision. In Proceedings of the 38th International Con- ference on Machine Learning, pages 8748-8763. PMLR, July 2021.","title":"Learning Transferable Visual Models From Natural Language Supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever","year":"2021","venue":"Proceedings of the 38th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rahimi and Recht, 2007] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Ad- vances in Neural Information Processing Systems, pages 1177-1184, 2007.","title":"Random features for large-scale kernel machines","authors":"Recht Rahimi, Ali Rahimi, Benjamin Recht","year":"2007","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rigotti et al., 2023] M. Rigotti, D. Antognini, R. Assaf, K. Bakirci, T. Frick, I. Giurgiu, K. Janoušková, F. Janicki, H. Jubran, C. Malossi, A. Meterez, and F. Scheidegger. To- wards Workflows for the Use of AI Foundation Models in Visual Inspection Applications. ce/papers, 6(5):605-613, 2023.","title":"Towards Workflows for the Use of AI Foundation Models in Visual Inspection Applications","authors":"M Rigotti, D Antognini, R Assaf, K Bakirci, T Frick, I Giurgiu, K Janoušková, F Janicki, H Jubran, C Malossi, A Meterez, F Scheidegger","year":"2023","venue":"ce/papers","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Wang et al., 2023] Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seg- GPT: Segmenting Everything In Context, April 2023.","title":"Seg-GPT: Segmenting Everything In Context","authors":"Wang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Zhang et al., 2023] Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Hao Dong, Peng Gao, and Hongsheng Li. Personalize Segment Anything Model with One Shot, May 2023.","title":"Personalize Segment Anything Model with One Shot","authors":"Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}