{"type":"ScholarlyArticle","title":"Visual Similarity Attention","authors":[{"name":"Meng Zheng","slug":"meng-zheng","given":"Meng","family":"Zheng","affiliation":"United Imaging Intelligence , Cambridge MA , USA"},{"name":"Srikrishna Karanam","slug":"srikrishna-karanam","given":"Srikrishna","family":"Karanam","affiliation":"United Imaging Intelligence , Cambridge MA , USA"},{"name":"Terrence Chen","slug":"terrence-chen","given":"Terrence","family":"Chen","affiliation":"United Imaging Intelligence , Cambridge MA , USA"},{"name":"Richard J. Radke","slug":"richard-j-radke","given":"Richard J.","family":"Radke","affiliation":"Rensselaer Polytechnic Institute , Troy NY , USA"},{"name":"Ziyan Wu","slug":"ziyan-wu","given":"Ziyan","family":"Wu","affiliation":"United Imaging Intelligence , Cambridge MA , USA"}],"abstract":"While there has been substantial progress in learning suitable distance metrics, these techniques in general lack transparency and decision reasoning, i.e., explaining why the input set of images is similar or dissimilar. In this work, we solve this key problem by proposing the first method to generate generic visual similarity explanations with gradient-based attention. We demonstrate that our technique is agnostic to the specific similarity model type, e.g., we show applicability to Siamese, triplet, and quadruplet models. Furthermore, we make our proposed similarity attention a principled part of the learning process, resulting in a new paradigm for learning similarity functions. We demonstrate that our learning mechanism results in more generalizable, as well as explainable, similarity models. Finally, we demonstrate the generality of our framework by means of experiments on a variety of tasks, including image retrieval, person re-identification, and low-shot semantic segmentation.","doi":"10.24963/ijcai.2022/241","url":"https://ijcai.org/proceedings/2022/241","pdf":"https://ijcai.org/proceedings/2022/0241.pdf","markdown":"https://ijcai.org/proceedings/2022/241.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/241","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":241,"pages":{"first":1728,"last":1735},"keywords":["Computer Vision: Image and Video retrieval","Computer Vision: Segmentation","Computer Vision: Transfer, low-shot, semi- and un- supervised learning","Computer Vision: Representation Learning","Computer Vision: Interpretability and Transparency"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986515","length_sec":86,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984658","length_sec":361,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Caelles et al., 2017] Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool. One-shot video object segmentation. In CVPR, 2017.","title":"One-shot video object segmentation","authors":"Kevis-Kokitsi Sergi Caelles, Jordi Maninis, Laura Pont-Tuset, Daniel Leal-Taixé, Luc Cremers, Van Gool","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen and Deng, 2019] B. Chen and W. Deng. Hybrid-attention based decoupled metric learning for zero-shot image retrieval. In CVPR, 2019.","title":"Hybrid-attention based decoupled metric learning for zero-shot image retrieval","authors":"B Chen, W Deng","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2020] L. Chen, J. Chen, H. Hajimirsadeghi, and G. Mori. Adapting grad-cam for embedding networks. In WACV, 2020.","title":"Adapting grad-cam for embedding networks","authors":"L Chen, J Chen, H Hajimirsadeghi, G Mori","year":"2020","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dong and Xing, 2018] Nanqing Dong and Eric P. Xing. Few-shot semantic segmentation with prototype learning. In BMVC, 2018.","title":"Few-shot semantic segmentation with prototype learning","authors":"Dong, Xing ;, Nanqing Dong, Eric P Xing","year":"2018","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Elezi et al., 2020] Ismail Elezi, Sebastiano Vascon, Alessandro Tor- cinovich, Marcello Pelillo, and Laura Leal-Taixe. The group loss for deep metric learning. In ECCV, 2020.","title":"The group loss for deep metric learning","authors":"Ismail Elezi, Sebastiano Vascon, Alessandro Torcinovich, Marcello Pelillo, Laura Leal-Taixe","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kim et al., 2018] Wonsik Kim, Bhavya Goyal, Kunal Chawla, Jungmin Lee, and Keunjoo Kwon. Attention-based ensemble for deep metric learning. In ECCV, 2018.","title":"Attention-based ensemble for deep metric learning","authors":"Wonsik Kim, Bhavya Goyal, Kunal Chawla, Jungmin Lee, Keunjoo Kwon","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Krause et al., 2013] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categoriza- tion. In ICCVW, pages 554-561, 2013.","title":"3d object representations for fine-grained categorization","authors":"Jonathan Krause, Michael Stark, Jia Deng, Li Fei-Fei","year":"2013","venue":"ICCVW","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Law et al., 2013] Marc T Law, Nicolas Thome, and Matthieu Cord. Quadruplet-wise image similarity learning. In ICCV, 2013.","title":"Quadruplet-wise image similarity learning","authors":"Nicolas Marc T Law, Matthieu Thome, Cord","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2018a] Kunpeng Li, Ziyan Wu, Kuan-Chuan Peng, Jan Ernst, and Yun Fu. Tell me where to look: Guided attention inference network. In CVPR, 2018.","title":"Tell me where to look: Guided attention inference network","authors":"Li","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2018b] Wei Li, Xiatian Zhu, and Shaogang Gong. Har- monious attention network for person re-identification. In CVPR, 2018.","title":"Harmonious attention network for person re-identification","authors":"Li","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mahendran and Vedaldi, 2015] Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In CVPR, 2015.","title":"Understanding deep image representations by inverting them","authors":"Vedaldi Mahendran, Aravindh Mahendran, Andrea Vedaldi","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Movshovitz-Attias et al., 2017] Y. Movshovitz-Attias, A. Toshev, T. K. Leung, S. Ioffe, and S. P. Singh. No fuss distance metric learning using proxies. In ICCV, 2017.","title":"No fuss distance metric learning using proxies","authors":"Y Movshovitz-Attias, A Toshev, T K Leung, S Ioffe, S P Singh","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Opitz et al., 2017] Michael Opitz, Georg Waltner, Horst Possegger, and Horst Bischof. BIER --boosting independent embeddings robustly. In ICCV, 2017.","title":"BIER --boosting independent embeddings robustly","authors":"Michael Opitz, Georg Waltner, Horst Possegger, Horst Bischof","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Plummer et al., 2020] Bryan Plummer, Mariya Vasileva, Vitali Pet- siuk, Kate Saenko, and David Forsyth. Why do these match? explaining the behavior of image similarity models. In ECCV, 2020.","title":"Why do these match? explaining the behavior of image similarity models","authors":"Bryan Plummer, Mariya Vasileva, Vitali Petsiuk, Kate Saenko, David Forsyth","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Qi et al., 2020] Qi Qi, Yan Yan, Zixuan Wu, Xiaoyu Wang, and Tianbao Yang. A simple and effective framework for pairwise deep metric learning. In ECCV, 2020.","title":"A simple and effective framework for pairwise deep metric learning","authors":"Qi Qi, Yan Yan, Zixuan Wu, Xiaoyu Wang, Tianbao Yang","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rakelly et al., 2018] Kate Rakelly, Evan Shelhamer, Trevor Darrell, Alyosha Efros, and Sergey Levine. Conditional networks for few- shot semantic segmentation. In ICLR Workshops, 2018.","title":"Conditional networks for fewshot semantic segmentation","authors":"Kate Rakelly, Evan Shelhamer, Trevor Darrell, Alyosha Efros, Sergey Levine","year":"2018","venue":"ICLR Workshops","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rippel et al., 2016] Oren Rippel, Manohar Paluri, Piotr Dollar, and Lubomir Bourdev. Metric learning with adaptive density discrimi- nation. In ICLR, 2016.","title":"Metric learning with adaptive density discrimination","authors":"Oren Rippel, Manohar Paluri, Piotr Dollar, Lubomir Bourdev","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ristani et al., 2016] Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi. Performance measures and a data set for multi-target, multi-camera tracking. In ECCVW, 2016.","title":"Performance measures and a data set for multi-target, multi-camera tracking","authors":"Ristani","year":"2016","venue":"ECCVW","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rother et al., 2004] Carsten Rother, Vladimir Kolmogorov, and An- drew Blake. Grabcut: Interactive foreground extraction using iterated graph cuts. In ACM transactions on graphics (TOG), volume 23, pages 309-314, 2004.","title":"Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. Grabcut: Interactive foreground extraction using iterated graph cuts","authors":"Rother","year":"2004","venue":"ACM transactions on graphics (TOG)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Selvaraju et al., 2017] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, 2017.","title":"Grad-cam: Visual explanations from deep networks via gradient-based localization","authors":"R Ramprasaath, Michael Selvaraju, Abhishek Cogswell, Ramakrishna Das, Devi Vedantam, Dhruv Parikh, Batra","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shaban et al., 2017] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. In BMVC, 2017.","title":"One-shot learning for semantic segmentation","authors":"Shaban","year":"2017","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Si et al., 2018] Jianlou Si, Honggang Zhang, Chun-Guang Li, Ja- son Kuen, Xiangfei Kong, Alex C Kot, and Gang Wang. Dual attention matching network for context-aware feature sequence based person re-identification. In CVPR, 2018.","title":"Dual attention matching network for context-aware feature sequence based person re-identification","authors":"Jianlou Si, Honggang Zhang, Chun-Guang Li, Jason Kuen, Xiangfei Kong, Alex C Kot, Gang Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sohn, 2016] K. Sohn. Improved deep metric learning with multi- class N-pair loss objective. In NIPS. 2016.","title":"Improved deep metric learning with multiclass N-pair loss objective","authors":"K Sohn","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Song et al., 2016] Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In CVPR, 2016.","title":"Deep metric learning via lifted structured feature embedding","authors":"Hyun Oh Song, Yu Xiang, Stefanie Jegelka, Silvio Savarese","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sun et al., 2017] Yifan Sun, Liang Zheng, Weijian Deng, and Shengjin Wang. SVDNet for pedestrian retrieval. In ICCV, 2017.","title":"SVDNet for pedestrian retrieval","authors":"Yifan Sun, Liang Zheng, Weijian Deng, Shengjin Wang","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sun et al., 2018] Yifan Sun, Liang Zheng, Yi Yang, Qi Tian, and Shengjin Wang. Beyond part models: Person retrieval with refined part pooling. In ECCV, 2018.","title":"Beyond part models: Person retrieval with refined part pooling","authors":"Yifan Sun, Liang Zheng, Yi Yang, Qi Tian, Shengjin Wang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wah et al., 2011] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200- 2011 dataset. 2011.","title":"The caltech-ucsd birds-200-2011 dataset","authors":"Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, Serge Belongie","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2018] Guanshuo Wang, Yufeng Yuan, Xiong Chen, Jiwei Li, and Xi Zhou. Learning Discriminative Features with Multiple Granularities for Person Re-Identification. In ACM MM, 2018.","title":"Learning Discriminative Features with Multiple Granularities for Person Re-Identification","authors":"Guanshuo Wang, Yufeng Yuan, Xiong Chen, Jiwei Li, Xi Zhou","year":"2018","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2019a] K. Wang, J. H. Liew, Y. Zou, D. Zhou, and J. Feng. Panet: Few-shot image semantic segmentation with prototype alignment. In ICCV, 2019.","title":"Panet: Few-shot image semantic segmentation with prototype alignment","authors":"Wang","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2019b] L. Wang, Z. Wu, S. Karanam, K. Peng, R. V. Singh, and B. D. L. Metaxas. Sharpen focus: Learning with attention separability and consistency. In ICCV, 2019.","title":"Sharpen focus: Learning with attention separability and consistency","authors":"Wang","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2019c] Xun Wang, Xintong Han, Weiling Huang, Dengke Dong, and Matthew R. Scott. Multi-similarity loss with general pair weighting for deep metric learning. In CVPR, 2019.","title":"Multi-similarity loss with general pair weighting for deep metric learning","authors":"Wang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wu et al., 2017] Chao-Yuan Wu, R Manmatha, Alexander J Smola, and Philipp Krahenbuhl. Sampling matters in deep embedding learning. In ICCV, 2017.","title":"Sampling matters in deep embedding learning","authors":"Chao-Yuan, R Wu, Alexander J Manmatha, Philipp Smola, Krahenbuhl","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yuan et al., 2017] Yuhui Yuan, Kuiyuan Yang, and Chao Zhang. Hard-aware deeply cascaded embedding. In ICCV, 2017.","title":"Hard-aware deeply cascaded embedding","authors":"Yuhui Yuan, Kuiyuan Yang, Chao Zhang","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zheng et al., 2019a] M. Zheng, S. Karanam, z. Wu, and R. J. Radke. Re-identification with consistent attentive siamese networks. In CVPR, 2019.","title":"Re-identification with consistent attentive siamese networks","authors":"Zheng","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zheng et al., 2019b] W. Zheng, Z. Chen, J. Lu, and J. Zhou. Hardness-aware deep metric learning. In CVPR, 2019.","title":"Hardness-aware deep metric learning","authors":"Zheng","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhong et al., 2017] Zhun Zhong, Liang Zheng, Donglin Cao, and Shaozi Li. Re-ranking person re-identification with k-reciprocal encoding. In CVPR, 2017.","title":"Re-ranking person re-identification with k-reciprocal encoding","authors":"Zhun Zhong, Liang Zheng, Donglin Cao, Shaozi Li","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhou et al., 2016] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, 2016.","title":"Learning deep features for discriminative localization","authors":"Zhou","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhu et al., 2020] Yuke Zhu, Yan Bai, and Yichen Wei. Spherical feature transform for deep metric learning. In ECCV, 2020.","title":"Spherical feature transform for deep metric learning","authors":"Yuke Zhu, Yan Bai, Yichen Wei","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}