{"type":"ScholarlyArticle","title":"Zero-Shot Sketch Based Image Retrieval via Modality Capacity Guidance","authors":[{"name":"Yanghong Zhou","slug":"yanghong-zhou","given":"Yanghong","family":"Zhou","affiliation":"Research Centre of Textiles for Future Fashion , The Hong Kong Polytechnic University; School of Fashion and Textiles , The Hong Kong Polytechnic University"},{"name":"Dawei Liu","slug":"dawei-liu","given":"Dawei","family":"Liu","affiliation":"School of Fashion and Textiles , The Hong Kong Polytechnic University"},{"name":"P. Y. Mok","slug":"p-y-mok","given":"P. Y.","family":"Mok","affiliation":"School of Fashion and Textiles , The Hong Kong Polytechnic University; Research Institute for Intelligent Wearable Systems , The Hong Kong Polytechnic University; Research Centre of Textiles for Future Fashion , The Hong Kong Polytechnic University"}],"abstract":"Zero-shot sketch-based image retrieval (ZS-SBIR), aiming to recognize and retrieve relevant photos based on freehand sketch queries that belong to unseen categories in the search set, has sparked considerable interest, benefiting from the rapid advancements in multimodal learning and feature representation research. Despite the recent improvements in performance, there are still rooms for refining feature representation and thus enhancing the generalization capabilities of the models. Most of the existing research efforts have primarily focused on learning the feature distribution of modalities within specific datasets, without considering the broader dataset-agnostic `population distribution' of relevant modalities. In this paper, we investigate the modality population distribution and apply such knowledge to guide feature learning. Specifically, we propose a modality capacity constraint loss to control the learning of population distribution for sketches and photos. This loss can be effectively combined with retrieval loss (e.g., triplet loss) or classification loss (e.g., InfoNCE loss) to enhance the performance of ZS-SBIR, through the fine-tuning process of pre-trained models like CLIP and DINO. Extensive experiment results have demonstrated our significant performance improvements, achieving  an increase of 7.3%/3.2% and 19.9%/10.3% in terms of mAP@200/P@200 compared to the state-of-the-art models on CLIP and DINO, respectively, on the Sketchy-ext dataset (split 2). Data, code, and supplementary information are available at https://github.com/YHdian0716/ZS-SBIR-MCC.git.","doi":"10.24963/ijcai.2024/197","url":"https://ijcai.org/proceedings/2024/197","pdf":"https://ijcai.org/proceedings/2024/0197.pdf","markdown":"https://ijcai.org/proceedings/2024/197.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/197","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":197,"pages":{"first":1780,"last":1787},"keywords":["Computer Vision: CV: Multimodal learning","Computer Vision: CV: Image and video retrieval","Computer Vision: CV: Recognition (object detection, categorization)","Computer Vision: CV: Representation learning"],"videos":[],"references":[{"position":1,"raw":"[Akata et al., 2016] Zeynep Akata, Mateusz Malinowski, Mario Fritz, and Bernt Schiele. Multi-cue zero-shot learn- ing with strong supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 59-68, 2016.","title":"Multi-cue zero-shot learning with strong supervision","authors":"Zeynep Akata, Mateusz Malinowski, Mario Fritz, Bernt Schiele","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Brunelli and Mich, 2001] Roberto Brunelli and Ornella Mich. Histograms analysis for image retrieval. Pattern Recognition, 34(8):1625-1637, 2001.","title":"Histograms analysis for image retrieval","authors":"Roberto Brunelli, Ornella Mich","year":"2001","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2018] Long Chen, Hanwang Zhang, Jun Xiao, Wei Liu, and Shih-Fu Chang. Zero-shot visual recog- nition using semantics-preserving adversarial embedding networks. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pages 1043-1052, 2018.","title":"Zero-shot visual recognition using semantics-preserving adversarial embedding networks","authors":"Long Chen, Hanwang Zhang, Jun Xiao, Wei Liu, Shih-Fu Chang","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dey et al., 2019a] Sounak Dey, Pau Riba, Anjan Dutta, Josep Llados, and Yi-Zhe Song. Doodle to search: Prac- tical zero-shot sketch-based image retrieval. In CVPR [2019b], pages 2179-2188.","title":"Doodle to search: Practical zero-shot sketch-based image retrieval","authors":"Dey","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dey et al., 2019b] Sounak Dey, Pau Riba, Anjan Dutta, Josep Llados, and Yi-Zhe Song. Doodle to search: Prac- tical zero-shot sketch-based image retrieval. In CVPR, pages 2179-2188, 2019.","title":"Doodle to search: Practical zero-shot sketch-based image retrieval","authors":"Dey","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dong et al., 2023] Shiyin Dong, Mingrui Zhu, Nannan Wang, Heng Yang, and Xinbo Gao. Adapt and align to im- prove zero-shot sketch-based image retrieval. arXiv, 2023.","title":"Adapt and align to improve zero-shot sketch-based image retrieval","authors":"Shiyin Dong, Mingrui Zhu, Nannan Wang, Heng Yang, Xinbo Gao","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dutta and Akata, 2019] Anjan Dutta and Zeynep Akata. Se- mantically tied paired cycle consistency for zero-shot sketch-based image retrieval. In CVPR, pages 5089-5098, 2019.","title":"Semantically tied paired cycle consistency for zero-shot sketch-based image retrieval","authors":"Anjan Dutta, Zeynep Akata","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Eitz et al., 2012] Mathias Eitz, James Hays, and Marc Alexa. How do humans sketch objects? ACM TOG, 31(4):1-10, 2012.","title":"How do humans sketch objects?","authors":"Eitz","year":"2012","venue":"ACM TOG","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Liu et al., 2017] Li Liu, Fumin Shen, Yuming Shen, Xiang- long Liu, and Ling Shao. Deep sketch hashing: Fast free- hand sketch-based image retrieval. In CVPR, pages 2862- 2871, 2017.","title":"Deep sketch hashing: Fast freehand sketch-based image retrieval","authors":"Li Liu, Fumin Shen, Yuming Shen, Xianglong Liu, Ling Shao","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Luo et al., 2020] Yuxuan Luo, Xizhao Wang, and Weipeng Cao. A novel dataset-specific feature extractor for zero- shot learning. Neurocomputing, 391:74-82, 2020.","title":"A novel dataset-specific feature extractor for zeroshot learning","authors":"Yuxuan Luo, Xizhao Wang, Weipeng Cao","year":"2020","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ren et al., 2023] Hao Ren, Ziqiang Zheng, Yang Wu, Hong Lu, Yang Yang, Ying Shan, and Sai-Kit Yeung. Ac- net: Approaching-and-centralizing network for zero-shot sketch-based image retrieval. IEEE TCSVT, 2023.","title":"Acnet: Approaching-and-centralizing network for zero-shot sketch-based image retrieval","authors":"Ziqiang Hao Ren, Yang Zheng, Hong Wu, Yang Lu, Ying Yang, Sai-Kit Shan, Yeung","year":"2023","venue":"IEEE TCSVT","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Sain et al., 2023a] Aneeshan Sain, Ayan Kumar Bhunia, Pinaki Nath Chowdhury, Subhadeep Koley, Tao Xiang, and Yi-Zhe Song. Clip for all things zero-shot sketch- based image retrieval, fine-grained or not. In CVPR, pages 2765-2775, 2023.","title":"Clip for all things zero-shot sketchbased image retrieval, fine-grained or not","authors":"Sain","year":"2023","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Sain et al., 2023b] Aneeshan Sain, Ayan Kumar Bhunia, Pinaki Nath Chowdhury, Subhadeep Koley, Tao Xiang, and Yi-Zhe Song. Clip for all things zero-shot sketch- based image retrieval, fine-grained or not. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 2765-2775, 2023.","title":"Clip for all things zero-shot sketchbased image retrieval, fine-grained or not","authors":"Sain","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Shigeto et al., 2015] Yutaro Shigeto, Ikumi Suzuki, Kazuo Hara, Masashi Shimbo, and Yuji Matsumoto. Ridge re- gression, hubness, and zero-shot learning. In Machine Learning and Knowledge Discovery in Databases: Eu- ropean Conference, ECML PKDD 2015, Porto, Portugal, September 7-11, 2015, Proceedings, Part I 15, pages 135- 151. Springer, 2015.","title":"Ridge regression, hubness, and zero-shot learning","authors":"Shigeto","year":"2015","venue":"Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2015","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Stricker and Swain, 1994] Stricker and Swain. The capacity of color histogram indexing. In 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 704-708. IEEE, 1994.","title":"The capacity of color histogram indexing","authors":"Swain ; Stricker, Stricker, Swain","year":"1994","venue":"Proceedings of IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tian et al., 2022] Jialin Tian, Xing Xu, Fumin Shen, Yang Yang, and Heng Tao Shen. Tvt: Three-way vision transformer through multi-modal hypersphere learning for zero-shot sketch-based image retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2370-2378, 2022.","title":"Tvt: Three-way vision transformer through multi-modal hypersphere learning for zero-shot sketch-based image retrieval","authors":"Jialin Tian, Xing Xu, Fumin Shen, Yang Yang, Heng Tao Shen","year":"2022","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tian et al., 2023] Jialin Tian, Xing Xu, Zuo Cao, Gong Zhang, Fumin Shen, and Yang Yang. Zero-shot sketch- based image retrieval with adaptive balanced discrim- inability and generalizability. In Proceedings of the 2023 ACM International Conference on Multimedia Retrieval, pages 407-415, 2023.","title":"Zero-shot sketchbased image retrieval with adaptive balanced discriminability and generalizability","authors":"Jialin Tian, Xing Xu, Zuo Cao, Gong Zhang, Fumin Shen, Yang Yang","year":"2023","venue":"Proceedings of the 2023 ACM International Conference on Multimedia Retrieval","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2021a] Wenjie Wang, Yufeng Shi, Shiming Chen, Qinmu Peng, Feng Zheng, and Xinge You. Norm- guided adaptive visual embedding for zero-shot sketch- based image retrieval. In IJCAI, pages 1106-1112, 2021.","title":"Normguided adaptive visual embedding for zero-shot sketchbased image retrieval","authors":"Wang","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2021b] Zhipeng Wang, Hao Wang, Jiexi Yan, Aming Wu, and Cheng Deng. Domain-smoothing network for zero-shot sketch-based image retrieval. arXiv preprint arXiv:2106.11841, 2021.","title":"Domain-smoothing network for zero-shot sketch-based image retrieval","authors":"Wang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2021/158"},{"position":20,"raw":"[Wang et al., 2022] Kai Wang, Yifan Wang, Xing Xu, Xin Liu, Weihua Ou, and Huimin Lu. Prototype-based selec- tive knowledge distillation for zero-shot sketch based im- age retrieval. In Proceedings of the 30th ACM Interna- tional Conference on Multimedia, pages 601-609, 2022.","title":"Prototype-based selective knowledge distillation for zero-shot sketch based image retrieval","authors":"Kai Wang, Yifan Wang, Xing Xu, Xin Liu, Weihua Ou, Huimin Lu","year":"2022","venue":"Proceedings of the 30th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang et al., 2017] Li Zhang, Tao Xiang, and Shaogang Gong. Learning a deep embedding model for zero-shot learning. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pages 2021-2030, 2017.","title":"Learning a deep embedding model for zero-shot learning","authors":"Li Zhang, Tao Xiang, Shaogang Gong","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}