{"type":"ScholarlyArticle","title":"SceneDiff: Generative Scene-Level Image Retrieval with Text and Sketch Using Diffusion Models","authors":[{"name":"Ran Zuo","slug":"ran-zuo","given":"Ran","family":"Zuo","affiliation":"Beijing Key Laboratory of Human-Computer Interaction , Institute of Software , Chinese Academy of Sciences; University of Chinese Academy of Sciences"},{"name":"Haoxiang Hu","slug":"haoxiang-hu","given":"Haoxiang","family":"Hu","affiliation":"Beijing Key Laboratory of Human-Computer Interaction , Institute of Software , Chinese Academy of Sciences; University of Chinese Academy of Sciences"},{"name":"Xiaoming Deng","slug":"xiaoming-deng","given":"Xiaoming","family":"Deng","affiliation":"Beijing Key Laboratory of Human-Computer Interaction , Institute of Software , Chinese Academy of Sciences; University of Chinese Academy of Sciences"},{"name":"Cangjun Gao","slug":"cangjun-gao","given":"Cangjun","family":"Gao","affiliation":"Beijing Key Laboratory of Human-Computer Interaction , Institute of Software , Chinese Academy of Sciences; University of Chinese Academy of Sciences"},{"name":"Zhengming Zhang","slug":"zhengming-zhang","given":"Zhengming","family":"Zhang","affiliation":"Beijing Key Laboratory of Human-Computer Interaction , Institute of Software , Chinese Academy of Sciences; University of Chinese Academy of Sciences"},{"name":"Yu-Kun Lai","slug":"yu-kun-lai","given":"Yu-Kun","family":"Lai","affiliation":"Cardiff University"},{"name":"Cuixia Ma","slug":"cuixia-ma","given":"Cuixia","family":"Ma","affiliation":"Beijing Key Laboratory of Human-Computer Interaction , Institute of Software , Chinese Academy of Sciences; University of Chinese Academy of Sciences; Key Laboratory of System Software and State Key Laboratory of Computer Science , Institute of Software , Chinese Academy of Sciences"},{"name":"Yong-Jin Liu","slug":"yong-jin-liu","given":"Yong-Jin","family":"Liu","affiliation":"Tsinghua University"},{"name":"Hongan Wang","slug":"hongan-wang","given":"Hongan","family":"Wang","affiliation":"Beijing Key Laboratory of Human-Computer Interaction , Institute of Software , Chinese Academy of Sciences; University of Chinese Academy of Sciences"}],"abstract":"Jointly using text and sketch for scene-level image retrieval utilizes the complementary between text and sketch to describe the fine-grained scene content and retrieve the target image, which plays a pivotal role in accurate image retrieval. Existing methods directly fuse the features of sketch and text and thus suffer from the bottleneck of limited utilization for crucial semantic and structural information, leading to inaccurate matching with images. In this paper,  we propose SceneDiff, a novel retrieval network that leverages a pre-trained diffusion model to establish a shared generative latent space, enabling a joint latent representation learning for both sketch and text features and precise alignment with the corresponding image. Specifically, we encode text, sketch and image features, and project them into the diffusion-based share space, conditioning the denoising process on sketch and text features to generate latent fusion features, while employing the pre-trained autoencoder for latent image features. Within this space, we introduce the content-aware feature transformation module to reconcile encoded sketch and image features with the diffusion latent space's dimensional requirements and preserve their visual content information. Then we augment the representation capability of the generated latent fusion features by integrating multiple samplings with partition attention, and utilize contrastive learning to align both direct fusion features and generated latent fusion features with corresponding image representations. Our method outperforms the state-of-the-art works through extensive experiments, providing a novel insight into the related retrieval field.","doi":"10.24963/ijcai.2024/202","url":"https://ijcai.org/proceedings/2024/202","pdf":"https://ijcai.org/proceedings/2024/0202.pdf","markdown":"https://ijcai.org/proceedings/2024/202.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/202","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":202,"pages":{"first":1825,"last":1833},"keywords":["Computer Vision: CV: Multimodal learning","Computer Vision: CV: Image and video retrieval"],"videos":[],"references":[{"position":1,"raw":"[Amit et al., 2021] Tomer Amit, Tal Shaharbany, Eliya Nachmani, and Lior Wolf. Segdiff: Image segmenta- tion with diffusion probabilistic models. arXiv preprint arXiv:2112.00390, 2021.","title":"Segdiff: Image segmentation with diffusion probabilistic models","authors":"Tomer Amit, Tal Shaharbany, Eliya Nachmani, Lior Wolf","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baranchuk et al., 2021] Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffusion mod- els. arXiv preprint arXiv:2112.03126, 2021.","title":"Label-efficient semantic segmentation with diffusion models","authors":"Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, Artem Babenko","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brempong et al., 2022] Emmanuel Asiedu Brempong, Si- mon Kornblith, Ting Chen, Niki Parmar, Matthias Min- derer, and Mohammad Norouzi. Denoising pretraining for semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4175-4186, 2022.","title":"Denoising pretraining for semantic segmentation","authors":"Emmanuel Asiedu, Brempong, Simon Kornblith, Ting Chen, Niki Parmar, Matthias Minderer, Mohammad Norouzi","year":"2022","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2023] Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for ob- ject detection. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 19830- 19843, 2023.","title":"Diffusiondet: Diffusion model for object detection","authors":"Chen","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chowdhury et al., 2022] Pinaki Nath Chowdhury, Anee- shan Sain, Ayan Kumar Bhunia, Tao Xiang, Yulia Gryadit- skaya, and Yi-Zhe Song. Fs-coco: Towards understanding of freehand sketches of common objects in context. In Eu- ropean Conference on Computer Vision, pages 253-270. Springer, 2022.","title":"Fs-coco: Towards understanding of freehand sketches of common objects in context","authors":"Pinaki Nath Chowdhury, Aneeshan Sain, Ayan Kumar Bhunia, Tao Xiang, Yulia Gryaditskaya, Yi-Zhe Song","year":"2022","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chowdhury et al., 2023] Pinaki Nath Chowdhury, Ayan Ku- mar Bhunia, Aneeshan Sain, Subhadeep Koley, Tao Xiang, and Yi-Zhe Song. Scenetrilogy: On human scene-sketch and its complementarity with photo and text. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10972-10983, 2023.","title":"Scenetrilogy: On human scene-sketch and its complementarity with photo and text","authors":"Pinaki Nath Chowdhury, Ayan Kumar Bhunia, Aneeshan Sain, Subhadeep Koley, Tao Xiang, Yi-Zhe Song","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dey et al., 2018] Sounak Dey, Anjan Dutta, Suman K Ghosh, Ernest Valveny, Josep Lladós, and Umapada Pal. Learning cross-modal deep embeddings for multi-object image retrieval using text and sketch. In 2018 24th inter- national conference on pattern recognition (ICPR), pages 916-921. IEEE, 2018.","title":"Learning cross-modal deep embeddings for multi-object image retrieval using text and sketch","authors":"Dey","year":"2018","venue":"2018 24th international conference on pattern recognition (ICPR)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gao et al., 2020] Chengying Gao, Qi Liu, Qi Xu, Limin Wang, Jianzhuang Liu, and Changqing Zou. Sketchycoco: Image generation from freehand scene sketches. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5174-5183, 2020.","title":"Sketchycoco: Image generation from freehand scene sketches","authors":"Gao","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Han and Schlangen, 2017] Ting Han and David Schlangen. Draw and tell: Multimodal descriptions outperform verbal-or sketch-only descriptions in an image retrieval task. In Proceedings of the Eighth International Joint Con- ference on Natural Language Processing (Volume 2: Short Papers), pages 361-365, 2017.","title":"Draw and tell: Multimodal descriptions outperform verbal-or sketch-only descriptions in an image retrieval task","authors":"Schlangen ; ] Ting Han, David Han, Schlangen","year":"2017","venue":"Proceedings of the Eighth International Joint Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jin et al., 2023] Peng Jin, Hao Li, Zesen Cheng, Kehan Li, Xiangyang Ji, Chang Liu, Li Yuan, and Jie Chen. Dif- fusionret: Generative text-video retrieval with diffusion model. arXiv preprint arXiv:2303.09867, 2023.","title":"Diffusionret: Generative text-video retrieval with diffusion model","authors":"Jin Peng, Hao Li, Zesen Cheng, Kehan Li, Xiangyang Ji, Chang Liu, Li Yuan, Jie Chen","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kipf and Welling, 2016] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.","title":"Semi-supervised classification with graph convolutional networks","authors":"N Thomas, Max Kipf, Welling","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2023] Pandeng Li, Chen-Wei Xie, Hongtao Xie, Liming Zhao, Lei Zhang, Yun Zheng, Deli Zhao, and Yongdong Zhang. Momentdiff: Generative video mo- ment retrieval from random to real. arXiv preprint arXiv:2307.02869, 2023.","title":"Momentdiff: Generative video moment retrieval from random to real","authors":"Pandeng Li, Chen-Wei Xie, Hongtao Xie, Liming Zhao, Lei Zhang, Yun Zheng, Deli Zhao, Yongdong Zhang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, Septem- ber 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"Computer Vision-ECCV 2014: 13th European Conference","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2020] Fang Liu, Changqing Zou, Xiaoming Deng, Ran Zuo, Yu-Kun Lai, Cuixia Ma, Yong-Jin Liu, and Hongan Wang. Scenesketcher: Fine-grained image retrieval with scene sketches. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIX 16, pages 718-734. Springer, 2020.","title":"Scenesketcher: Fine-grained image retrieval with scene sketches","authors":"Fang Liu, Changqing Zou, Xiaoming Deng, Ran Zuo, Yu-Kun Lai, Cuixia Ma, Yong-Jin Liu, Hongan Wang","year":"2020","venue":"Computer Vision-ECCV 2020: 16th European Conference","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mou et al., 2023] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more control- lable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453, 2023.","title":"T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models","authors":"Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, Xiaohu Qie","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Oord et al., 2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03748, 2018.","title":"Representation learning with contrastive predictive coding","authors":"Aaron Van Den Oord, Yazhe Li, Oriol Vinyals","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rombach et al., 2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.","title":"High-resolution image synthesis with latent diffusion models","authors":"Rombach","year":"2022","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sangkloy et al., 2022] Patsorn Sangkloy, Wittawat Jitkrit- tum, Diyi Yang, and James Hays. A sketch is worth a thou- sand words: Image retrieval with text and sketch. In Eu- ropean Conference on Computer Vision, pages 251-267.","title":"Patsorn Sangkloy, Wittawat Jitkrittum, Diyi Yang, and James Hays. A sketch is worth a thousand words: Image retrieval with text and sketch","authors":"Sangkloy","year":"2022","venue":"Eu-ropean Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Song et al., 2017] Jifei Song, Yi-Zhe Song, Tony Xiang, and Timothy M Hospedales. Fine-grained image retrieval: the text/sketch input dilemma. In BMVC, volume 2, page 7, 2017.","title":"Fine-grained image retrieval: the text/sketch input dilemma","authors":"Song","year":"2017","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Song et al., 2020] Jiaming Song, Chenlin Meng, and Ste- fano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.","title":"Denoising diffusion implicit models","authors":"Jiaming Song, Chenlin Meng, Stefano Ermon","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Szegedy et al., 2016] Christian Szegedy, Vincent Van- houcke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.","title":"Rethinking the inception architecture for computer vision","authors":"Szegedy","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yu et al., 2016] Qian Yu, Feng Liu, Yi-Zhe Song, Tao Xi- ang, Timothy M Hospedales, and Chen-Change Loy. Sketch me that shoe. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 799-807, 2016.","title":"Sketch me that shoe","authors":"Qian Yu, Feng Liu, Yi-Zhe Song, Tao Xiang, Timothy M Hospedales, Chen-Change Loy","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2023a] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image dif- fusion models. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 3836-3847, 2023.","title":"Adding conditional control to text-to-image diffusion models","authors":"Zhang","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2023b] Zhengming Zhang, Xiaoming Deng, Jinyao Li, Yukun Lai, Cuixia Ma, Yongjin Liu, and Hongan Wang. Stroke-based semantic segmentation for scene-level free-hand sketches. The Visual Computer, 39(12):6309-6321, 2023.","title":"Stroke-based semantic segmentation for scene-level free-hand sketches","authors":"Zhang","year":"2023","venue":"The Visual Computer","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhao et al., 2023a] Henghao Zhao, Kevin Qinghong Lin, Rui Yan, and Zechao Li. Diffusionvmr: Diffusion model for video moment retrieval. arXiv preprint arXiv:2308.15109, 2023.","title":"Diffusionvmr: Diffusion model for video moment retrieval","authors":"Zhao","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhao et al., 2023b] Liming Zhao, Kecheng Zheng, Yun Zheng, Deli Zhao, and Jingren Zhou. Rleg: vision- language representation learning with diffusion-based em- bedding generation. In International Conference on Ma- chine Learning, pages 42247-42258. PMLR, 2023.","title":"Rleg: visionlanguage representation learning with diffusion-based embedding generation","authors":"Zhao","year":"2023","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhao et al., 2023c] Yuzhong Zhao, Qixiang Ye, Weijia Wu, Chunhua Shen, and Fang Wan. Generative prompt model for weakly supervised object localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6351-6361, 2023.","title":"Generative prompt model for weakly supervised object localization","authors":"Zhao","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zou et al., 2018] Changqing Zou, Qian Yu, Ruofei Du, Haoran Mo, Yi-Zhe Song, Tao Xiang, Chengying Gao, Baoquan Chen, and Hao Zhang. Sketchyscene: Richly- annotated scene sketches. In Proceedings of the european conference on computer vision (ECCV), pages 421-436, 2018.","title":"Sketchyscene: Richlyannotated scene sketches","authors":"Changqing Zou, Qian Yu, Ruofei Du, Haoran Mo, Yi-Zhe Song, Tao Xiang, Chengying Gao, Baoquan Chen, Hao Zhang","year":"2018","venue":"Proceedings of the european conference on computer vision (ECCV)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}