{"type":"ScholarlyArticle","title":"Expressiveness is Effectiveness: Self-supervised Fashion-aware CLIP for Video-to-Shop Retrieval","authors":[{"name":"Likai Tian","slug":"likai-tian","given":"Likai","family":"Tian","affiliation":"National Engineering Research Center for Multimedia Software , Institute of Artificial Intelligence ,; Hubei Key Laboratory of Multimedia and Network Communication Engineering"},{"name":"Zhengwei Yang","slug":"zhengwei-yang","given":"Zhengwei","family":"Yang","affiliation":"National Engineering Research Center for Multimedia Software , Institute of Artificial Intelligence ,; Hubei Key Laboratory of Multimedia and Network Communication Engineering"},{"name":"Zechao Hu","slug":"zechao-hu","given":"Zechao","family":"Hu","affiliation":"National Engineering Research Center for Multimedia Software , Institute of Artificial Intelligence ,; Hubei Key Laboratory of Multimedia and Network Communication Engineering"},{"name":"Hao Li","slug":"hao-li","given":"Hao","family":"Li","affiliation":"National Engineering Research Center for Multimedia Software , Institute of Artificial Intelligence ,; Hubei Key Laboratory of Multimedia and Network Communication Engineering; School of Computer Science , Wuhan University , China"},{"name":"Yifang Yin","slug":"yifang-yin","given":"Yifang","family":"Yin","affiliation":"Institute for Infocomm Research , A*STAR , Singapore"},{"name":"Zheng Wang","slug":"zheng-wang","given":"Zheng","family":"Wang","affiliation":"National Engineering Research Center for Multimedia Software , Institute of Artificial Intelligence ,; Hubei Key Laboratory of Multimedia and Network Communication Engineering"}],"abstract":"The rise of online shopping and social media has spurred the Video-to-Shop Retrieval (VSR) task, which involves identifying fashion items (e.g., clothing) in videos and matching them with identical products provided by stores. In real-world scenarios, human movement in dynamic video scenes can cause substantial morphological alterations of fashion items with aspects of occlusion, shifting viewpoints (parallax), and partial visibility (truncation). This results in those high-quality frames being overwhelmed by a vast of redundant ones, which makes the retrieval less effectiveness. To this end, this paper introduces a framework, named Self-supervised Fashion-aware CLIP (SF-CLIP), for effective VSR. The SF-CLIP enables the discovery of salient frames with high fashion expressiveness via generating pseudo-labels from three key aspects of fashion expressiveness to assess occlusion, parallax, and truncation. With such pseudo-labels, the ability of CLIP is expanded to facilitate the discovery of salient frames. Furthermore, to encompass the comprehensive representations among salient frames, a dual-branch graph-based fusion module is proposed to extract and integrate inter-frame features. Extensive experiments demonstrate the superiority of SF-CLIP over the state-of-the-arts.","doi":"10.24963/ijcai.2024/148","url":"https://ijcai.org/proceedings/2024/148","pdf":"https://ijcai.org/proceedings/2024/0148.pdf","markdown":"https://ijcai.org/proceedings/2024/148.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/148","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":148,"pages":{"first":1335,"last":1343},"keywords":["Computer Vision: CV: Image and video retrieval","Computer Vision: CV: Interpretability and transparency"],"videos":[],"references":[{"position":1,"raw":"[Cheng et al., 2017] Zhi-Qi Cheng, Xiao Wu, Yang Liu, and Xian-Sheng Hua. Video2shop: Exact matching clothes in videos to online shopping images. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 4048-4056, 2017.","title":"Video2shop: Exact matching clothes in videos to online shopping images","authors":"Zhi-Qi Cheng, Xiao Wu, Yang Liu, Xian-Sheng Hua","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chia et al., 2022] Patrick John Chia, Giuseppe Attanasio, Federico Bianchi, Silvia Terragni, Ana Rita Magalhães, Diogo Goncalves, Ciro Greco, and Jacopo Tagliabue. Con- trastive language and vision learning of general fashion concepts. Scientific Reports, 12(1):18958, 2022.","title":"Contrastive language and vision learning of general fashion concepts","authors":"Patrick John, Chia, Giuseppe Attanasio, Federico Bianchi, Silvia Terragni, Ana Rita Magalhães, Diogo Goncalves, Ciro Greco, Jacopo Tagliabue","year":"2022","venue":"Scientific Reports","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Contributors, 2020] MMPose Contributors. Openmmlab pose estimation toolbox and benchmark. https://github. com/open-mmlab/mmpose, 2020.","title":"MMPose Contributors. Openmmlab pose estimation toolbox and benchmark","authors":null,"year":"2020","venue":null,"doi":null,"url":"https://github.com/open-mmlab/mmpose,2020","internal_url":null},{"position":4,"raw":"[Ge et al., 2019] Yuying Ge, Ruimao Zhang, Xiaogang Wang, Xiaoou Tang, and Ping Luo. Deepfashion2: A ver- satile benchmark for detection, pose estimation, segmenta- tion and re-identification of clothing images. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5337-5345, 2019.","title":"Deepfashion2: A versatile benchmark for detection, pose estimation, segmentation and re-identification of clothing images","authors":"Yuying Ge, Ruimao Zhang, Xiaogang Wang, Xiaoou Tang, Ping Luo","year":"2019","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Godi et al., 2022] Marco Godi, Christian Joppi, Geri Sk- enderi, and Marco Cristani. Movingfashion: a bench- mark for the video-to-shop challenge. In Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, pages 1678-1686, 2022.","title":"Movingfashion: a benchmark for the video-to-shop challenge","authors":"Marco Godi, Christian Joppi, Geri Skenderi, Marco Cristani","year":"2022","venue":"Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gorti et al., 2022] Satya Krishna Gorti, Noël Vouitsis, Jun- wei Ma, Keyvan Golestan, Maksims Volkovs, Animesh Garg, and Guangwei Yu. X-pool: Cross-modal language- video attention for text-video retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5006-5015, 2022.","title":"X-pool: Cross-modal languagevideo attention for text-video retrieval","authors":"Krishna Satya, Noël Gorti, Junwei Vouitsis, Keyvan Ma, Maksims Golestan, Animesh Volkovs, Guangwei Garg, Yu","year":"2022","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Goyal et al., 2023] Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19338- 19347, 2023.","title":"Finetune like you pretrain: Improved finetuning of zero-shot vision models","authors":"Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, Aditi Raghunathan","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Guo et al., 2023] Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzheng Ma, Xupeng Miao, Xuming He, and Bin Cui. Calip: Zero-shot enhancement of clip with parameter-free attention. In Proceedings of the AAAI Conference on Arti- ficial Intelligence, volume 37, pages 746-754, 2023.","title":"Calip: Zero-shot enhancement of clip with parameter-free attention","authors":"Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzheng Ma, Xupeng Miao, Xuming He, Bin Cui","year":"2023","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hegde et al., 2023] Deepti Hegde, Jeya Maria Jose Vala- narasu, and Vishal Patel. Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 2028-2038, 2023.","title":"Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition","authors":"Deepti Hegde, Maria Jose Jeya, Vishal Valanarasu, Patel","year":"2023","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ji et al., 2019] Zhong Ji, Kailin Xiong, Yanwei Pang, and Xuelong Li. Video summarization with attention-based encoder-decoder networks. IEEE Transactions on Cir- cuits and Systems for Video Technology, 30(6):1709-1717, 2019.","title":"Video summarization with attention-based encoder-decoder networks","authors":"Zhong Ji, Kailin Xiong, Yanwei Pang, Xuelong Li","year":"2019","venue":"IEEE Transactions on Circuits and Systems for Video Technology","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2022a] Chuyi Li, Lulu Li, Hongliang Jiang, Kai- heng Weng, Yifei Geng, Liang Li, Zaidan Ke, Qingyuan Li, Meng Cheng, Weiqiang Nie, et al. Yolov6: A single- stage object detection framework for industrial applica- tions. arXiv preprint arXiv:2209.02976, 2022.","title":"Yolov6: A singlestage object detection framework for industrial applications","authors":"Li","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2022b] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900, 2022.","title":"Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation","authors":"Li","year":"2022","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2019] Chih-Ting Liu, Chih-Wei Wu, Yu- Chiang Frank Wang, and Shao-Yi Chien. Spatially and temporally efficient non-local attention network for video-based person re-identification. arXiv preprint arXiv:1908.01683, 2019.","title":"Spatially and temporally efficient non-local attention network for video-based person re-identification","authors":"Chih-Ting Liu, Chih-Wei Wu, Yu-Chiang Frank, Wang, Shao-Yi Chien","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2022] Tianrui Liu, Qingjie Meng, Jun-Jie Huang, Athanasios Vlontzos, Daniel Rueckert, and Bernhard Kainz. Video summarization through reinforcement learn- ing with a 3d spatio-temporal u-net. IEEE Transactions on Image Processing, 31:1573-1586, 2022.","title":"Video summarization through reinforcement learning with a 3d spatio-temporal u-net","authors":"Tianrui Liu, Qingjie Meng, Jun-Jie Huang, Athanasios Vlontzos, Daniel Rueckert, Bernhard Kainz","year":"2022","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lu et al., 2022] Haoyu Lu, Mingyu Ding, Nanyi Fei, Yuqi Huo, and Zhiwu Lu. Lgdn: Language-guided denoising network for video-language modeling. Advances in Neural Information Processing Systems, 35:25198-25211, 2022.","title":"Lgdn: Language-guided denoising network for video-language modeling","authors":"Haoyu Lu, Mingyu Ding, Nanyi Fei, Yuqi Huo, Zhiwu Lu","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Narasimhan et al., 2021] Medhini Narasimhan, Anna Rohrbach, and Trevor Darrell. Clip-it! language-guided video summarization. Advances in Neural Information Processing Systems, 34:13988-14000, 2021.","title":"Clip-it! language-guided video summarization","authors":"Narasimhan","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Porrello et al., 2020] Angelo Porrello, Luca Bergamini, and Simone Calderara. Robust re-identification by multiple views knowledge distillation. In European Conference on Computer Vision, pages 93-110, 2020.","title":"Robust re-identification by multiple views knowledge distillation","authors":"Angelo Porrello, Luca Bergamini, Simone Calderara","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In International conference on machine learning, pages 8748-8763, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rasheed et al., 2023] Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Fine-tuned clip models are efficient video learners. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 6545-6554, 2023.","title":"Fine-tuned clip models are efficient video learners","authors":"Rasheed","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sanghi et al., 2022] Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Yi Cheng, Marco Fumero, and Kamal Rahimi Malekshan. Clip-forge: Towards zero-shot text-to-shape generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18603-18613, 2022.","title":"Clip-forge: Towards zero-shot text-to-shape generation","authors":"Aditya Sanghi, Hang Chu, Ye Joseph G Lambourne, Chin-Yi Wang, Marco Cheng, Kamal Fumero, Rahimi Malekshan","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Veličković et al., 2018] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In Interna- tional Conference on Learning Representations, 2018.","title":"Graph attention networks","authors":"Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, Yoshua Bengio","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2023] Jinpeng Wang, Pan Zhou, Mike Zheng Shou, and Shuicheng Yan. Position-guided text prompt for vision-language pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23242-23251, 2023.","title":"Position-guided text prompt for vision-language pre-training","authors":"Jinpeng Wang, Pan Zhou, Mike Zheng Shou, Shuicheng Yan","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2021] Dehao Wu, Yi Li, Yinghong Zhang, and Yuesheng Zhu. Multi-dimensional attentive hierarchical graph pooling network for video-text retrieval. In 2021 IEEE International Conference on Multimedia and Expo, pages 1-6, 2021.","title":"Multi-dimensional attentive hierarchical graph pooling network for video-text retrieval","authors":"Dehao Wu, Yi Li, Yinghong Zhang, Yuesheng Zhu","year":"2021","venue":"2021 IEEE International Conference on Multimedia and Expo","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yan et al., 2020] Yichao Yan, Jie Qin, Jiaxin Chen, Li Liu, Fan Zhu, Ying Tai, and Ling Shao. Learning multi-granular hypergraphs for video-based person re- identification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2899- 2908, 2020.","title":"Learning multi-granular hypergraphs for video-based person reidentification","authors":"Yichao Yan, Jie Qin, Jiaxin Chen, Li Liu, Fan Zhu, Ying Tai, Ling Shao","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2023] Zhengwei Yang, Meng Lin, Xian Zhong, Yu Wu, and Zheng Wang. Good is bad: Causality inspired cloth-debiasing for cloth-changing person re- identification. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 1472-1481, 2023.","title":"Good is bad: Causality inspired cloth-debiasing for cloth-changing person reidentification","authors":"Zhengwei Yang, Meng Lin, Xian Zhong, Yu Wu, Zheng Wang","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhou et al., 2023] Ziqin Zhou, Yinjie Lei, Bowen Zhang, Lingqiao Liu, and Yifan Liu. Zegclip: Towards adapting clip for zero-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 11175-11185, 2023.","title":"Zegclip: Towards adapting clip for zero-shot semantic segmentation","authors":"Zhou","year":"2023","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}