{"type":"ScholarlyArticle","title":"Contrastive Transformer Cross-Modal Hashing for Video-Text Retrieval","authors":[{"name":"Xiaobo Shen","slug":"xiaobo-shen","given":"Xiaobo","family":"Shen","affiliation":"Nanjing University of Science and Technology"},{"name":"Qianxin Huang","slug":"qianxin-huang","given":"Qianxin","family":"Huang","affiliation":"Nanjing University of Science and Technology"},{"name":"Long Lan","slug":"long-lan","given":"Long","family":"Lan","affiliation":"National University of Defense Technology"},{"name":"Yuhui Zheng","slug":"yuhui-zheng","given":"Yuhui","family":"Zheng","affiliation":"Qinghai Normal University"}],"abstract":"As video-based social networks continue to grow exponentially, there is a rising interest in video retrieval using natural language. Cross-modal hashing, which learns compact hash code for encoding multi-modal data, has proven to be widely effective in large-scale cross-modal retrieval, e.g., image-text retrieval, primarily due to its computation and storage efficiency. However, when applied to video-text retrieval, existing cross-modal hashing methods generally  extract features at the frame- or word-level for videos and texts individually, thereby ignoring their long-term dependencies. To address this issue, we propose Contrastive Transformer Cross-Modal Hashing (CTCH), a novel approach designed for video-text retrieval task. CTCH employs bidirectional transformer encoder to encode video and text and leverages their long-term dependencies. CTCH further introduces supervised multi-modality contrastive loss that effectively exploits inter-modality and intra-modality similarities among videos and texts. The experimental results on three video benchmark datasets demonstrate that CTCH outperforms the state-of-the-arts in video-text retrieval tasks.","doi":"10.24963/ijcai.2024/136","url":"https://ijcai.org/proceedings/2024/136","pdf":"https://ijcai.org/proceedings/2024/0136.pdf","markdown":"https://ijcai.org/proceedings/2024/136.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/136","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":136,"pages":{"first":1227,"last":1235},"keywords":["Machine Learning: ML: Multi-view learning","Computer Vision: CV: Image and video retrieval","Machine Learning: ML: Multi-modal learning"],"videos":[],"references":[{"position":1,"raw":"[Carreira and Zisserman, 2017] João Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. In CVPR, pages 4724-4733, 2017.","title":"Quo vadis, action recognition? A new model and the kinetics dataset","authors":"João Carreira, Andrew Zisserman","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2020] Ting Chen, Simon Kornblith, Moham- mad Norouzi, and Geoffrey E. Hinton. A simple frame- work for contrastive learning of visual representations. In ICML, volume 119, pages 1597-1607, 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E Hinton","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2021] Yanbei Chen, Yongqin Xian, A. Sophia Koepke, Ying Shan, and Zeynep Akata. Distilling audio- visual knowledge by compositional contrastive learning. In CVPR, pages 7016-7025, 2021.","title":"Distilling audiovisual knowledge by compositional contrastive learning","authors":"Chen","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understand- ing. In NAACL, pages 4171-4186, 2019.","title":"BERT: pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jiang and Li, 2017] Qing-Yuan Jiang and Wu-Jun Li. Deep cross-modal hashing. In CVPR, pages 3270-3278, 2017.","title":"Deep cross-modal hashing","authors":"Li ; Qing-Yuan Jiang, Wu-Jun Jiang, Li","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jin et al., 2023] Lu Jin, Zechao Li, and Jinhui Tang. Deep semantic multimodal hashing network for scalable image- text and video-text retrievals. IEEE TNNLS, 34(4):1838- 1851, 2023.","title":"Deep semantic multimodal hashing network for scalable imagetext and video-text retrievals","authors":"Lu Jin, Zechao Li, Jinhui Tang","year":"2023","venue":"IEEE TNNLS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Khosla et al., 2020] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised con- trastive learning. NeurIPS, 33:18661-18673, 2020.","title":"Supervised contrastive learning","authors":"Khosla","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Krishna et al., 2017] Ranjay Krishna, Kenji Hata, Fred- eric Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense- captioning events in videos. In ICCV, pages 706-715, 2017.","title":"Densecaptioning events in videos","authors":"Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, Juan Carlos Niebles","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2018] Chao Li, Cheng Deng, Ning Li, Wei Liu, Xinbo Gao, and Dacheng Tao. Self-supervised adversar- ial hashing networks for cross-modal retrieval. In CVPR, pages 4242-4251, 2018.","title":"Self-supervised adversarial hashing networks for cross-modal retrieval","authors":"Chao Li, Cheng Deng, Ning Li, Wei Liu, Xinbo Gao, Dacheng Tao","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2019] Shuyan Li, Zhixiang Chen, Jiwen Lu, Xiu Li, and Jie Zhou. Neighborhood preserving hashing for scalable video retrieval. In ICCV, pages 8211-8220, 2019.","title":"Neighborhood preserving hashing for scalable video retrieval","authors":"Shuyan Li, Zhixiang Chen, Jiwen Lu, Xiu Li, Jie Zhou","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2020] Zechao Li, Jinhui Tang, Liyan Zhang, and Jian Yang. Weakly-supervised semantic guided hashing for social image retrieval. IJCV, 128(8):2265-2278, 2020.","title":"Weakly-supervised semantic guided hashing for social image retrieval","authors":"Zechao Li, Jinhui Tang, Liyan Zhang, Jian Yang","year":"2020","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2021] Shuyan Li, Xiu Li, Jiwen Lu, and Jie Zhou. Self-supervised video hashing via bidirectional transform- ers. In CVPR, pages 13549-13558, 2021.","title":"Self-supervised video hashing via bidirectional transformers","authors":"Shuyan Li, Xiu Li, Jiwen Lu, Jie Zhou","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2022] Shuyan Li, Xiu Li, Jiwen Lu, and Jie Zhou. Structure-adaptive neighborhood preserving hashing for scalable video search. IEEE TCSVT, 32(4):2441-2454, 2022.","title":"Structure-adaptive neighborhood preserving hashing for scalable video search","authors":"Shuyan Li, Xiu Li, Jiwen Lu, Jie Zhou","year":"2022","venue":"IEEE TCSVT","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2015] Zijia Lin, Guiguang Ding, Mingqing Hu, and Jianmin Wang. Semantics-preserving hashing for cross-view retrieval. In CVPR, pages 3864-3872, 2015.","title":"Semantics-preserving hashing for cross-view retrieval","authors":"Lin","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lipton, 2015] Zachary Chase Lipton. A critical review of recurrent neural networks for sequence learning. CoRR, abs/1506.00019, 2015.","title":"A critical review of recurrent neural networks for sequence learning","authors":"Zachary Chase, Lipton","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Over et al., 2010] Paul Over, George Awad, Jon Fiscus, Brian Antonishek, Martial Michel, Wessel Kraaij, Alan F. Smeaton, and Georges Quénot. TRECVID 2010 -an overview of the goals, tasks, data, evaluation mechanisms and metrics. In TRECVID 2010 workshop participants notebook papers, 2010.","title":"TRECVID 2010 -an overview of the goals, tasks, data, evaluation mechanisms and metrics","authors":"Paul Over, George Awad, Jon Fiscus, Brian Antonishek, Martial Michel, Wessel Kraaij, Alan F Smeaton, Georges Quénot","year":"2010","venue":"TRECVID 2010 workshop participants notebook papers","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pascanu et al., 2013] Razvan Pascanu, Tomás Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In ICML, volume 28, pages 1310-1318, 2013.","title":"On the difficulty of training recurrent neural networks","authors":"Razvan Pascanu, Tomás Mikolov, Yoshua Bengio","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Qi et al., 2021] Mengshi Qi, Jie Qin, Yi Yang, Yunhong Wang, and Jiebo Luo. Semantics-aware spatial-temporal binaries for cross-modal video retrieval. IEEE TIP, 30:2989-3004, 2021.","title":"Semantics-aware spatial-temporal binaries for cross-modal video retrieval","authors":"Mengshi Qi, Jie Qin, Yi Yang, Yunhong Wang, Jiebo Luo","year":"2021","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Qian et al., 2021] Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang, Serge J. Belongie, and Yin Cui. Spatiotemporal contrastive video represen- tation learning. In CVPR, pages 6964-6974, 2021.","title":"Spatiotemporal contrastive video representation learning","authors":"Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang, Serge J Belongie, Yin Cui","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S Bernstein, Alexander C Berg, Li Fei-Fei","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sigurdsson et al., 2016] Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collec- tion for activity understanding. In ECCV, volume 9905, pages 510-526, 2016.","title":"Hollywood in homes: Crowdsourcing data collection for activity understanding","authors":"A Gunnar, Gül Sigurdsson, Xiaolong Varol, Ali Wang, Ivan Farhadi, Abhinav Laptev, Gupta","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Song et al., 2018] Jingkuan Song, Hanwang Zhang, Xiang- peng Li, Lianli Gao, Meng Wang, and Richang Hong. Self-supervised video hashing with hierarchical binary auto-encoder. IEEE TIP, 27(7):3210-3221, 2018.","title":"Self-supervised video hashing with hierarchical binary auto-encoder","authors":"Song","year":"2018","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sun et al., 2019] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In ICCV, pages 7463-7472, 2019.","title":"Videobert: A joint model for video and language representation learning","authors":"Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, Cordelia Schmid","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wei and Zou, 2019] Jason W. Wei and Kai Zou. EDA: easy data augmentation techniques for boosting performance on text classification tasks. In EMNLP, pages 6381-6387, 2019.","title":"EDA: easy data augmentation techniques for boosting performance on text classification tasks","authors":"Jason W Wei, Kai Zou","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Weiss et al., 2008] Yair Weiss, Antonio Torralba, and Robert Fergus. Spectral hashing. In NeurIPS, pages 1753- 1760, 2008.","title":"Spectral hashing","authors":"Yair Weiss, Antonio Torralba, Robert Fergus","year":"2008","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wu et al., 2019] Gengshen Wu, Jungong Han, Yuchen Guo, Li Liu, Guiguang Ding, Qiang Ni, and Ling Shao. Unsu- pervised deep video hashing via balanced code for large- scale video retrieval. IEEE TIP, 28(4):1993-2007, 2019.","title":"Unsupervised deep video hashing via balanced code for largescale video retrieval","authors":"Wu","year":"2019","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xu et al., 2016] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR-VTT: A large video description dataset for bridging video and language. In CVPR, pages 5288-5296, 2016.","title":"MSR-VTT: A large video description dataset for bridging video and language","authors":"Jun Xu, Tao Mei, Ting Yao, Yong Rui","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yang et al., 2018] Huei-Fang Yang, Kevin Lin, and Chu- Song Chen. Supervised learning of semantics-preserving hash via deep convolutional neural networks. IEEE TPAMI, 40(2):437-451, 2018.","title":"Supervised learning of semantics-preserving hash via deep convolutional neural networks","authors":"Huei-Fang Yang, Kevin Lin, Chu-Song Chen","year":"2018","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yu et al., 2022] En Yu, Jianhua Ma, Jiande Sun, Xiaojun Chang, Huaxiang Zhang, and Alexander G. Hauptmann. Deep discrete cross-modal hashing with multiple supervi- sion. Neurocomputing, 486:215-224, 2022.","title":"Deep discrete cross-modal hashing with multiple supervision","authors":"En Yu, Jianhua Ma, Jiande Sun, Xiaojun Chang, Huaxiang Zhang, Alexander G Hauptmann","year":"2022","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang and Li, 2014] Dongqing Zhang and Wu-Jun Li. Large-scale supervised multimodal hashing with seman- tic correlation maximization. In AAAI, pages 2177-2183, 2014.","title":"Large-scale supervised multimodal hashing with semantic correlation maximization","authors":"Li ; Zhang, Dongqing Zhang, Wu-Jun Li","year":"2014","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2016] Hanwang Zhang, Meng Wang, Richang Hong, and Tat-Seng Chua. Play and rewind: Optimizing binary representations of videos by self-supervised tempo- ral hashing. In MM, pages 781-790, 2016.","title":"Play and rewind: Optimizing binary representations of videos by self-supervised temporal hashing","authors":"Hanwang Zhang, Meng Wang, Richang Hong, Tat-Seng Chua","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}