{"type":"ScholarlyArticle","title":"Exploiting Visual Semantic Reasoning for Video-Text Retrieval","authors":[{"name":"Zerun Feng","slug":"zerun-feng","given":"Zerun","family":"Feng","affiliation":"Beijing Key Laboratory of Network System Architecture and Convergence , School of Information and Communication Engineering , Beijing University of Posts and Telecommunications , Beijing , China"},{"name":"Zhimin Zeng","slug":"zhimin-zeng","given":"Zhimin","family":"Zeng","affiliation":"Beijing Key Laboratory of Network System Architecture and Convergence , School of Information and Communication Engineering , Beijing University of Posts and Telecommunications , Beijing , China; Beijing Laboratory of Advanced Information Networks , Beijing , China"},{"name":"Caili Guo","slug":"caili-guo","given":"Caili","family":"Guo","affiliation":"Beijing Key Laboratory of Network System Architecture and Convergence , School of Information and Communication Engineering , Beijing University of Posts and Telecommunications , Beijing , China; Beijing Laboratory of Advanced Information Networks , Beijing , China"},{"name":"Zheng Li","slug":"zheng-li","given":"Zheng","family":"Li","affiliation":"Beijing Key Laboratory of Network System Architecture and Convergence , School of Information and Communication Engineering , Beijing University of Posts and Telecommunications , Beijing , China"}],"abstract":"Video retrieval is a challenging research topic bridging the vision and language areas and has attracted broad attention in recent years. Previous works have been devoted to representing videos by directly encoding from frame-level features. In fact, videos consist of various and abundant semantic relations to which existing methods pay less attention. To address this issue, we propose a Visual Semantic Enhanced Reasoning Network (ViSERN) to exploit reasoning between frame regions. Specifically, we consider frame regions as vertices and construct a fully-connected semantic correlation graph. Then, we perform reasoning by novel random walk rule-based graph convolutional networks to generate region features involved with semantic relations. With the benefit of reasoning, semantic interactions between regions are considered, while the impact of redundancy is suppressed. Finally, the region features are aggregated to form frame-level features for further encoding to measure video-text similarity. Extensive experiments on two public benchmark datasets validate the effectiveness of our method by achieving state-of-the-art performance due to the powerful semantic reasoning.","doi":"10.24963/ijcai.2020/140","url":"https://ijcai.org/proceedings/2020/140","pdf":"https://ijcai.org/proceedings/2020/0140.pdf","markdown":"https://ijcai.org/proceedings/2020/140.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/140","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":140,"pages":{"first":1005,"last":1011},"keywords":["Computer Vision: Language and Vision","Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24173/file_24173.m3u8","length_sec":282,"provider":"hls"}],"references":[{"position":1,"raw":"[Anderson et al., 2018] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, and Stephen Gould Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6077-6086, 2018.","title":"Bottom-up and top-down attention for image captioning and visual question answering","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bruna et al., 2014] Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann Lecun. Spectral networks and locally connected networks on graphs. In International Con- ference on Learning Representations (ICLR2014), CBLS, April 2014, 2014.","title":"Spectral networks and locally connected networks on graphs","authors":"Joan Bruna, Wojciech Zaremba, Arthur Szlam, Yann Lecun","year":"2014","venue":"International Conference on Learning Representations (ICLR2014)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen and Dolan, 2011] David L Chen and William B Dolan. Collecting highly parallel data for paraphrase eval- uation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Lan- guage Technologies-Volume 1, pages 190-200. Associa- tion for Computational Linguistics, 2011.","title":"Collecting highly parallel data for paraphrase evaluation","authors":"L David, William B Chen, Dolan","year":"2011","venue":"Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Defferrard et al., 2016] Michaël Defferrard, Xavier Bres- son, and Pierre Vandergheynst. Convolutional neural net- works on graphs with fast localized spectral filtering. In Advances in neural information processing systems, pages 3844-3852, 2016.","title":"Convolutional neural networks on graphs with fast localized spectral filtering","authors":"Michaël Defferrard, Xavier Bresson, Pierre Vandergheynst","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dong et al., 2018] Jianfeng Dong, Xirong Li, and Cees GM Snoek. Predicting visual features from text for image and video caption retrieval. TMM, 20(12):3377-3388, 2018.","title":"Predicting visual features from text for image and video caption retrieval","authors":"Jianfeng Dong, Xirong Li, G M Cees, Snoek","year":"2018","venue":"TMM","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dong et al., 2019] Jianfeng Dong, Xirong Li, Chaoxi Xu, Shouling Ji, Yuan He, Gang Yang, and Xun Wang. Dual encoding for zero-example video retrieval. In CVPR, pages 9346-9355, 2019.","title":"Dual encoding for zero-example video retrieval","authors":"Jianfeng Dong, Xirong Li, Chaoxi Xu, Shouling Ji, Yuan He, Gang Yang, Xun Wang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Faghri et al., 2018] Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Im- proving visual-semantic embeddings with hard negatives. BMVC, 2018.","title":"Vse++: Improving visual-semantic embeddings with hard negatives","authors":"Faghri","year":"2018","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Grover and Leskovec, 2016] Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pages 855-864. ACM, 2016.","title":"node2vec: Scalable feature learning for networks","authors":"Leskovec Grover, Aditya Grover, Jure Leskovec","year":"2016","venue":"Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hammond et al., 2011] David K Hammond, Pierre Van- dergheynst, and Rémi Gribonval. Wavelets on graphs via spectral graph theory. Applied and Computational Har- monic Analysis, 30(2):129-150, 2011.","title":"Wavelets on graphs via spectral graph theory","authors":"Pierre David K Hammond, Rémi Vandergheynst, Gribonval","year":"2011","venue":"Applied and Computational Harmonic Analysis","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Katsuki and Constantinidis, 2014] Fumi Katsuki and Chris- tos Constantinidis. Bottom-up and top-down attention: different processes and overlapping neural systems. The Neuroscientist, 20(5):509-521, 2014.","title":"Bottom-up and top-down attention: different processes and overlapping neural systems","authors":"Constantinidis Katsuki","year":"2014","venue":"The Neuroscientist","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kipf and Welling, 2017] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. 2017. [Kiros et al., 2014]","title":"Semi-supervised classification with graph convolutional networks","authors":"Thomas N Kipf, Max Welling","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539, 2014.","title":"Unifying visual-semantic embeddings with multimodal neural language models","authors":"Ryan Kiros, Ruslan Salakhutdinov, Richard S Zemel","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krishna et al., 2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Inter- national Journal of Computer Vision, 123(1):32-73, 2017.","title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","authors":"Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma","year":"2017","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lee et al., 2018] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In Proceedings of the European Conference on Computer Vision (ECCV), pages 201-216, 2018.","title":"Stacked cross attention for image-text matching","authors":"Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, Xiaodong He","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2019] Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu. Visual semantic reasoning for image-text matching. In Proceedings of the IEEE International Conference on Computer Vision, pages 4654-4662, 2019.","title":"Visual semantic reasoning for image-text matching","authors":"Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, Yun Fu","year":"2019","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2019] Xinchen Liu, Wu Liu, Meng Zhang, Jing- wen Chen, Lianli Gao, Chenggang Yan, and Tao Mei. Social relation recognition from videos via multi-scale spatial-temporal reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3566-3574, 2019.","title":"Social relation recognition from videos via multi-scale spatial-temporal reasoning","authors":"Xinchen Liu, Wu Liu, Meng Zhang, Jingwen Chen, Lianli Gao, Chenggang Yan, Tao Mei","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mithun et al., 2018] Niluthpol Chowdhury Mithun, Juncheng Li, Florian Metze, and Amit K Roy-Chowdhury. Learning joint embedding with multimodal cues for cross-modal video-text retrieval. In ICMR, pages 19-27, 2018.","title":"Learning joint embedding with multimodal cues for cross-modal video-text retrieval","authors":"Mithun","year":"2018","venue":"ICMR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Perozzi et al., 2014] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social repre- sentations. In Proceedings of the 20th ACM SIGKDD in- ternational conference on Knowledge discovery and data mining, pages 701-710. ACM, 2014.","title":"Deepwalk: Online learning of social representations","authors":"Bryan Perozzi, Rami Al-Rfou, Steven Skiena","year":"2014","venue":"Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time ob- ject detection with region proposal networks. In Advances in neural information processing systems, pages 91-99, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing sys- tems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang and Gupta, 2018] Xiaolong Wang and Abhinav Gupta. Videos as space-time region graphs. In Proceed- ings of the European Conference on Computer Vision (ECCV), pages 399-417, 2018.","title":"Videos as space-time region graphs","authors":"Wang, Gupta, Xiaolong Wang, Abhinav Gupta","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu et al., 2016] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE confer- ence on computer vision and pattern recognition, pages 5288-5296, 2016.","title":"Msr-vtt: A large video description dataset for bridging video and language","authors":"Jun Xu, Tao Mei, Ting Yao, Yong Rui","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}