{"type":"ScholarlyArticle","title":"Dig into Multi-modal Cues for Video Retrieval with Hierarchical Alignment","authors":[{"name":"Wenzhe Wang","slug":"wenzhe-wang","given":"Wenzhe","family":"Wang","affiliation":""},{"name":"Mengdan Zhang","slug":"mengdan-zhang","given":"Mengdan","family":"Zhang","affiliation":""},{"name":"Runnan Chen","slug":"runnan-chen","given":"Runnan","family":"Chen","affiliation":""},{"name":"Guanyu Cai","slug":"guanyu-cai","given":"Guanyu","family":"Cai","affiliation":""},{"name":"Penghao Zhou","slug":"penghao-zhou","given":"Penghao","family":"Zhou","affiliation":""},{"name":"Pai Peng","slug":"pai-peng","given":"Pai","family":"Peng","affiliation":""},{"name":"Xiaowei Guo","slug":"xiaowei-guo","given":"Xiaowei","family":"Guo","affiliation":""},{"name":"Jian Wu","slug":"jian-wu","given":"Jian","family":"Wu","affiliation":""},{"name":"Xing Sun","slug":"xing-sun","given":"Xing","family":"Sun","affiliation":""}],"abstract":"Multi-modal cues presented in videos are usually beneficial for the challenging video-text retrieval task on internet-scale datasets. Recent video retrieval methods take advantage of multi-modal cues by aggregating them to holistic high-level semantics for matching with text representations in a global view. In contrast to this global alignment, the local alignment of detailed semantics encoded within both multi-modal cues and distinct phrases is still not well conducted. Thus, in this paper, we leverage the hierarchical video-text alignment to fully explore the detailed diverse characteristics in multi-modal cues for fine-grained alignment with local semantics from phrases, as well as to capture a high-level semantic correspondence. Specifically, multi-step attention is learned for progressively comprehensive local alignment and a holistic transformer is utilized to summarize multi-modal cues for global alignment. With hierarchical alignment, our model outperforms state-of-the-art methods on three public video retrieval datasets.","doi":"10.24963/ijcai.2021/154","url":"https://ijcai.org/proceedings/2021/154","pdf":"https://ijcai.org/proceedings/2021/0154.pdf","markdown":"https://ijcai.org/proceedings/2021/154.md","bibtex":"https://ijcai.org/proceedings/2021/bibtex/154","volume":{"slug":"2021","title":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2021","url":"https://ijcai.org/proceedings/2021/","year":2021,"location":"Montreal","dates":"19-27 August 2021","editors":"Prof. Zhi-Hua Zhou","isbn":"978-0-9992411-9-6","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2021"},"section":"Computer Vision II","track":"Main Track","number":154,"pages":{"first":1113,"last":1121},"keywords":["Computer Vision: Language and Vision","Machine Learning: Deep Learning"],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}