{"type":"ScholarlyArticle","title":"Object-Aware Multi-Branch Relation Networks for Spatio-Temporal Video Grounding","authors":[{"name":"Zhu Zhang","slug":"zhu-zhang","given":"Zhu","family":"Zhang","affiliation":"College of Computer Science , Zhejiang University , China"},{"name":"Zhou Zhao","slug":"zhou-zhao","given":"Zhou","family":"Zhao","affiliation":"College of Computer Science , Zhejiang University , China"},{"name":"Zhijie Lin","slug":"zhijie-lin","given":"Zhijie","family":"Lin","affiliation":"College of Computer Science , Zhejiang University , China"},{"name":"Baoxing Huai","slug":"baoxing-huai","given":"Baoxing","family":"Huai","affiliation":"Huawei Cloud & AI , China"},{"name":"Jing Yuan","slug":"jing-yuan","given":"Jing","family":"Yuan","affiliation":"Huawei Cloud & AI , China"}],"abstract":"Spatio-temporal video grounding aims to retrieve the spatio-temporal tube of a queried object according to the given sentence. Currently, most existing grounding methods are restricted to well-aligned segment-sentence pairs. In this paper, we explore spatio-temporal video grounding on unaligned data and multi-form sentences. This challenging task requires to capture critical object relations to identify the queried target. However, existing approaches cannot distinguish notable objects and remain in ineffective relation modeling between unnecessary objects. Thus, we propose a novel object-aware multi-branch relation network for object-aware relation discovery. Concretely, we first devise multiple branches to develop object-aware region modeling, where each branch focuses on a crucial object mentioned in the sentence. We then propose multi-branch relation reasoning to capture critical object relationships between the main branch and auxiliary branches. Moreover, we apply a diversity loss to make each branch only pay attention to its corresponding object and boost multi-branch learning. The extensive experiments show the effectiveness of our proposed method.","doi":"10.24963/ijcai.2020/149","url":"https://ijcai.org/proceedings/2020/149","pdf":"https://ijcai.org/proceedings/2020/0149.pdf","markdown":"https://ijcai.org/proceedings/2020/149.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/149","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":149,"pages":{"first":1069,"last":1075},"keywords":["Computer Vision: Language and Vision"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25052/file_25052.m3u8","length_sec":298,"provider":"hls"}],"references":[{"position":1,"raw":"[Chen et al., 2019a] Jingyuan Chen, Lin Ma, Xinpeng Chen, Zequn Jie, and Jiebo Luo. Localizing natural language in videos. In AAAI, 2019.","title":"Localizing natural language in videos","authors":"Chen","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2019b] Zhenfang Chen, Lin Ma, Wenhan Luo, and Kwan-Yee K Wong. Weakly-supervised spatio- temporally grounding natural sentence in video. 2019.","title":"Weakly-supervised spatiotemporally grounding natural sentence in video","authors":"Chen","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2018] Chaorui Deng, Qi Wu, Qingyao Wu, Fuyuan Hu, Fan Lyu, and Mingkui Tan. Visual grounding via accumulated attention. In CVPR, pages 7746-7755, 2018.","title":"Visual grounding via accumulated attention","authors":"Chaorui Deng, Qi Wu, Qingyao Wu, Fuyuan Hu, Fan Lyu, Mingkui Tan","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gao et al., 2017] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. TALL: temporal activity localization via language query. In ICCV, pages 5277-5285, 2017.","title":"TALL: temporal activity localization via language query","authors":"Jiyang Gao, Chen Sun, Zhenheng Yang, Ram Nevatia","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hendricks et al., 2017] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural lan- guage. In ICCV, pages 5803-5812, 2017.","title":"Localizing moments in video with natural language","authors":"Anne Lisa, Oliver Hendricks, Eli Wang, Josef Shechtman, Trevor Sivic, Bryan Darrell, Russell","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hu et al., 2016] Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. Natural language object retrieval. In CVPR, pages 4555-4564, 2016.","title":"Natural language object retrieval","authors":"Hu","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2017] Ronghang Hu, Marcus Rohrbach, Jacob Andreas, Trevor Darrell, and Kate Saenko. Modeling re- lationships in referential expressions with compositional modular networks. In CVPR, pages 1115-1124, 2017.","title":"Modeling relationships in referential expressions with compositional modular networks","authors":"Hu","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2018] De-An Huang, Shyamal Buch, Lucio Dery, Animesh Garg, Li Fei-Fei, and Juan Carlos Niebles. Finding \"it\": Weakly-supervised reference-aware visual grounding in instructional videos. In CVPR, June 2018.","title":"Finding \"it\": Weakly-supervised reference-aware visual grounding in instructional videos","authors":"De-An, Shyamal Huang, Lucio Buch, Animesh Dery, Li Garg, Juan Carlos Fei-Fei, Niebles","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lin et al., 2020a] Zhijie Lin, Zhou Zhao, Zhu Zhang, Qi Wang, and Huasheng Liu. Weakly-supervised video moment retrieval via semantic completion network. In AAAI, 2020.","title":"Weakly-supervised video moment retrieval via semantic completion network","authors":"Lin","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin et al., 2020b] Zhijie Lin, Zhou Zhao, Zhu Zhang, Zijian Zhang, and Deng Cai. Moment retrieval via cross-modal interaction networks with query reconstruction. TIP, 29:3750-3762, 2020.","title":"Moment retrieval via cross-modal interaction networks with query reconstruction","authors":"Lin","year":"2020","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mithun et al., 2019] Niluthpol Chowdhury Mithun, Sujoy Paul, and Amit K Roy-Chowdhury. Weakly supervised video moment retrieval from text queries. In CVPR, pages 11592-11601, 2019.","title":"Weakly supervised video moment retrieval from text queries","authors":"Mithun","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Nagaraja et al., 2016] Varun K Nagaraja, Vlad I Morariu, and Larry S Davis. Modeling context between objects for referring expression understanding. In ECCV, pages 792- 807. Springer, 2016.","title":"Modeling context between objects for referring expression understanding","authors":"Vlad I Varun K Nagaraja, Larry S Morariu, Davis","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, pages 91-99, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Rohrbach et al., 2016] Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. Ground- ing of textual phrases in images by reconstruction. In ECCV, pages 817-834. Springer, 2016. Shang et al., 2019] Xindi Shang, Donglin Di, Junbin Xiao, Yu Cao, Xun Yang, and Tat-Seng Chua. Annotating ob- jects and relations in user-generated videos. In ICMR, pages 279-287. ACM, 2019.","title":"Annotating objects and relations in user-generated videos","authors":"Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, Bernt Schiele, ; Shang","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"Shi et al., ] Jing Shi, Jia Xu, Boqing Gong, and Chenliang Xu. Not all frames are equal: Weakly-supervised video grounding with contextual similarity and visual clustering losses. In CVPR.","title":"Not all frames are equal: Weakly-supervised video grounding with contextual similarity and visual clustering losses","authors":"Shi","year":null,"venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Yamaguchi et al., 2017] Masataka Yamaguchi, Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada. Spatio- temporal person retrieval via natural language queries. In ICCV, pages 1453-1462, 2017.","title":"Spatiotemporal person retrieval via natural language queries","authors":"Masataka Yamaguchi, Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Yang et al., 2019a] Sibei Yang, Guanbin Li, and Yizhou Yu. Cross-modal relationship inference for grounding refer- ring expressions. In CVPR, pages 4145-4154, 2019.","title":"Cross-modal relationship inference for grounding referring expressions","authors":"Yang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Yang et al., 2019b] Sibei Yang, Guanbin Li, and Yizhou Yu. Dynamic graph attention for referring expression compre- hension. In ICCV, pages 4644-4653, 2019.","title":"Dynamic graph attention for referring expression comprehension","authors":"Yang","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Yu et al., 2018] Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. Mat- tnet: Modular attention network for referring expression comprehension. In CVPR, pages 1307-1315, 2018.","title":"Mattnet: Modular attention network for referring expression comprehension","authors":"Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, Tamara L Berg","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yuan et al., 2019] Yitian Yuan, Lin Ma, Jingwen Wang, Wei Liu, and Wenwu Zhu. Semantic conditioned dynamic modulation for temporal sentence grounding in videos. In NIPS, pages 534-544, 2019.","title":"Semantic conditioned dynamic modulation for temporal sentence grounding in videos","authors":"Yitian Yuan, Lin Ma, Jingwen Wang, Wei Liu, Wenwu Zhu","year":"2019","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang et al., 2019a] Da Zhang, Xiyang Dai, Xin Wang, Yuan-Fang Wang, and Larry S Davis. Man: Moment align- ment network for natural language moment retrieval via iterative graph adjustment. In CVPR, pages 1247-1257, 2019.","title":"Man: Moment alignment network for natural language moment retrieval via iterative graph adjustment","authors":"Zhang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2019b] Songyang Zhang, Houwen Peng, Jian- long Fu, and Jiebo Luo. Learning 2d temporal adjacent networks for moment localization with natural language. arXiv preprint arXiv:1912.03590, 2019.","title":"Learning 2d temporal adjacent networks for moment localization with natural language","authors":"Zhang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2019c] Zhu Zhang, Zhijie Lin, Zhou Zhao, and Zhenxin Xiao. Cross-modal interaction networks for query-based moment retrieval in videos. In SIGIR, 2019.","title":"Cross-modal interaction networks for query-based moment retrieval in videos","authors":"Zhang","year":"2019","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2019d] Zhu Zhang, Zhou Zhao, Zhijie Lin, Jingkuan Song, and Deng Cai. Localizing unseen activ- ities in video via image query. In IJCAI, 2019.","title":"Localizing unseen activities in video via image query","authors":"Zhang","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/610"},{"position":25,"raw":"[Zhang et al., 2020] Zhu Zhang, Zhou Zhao, Yang Zhao, Qi Wang, Huasheng Liu, and Lianli Gao. Where does it exist: Spatio-temporal video grounding for multi-form sentences. 2020.","title":"Where does it exist: Spatio-temporal video grounding for multi-form sentences","authors":"Zhu Zhang, Zhou Zhao, Yang Zhao, Qi Wang, Huasheng Liu, Lianli Gao","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhou et al., 2018] Luowei Zhou, Nathan Louis, and Jason J Corso. Weakly-supervised video object grounding from text by loss weighting and object interaction. arXiv preprint arXiv:1805.02834, 2018.","title":"Weakly-supervised video object grounding from text by loss weighting and object interaction","authors":"Zhou","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhuang et al., 2018] Bohan Zhuang, Qi Wu, Chunhua Shen, Ian Reid, and Anton van den Hengel. Parallel attention: A unified framework for visual object discovery through dialogs and queries. In CVPR, pages 4252-4261, 2018.","title":"Parallel attention: A unified framework for visual object discovery through dialogs and queries","authors":"Bohan Zhuang, Qi Wu, Chunhua Shen, Ian Reid, Anton Van Den, Hengel","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}