{"type":"ScholarlyArticle","title":"Entity-aware and Motion-aware Transformers for Language-driven Action Localization","authors":[{"name":"Shuo Yang","slug":"shuo-yang","given":"Shuo","family":"Yang","affiliation":"Beijing Laboratory of Intelligent Information Technology , School of Computer Science , Beijing Institute of Technology"},{"name":"Xinxiao Wu","slug":"xinxiao-wu","given":"Xinxiao","family":"Wu","affiliation":"Beijing Laboratory of Intelligent Information Technology , School of Computer Science , Beijing Institute of Technology"}],"abstract":"Language-driven action localization in videos is a challenging task that involves not only visual-linguistic matching but also action boundary prediction. Recent progress has been achieved through aligning language queries to video segments, but estimating precise boundaries is still under-explored. In this paper, we propose entity-aware and motion-aware Transformers that progressively localize actions in videos by first coarsely locating clips with entity queries and then finely predicting exact boundaries in a shrunken temporal region with motion queries. The entity-aware Transformer incorporates the textual entities into visual representation learning via cross-modal and cross-frame attentions to facilitate attending action-related video clips. The motion-aware Transformer captures fine-grained motion changes at multiple temporal scales via integrating long short-term memory into the self-attention module to further improve the precision of action boundary prediction. Extensive experiments on the Charades-STA and TACoS datasets demonstrate that our method achieves better performance than existing methods.","doi":"10.24963/ijcai.2022/216","url":"https://ijcai.org/proceedings/2022/216","pdf":"https://ijcai.org/proceedings/2022/0216.pdf","markdown":"https://ijcai.org/proceedings/2022/216.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/216","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":216,"pages":{"first":1552,"last":1558},"keywords":["Computer Vision: Vision and language","Computer Vision: Video analysis and understanding","Computer Vision: Image and Video retrieval"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986547","length_sec":80,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984690","length_sec":307,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Chen and Jiang, 2019] Shaoxiang Chen and Yu-Gang Jiang. Semantic proposal for activity localization in videos via sentence query. In Proc. of AAAI, 2019.","title":"Semantic proposal for activity localization in videos via sentence query","authors":"Chen, Jiang, Shaoxiang Chen, Yu-Gang Jiang","year":"2019","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ding et al., 2021] Xinpeng Ding, Nannan Wang, et al. Support-set based cross-supervision for video grounding. In Proc. of ICCV, 2021.","title":"Support-set based cross-supervision for video grounding","authors":"Ding","year":"2021","venue":"Proc. of ICCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Gao et al., 2017] Jiyang Gao, Chen Sun, et al. Tall: Tem- poral activity localization via language query. In Proc. of ICCV, 2017.","title":"Tall: Temporal activity localization via language query","authors":"Jiyang Gao, Chen Sun","year":"2017","venue":"Proc. of ICCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ghosh et al., 2019] Soham Ghosh, Anuva Agarwal, et al. Excl: Extractive clip localization using natural language descriptions. In Proc. of ACL, 2019.","title":"Excl: Extractive clip localization using natural language descriptions","authors":"Soham Ghosh, Anuva Agarwal","year":"2019","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hahn et al., 2019] Meera Hahn, Asim Kadav, et al. Tripping through time: Efficient localization of activities in videos. In BMVC, 2019.","title":"Tripping through time: Efficient localization of activities in videos","authors":"Meera Hahn, Asim Kadav","year":"2019","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hendricks et al., 2017] Lisa Anne Hendricks, Oliver Wang, et al. Localizing moments in video with natural language. In Proc. of ICCV, 2017.","title":"Localizing moments in video with natural language","authors":"Anne Lisa, Oliver Hendricks, Wang","year":"2017","venue":"Proc. of ICCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2021] Kun Li, Dan Guo, and Meng Wang. Proposal-free video grounding with contextual pyramid network. In Proc. of AAAI, 2021.","title":"Proposal-free video grounding with contextual pyramid network","authors":"Kun Li, Dan Guo, Meng Wang","year":"2021","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Liu et al., 2018] Meng Liu, Xiang Wang, et al. Attentive moment retrieval in videos. In Proc. of ACM SIGIR, 2018.","title":"Attentive moment retrieval in videos","authors":"Meng Liu, Xiang Wang","year":"2018","venue":"Proc. of ACM SIGIR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2020] Daizong Liu, Xiaoye Qu, et al. Jointly cross-and self-modal graph attention network for query- based moment localization. In Proc. of ACM MM, 2020.","title":"Jointly cross-and self-modal graph attention network for querybased moment localization","authors":"Liu","year":"2020","venue":"Proc. of ACM MM","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2021] Daizong Liu, Xiaoye Qu, et al. Context- aware biaffine localizing network for temporal sentence grounding. In Proc. of CVPR, 2021.","title":"Contextaware biaffine localizing network for temporal sentence grounding","authors":"Liu","year":"2021","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lu et al., 2019] Chujie Lu, Long Chen, et al. Debug: A dense bottom-up grounding approach for natural language video localization. In Proc. of EMNLP, 2019.","title":"Debug: A dense bottom-up grounding approach for natural language video localization","authors":"Lu","year":"2019","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mun et al., 2020] Jonghwan Mun, Minsu Cho, and Bo- hyung Han. Local-global video-text interactions for tem- poral grounding. In Proc. of CVPR, 2020.","title":"Local-global video-text interactions for temporal grounding","authors":"Mun","year":"2020","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Nan et al., 2021] Guoshun Nan, Rui Qiao, Yao Xiao, et al. Interventional video grounding with dual contrastive learning. In Proc. of CVPR, 2021.","title":"Interventional video grounding with dual contrastive learning","authors":"Nan","year":"2021","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ning et al., 2021] Ke Ning, Lingxi Xie, et al. Interaction- integrated network for natural language moment localiza- tion. TIP, 2021.","title":"Interactionintegrated network for natural language moment localization","authors":"Ke Ning, Lingxi Xie","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, et al. Glove: Global vectors for word representa- tion. In Proc. of EMNLP, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher","year":"2014","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Qu et al., 2020] Xiaoye Qu, Pengwei Tang, et al. Fine- grained iterative attention network for temporal language localization in videos. In Proc. of ACM MM, 2020. [Regneri et al., 2013]","title":"Finegrained iterative attention network for temporal language localization in videos","authors":"Xiaoye Qu, Pengwei Tang","year":"2013","venue":"Proc. of ACM MM","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Michaela Regneri, Marcus Rohrbach, et al. Grounding action descriptions in videos. TACL, 2013.","title":"Grounding action descriptions in videos","authors":"Michaela Regneri, Marcus Rohrbach","year":"2013","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Rohrbach et al., 2012] Marcus Rohrbach, Michaela Reg- neri, et al. Script data for attribute-based recognition of composite activities. In ECCV, 2012.","title":"Script data for attribute-based recognition of composite activities","authors":"Rohrbach","year":"2012","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sigurdsson et al., 2016] Gunnar A Sigurdsson, Gül Varol, et al. Hollywood in homes: Crowdsourcing data collec- tion for activity understanding. In Proc. of ECCV, 2016.","title":"Hollywood in homes: Crowdsourcing data collection for activity understanding","authors":"Gül Gunnar A Sigurdsson, Varol","year":"2016","venue":"Proc. of ECCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tang et al., 2022] Haoyu Tang, Jihua Zhu, et al. Frame-wise cross-modal matching for video moment retrieval. IEEE Transactions on Multimedia, 2022.","title":"Frame-wise cross-modal matching for video moment retrieval","authors":"Haoyu Tang, Jihua Zhu","year":"2022","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, et al. Attention is all you need. In NeurIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2021] Hao Wang, Zheng-Jun Zha, et al. Struc- tured multi-level interaction network for video moment lo- calization via language query. In Proc. of CVPR, 2021.","title":"Structured multi-level interaction network for video moment localization via language query","authors":"Hao Wang, Zheng-Jun Zha","year":"2021","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2020] Jie Wu, Guanbin Li, et al. Tree-structured policy based progressive reinforcement learning for tem- porally language grounding in video. In Proc. of AAAI, 2020.","title":"Tree-structured policy based progressive reinforcement learning for temporally language grounding in video","authors":"Jie Wu, Guanbin Li","year":"2020","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xiao et al., 2021] Shaoning Xiao, Long Chen, et al. Bound- ary proposal network for two-stage natural language video localization. In Proc. of AAAI, 2021.","title":"Boundary proposal network for two-stage natural language video localization","authors":"Xiao","year":"2021","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yuan et al., 2019] Yitian Yuan, Tao Mei, and Wenwu Zhu. To find where you talk: Temporal sentence localization in video with attention based location regression. In Proc. of AAAI, 2019.","title":"To find where you talk: Temporal sentence localization in video with attention based location regression","authors":"Yuan","year":"2019","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yuan et al., 2020] Yitian Yuan, Lin Ma, et al. Semantic conditioned dynamic modulation for temporal sentence grounding in videos. TPAMI, 2020.","title":"Semantic conditioned dynamic modulation for temporal sentence grounding in videos","authors":"Yitian Yuan, Lin Ma","year":"2020","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zeng et al., 2020] Runhao Zeng, Haoming Xu, et al. Dense regression network for video grounding. In Proc. of CVPR, 2020.","title":"Dense regression network for video grounding","authors":"Zeng","year":"2020","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2020] Songyang Zhang, Houwen Peng, and Jiebo Luo. Learning 2d temporal adjacent networks for moment localization with natural language. In Proc. of AAAI, 2020.","title":"Learning 2d temporal adjacent networks for moment localization with natural language","authors":"Songyang Zhang, Houwen Peng, Jiebo Luo","year":"2020","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2021a] Hao Zhang, Aixin Sun, et al. Natural language video localization: A revisit in span-based ques- tion answering framework. TPAMI, 2021.","title":"Natural language video localization: A revisit in span-based question answering framework","authors":"Zhang","year":"2021","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2021b] Hao Zhang, Aixin Sun, Wei Jing, et al. Parallel attention network with sequence matching for video grounding. In Proc. of ACL, 2021.","title":"Parallel attention network with sequence matching for video grounding","authors":"Zhang","year":"2021","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2021c] Mingxing Zhang, Yang Yang, et al. Multi-stage aggregated transformer network for temporal language localization in videos. In Proc. of CVPR, 2021.","title":"Multi-stage aggregated transformer network for temporal language localization in videos","authors":"Zhang","year":"2021","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhao et al., 2021] Yang Zhao, Zhou Zhao, et al. Cascaded prediction network via segment tree for temporal video grounding. In Proc. of CVPR, 2021.","title":"Cascaded prediction network via segment tree for temporal video grounding","authors":"Yang Zhao, Zhou Zhao","year":"2021","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhou et al., 2021] Hao Zhou, Chongyang Zhang, et al. Em- bracing uncertainty: Decoupling and de-bias for robust temporal grounding. In Proc. of CVPR, 2021.","title":"Embracing uncertainty: Decoupling and de-bias for robust temporal grounding","authors":"Hao Zhou, Chongyang Zhang","year":"2021","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}