{"type":"ScholarlyArticle","title":"Polar Relative Positional Encoding for Video-Language Segmentation","authors":[{"name":"Ke Ning","slug":"ke-ning","given":"Ke","family":"Ning","affiliation":"Zhejiang University"},{"name":"Lingxi Xie","slug":"lingxi-xie","given":"Lingxi","family":"Xie","affiliation":"Huawei Noah's Ark Lab"},{"name":"Fei Wu","slug":"fei-wu","given":"Fei","family":"Wu","affiliation":"Zhejiang University"},{"name":"Qi Tian","slug":"qi-tian","given":"Qi","family":"Tian","affiliation":"Huawei Noah's Ark Lab"}],"abstract":"In this paper, we tackle a challenging task named video-language segmentation. Given a video and a sentence in natural language, the goal is to segment the object or actor described by the sentence in video frames. To accurately denote a target object, the given sentence usually refers to multiple attributes, such as nearby objects with spatial relations, etc. In this paper, we propose a novel Polar Relative Positional Encoding (PRPE) mechanism that represents spatial relations in a ``linguistic'' way, i.e., in terms of direction and range. Sentence feature can interact with positional embeddings in a more direct way to extract the implied relative positional relations. We also propose parameterized functions for these positional embeddings to adapt real-value directions and ranges. With PRPE, we design a Polar Attention Module (PAM) as the basic module for vision-language fusion. Our method outperforms previous best method by a large margin of 11.4% absolute improvement in terms of mAP on the challenging A2D Sentences dataset. Our method also achieves competitive performances on the J-HMDB Sentences dataset.","doi":"10.24963/ijcai.2020/132","url":"https://ijcai.org/proceedings/2020/132","pdf":"https://ijcai.org/proceedings/2020/0132.pdf","markdown":"https://ijcai.org/proceedings/2020/132.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/132","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":132,"pages":{"first":948,"last":954},"keywords":["Computer Vision: Video: Events, Activities and Surveillance","Computer Vision: Language and Vision","Computer Vision: Action Recognition"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24633/file_24633.m3u8","length_sec":263,"provider":"hls"}],"references":[{"position":1,"raw":"[Anderson et al., 2018] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision- and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR, 2018.","title":"Visionand-language navigation: Interpreting visually-grounded navigation instructions in real environments","authors":"Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, Anton Van Den, Hengel","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anne Hendricks et al., 2017] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natu- ral language. In ICCV, 2017.","title":"Localizing moments in video with natural language","authors":"Anne Lisa, Oliver Hendricks, Eli Wang, Josef Shechtman, Trevor Sivic, Bryan Darrell, Russell","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Ji- asen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015.","title":"Vqa: Visual question answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bello et al., 2019] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V Le. Attention aug- mented convolutional networks. In ICCV, 2019.","title":"Attention augmented convolutional networks","authors":"Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, Quoc V Le","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Carreira and Zisserman, 2017] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017.","title":"Quo vadis, action recognition? a new model and the kinetics dataset","authors":"Joao Carreira, Andrew Zisserman","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[De Vries et al., 2017] Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. In NeurIPS, 2017.","title":"Modulating early visual processing by language","authors":"De Vries","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gao et al., 2017] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In ICCV, 2017.","title":"Tall: Temporal activity localization via language query","authors":"Jiyang Gao, Chen Sun, Zhenheng Yang, Ram Nevatia","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gavrilyuk et al., 2018] Kirill Gavrilyuk, Amir Ghodrati, Zhenyang Li, and Cees GM Snoek. Actor and action video segmentation from a sentence. In CVPR, 2018.","title":"Actor and action video segmentation from a sentence","authors":"Kirill Gavrilyuk, Amir Ghodrati, Zhenyang Li, G M Cees, Snoek","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gu et al., 2018] Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheen- dra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. Ava: A video dataset of spatio- temporally localized atomic visual actions. In CVPR, 2018.","title":"Ava: A video dataset of spatiotemporally localized atomic visual actions","authors":"Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hu et al., 2016a] Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. Segmentation from natural language ex- pressions. In ECCV, 2016.","title":"Segmentation from natural language expressions","authors":"Hu","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hu et al., 2016b] Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. Natural language object retrieval. In CVPR, 2016.","title":"Natural language object retrieval","authors":"Hu","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huang et al., 2019] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer: Generating music with long-term structure. In ICLR, 2019. [Jhuang et al., 2013] Hueihan Jhuang, Juergen Gall, Silvia Zuffi, Cordelia Schmid, and Michael J Black. Towards understanding action recognition. In ICCV, 2013.","title":"Music transformer: Generating music with long-term structure","authors":"Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, Douglas Eck","year":"2013","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jiang et al., 2014] Yu-Gang Jiang, Jingen Liu, A Roshan Zamir, George Toderici, Ivan Laptev, Mubarak Shah, and Rahul Sukthankar. Thumos challenge: Action recogni- tion with a large number of classes. http://crcv.ucf.edu/ THUMOS14/, 2014.","title":"Thumos challenge: Action recognition with a large number of classes","authors":"Yu-Gang Jiang, Jingen Liu, Roshan Zamir, George Toderici, Ivan Laptev, Mubarak Shah, Rahul Sukthankar","year":null,"venue":null,"doi":null,"url":"http://crcv.ucf.edu/THUMOS14/,2014","internal_url":null},{"position":15,"raw":"[Johnson et al., 2017] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for com- positional language and elementary visual reasoning. In CVPR, 2017.","title":"Clevr: A diagnostic dataset for compositional language and elementary visual reasoning","authors":"Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, Ross Girshick","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2017] Zhenyang Li, Ran Tao, Efstratios Gavves, Cees GM Snoek, and Arnold WM Smeulders. Tracking by natural language specification. In CVPR, 2017.","title":"Tracking by natural language specification","authors":"Li","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[McIntosh et al., 2018] Bruce McIntosh, Kevin Duarte, Yo- gesh S Rawat, and Mubarak Shah. Multi-modal cap- sule routing for actor and action video segmentation con- ditioned on natural language queries. arXiv preprint arXiv:1812.00303, 2018.","title":"Multi-modal capsule routing for actor and action video segmentation conditioned on natural language queries","authors":"Bruce Mcintosh, Kevin Duarte, Yogesh S Rawat, Mubarak Shah","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Parmar et al., 2019] Niki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self-attention in vision models. In NeurIPS, 2019.","title":"Stand-alone self-attention in vision models","authors":"Niki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan Bello, Anselm Levskaya, Jon Shlens","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Perazzi et al., 2016] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In CVPR, 2016.","title":"A benchmark dataset and evaluation methodology for video object segmentation","authors":"Federico Perazzi, Jordi Pont-Tuset, Brian Mcwilliams, Luc Van Gool, Markus Gross, Alexander Sorkine-Hornung","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shaw et al., 2018] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. In NAACL, 2018.","title":"Self-attention with relative position representations","authors":"Peter Shaw, Jakob Uszkoreit, Ashish Vaswani","year":"2018","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Two-stream convolutional networks for action recognition in videos. In NeurIPS, 2014.","title":"Two-stream convolutional networks for action recognition in videos","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tran et al., 2015] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In ICCV, 2015.","title":"Learning spatiotemporal features with 3d convolutional networks","authors":"Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, Manohar Paluri","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2018] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018.","title":"Non-local neural networks","authors":"Xiaolong Wang, Ross Girshick, Abhinav Gupta, Kaiming He","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2019] Hao Wang, Cheng Deng, Junchi Yan, and Dacheng Tao. Asymmetric cross-guided attention net- work for actor and action video segmentation from natural language query. In ICCV, 2019.","title":"Asymmetric cross-guided attention network for actor and action video segmentation from natural language query","authors":"Hao Wang, Cheng Deng, Junchi Yan, Dacheng Tao","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xu et al., 2015] Chenliang Xu, Shao-Hang Hsieh, Caiming Xiong, and Jason J Corso. Can humans fly? action under- standing with multiple classes of actors. In CVPR, 2015.","title":"Can humans fly? action understanding with multiple classes of actors","authors":"Xu","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}