{"type":"ScholarlyArticle","title":"Hierarchical Attention Based Spatial-Temporal Graph-to-Sequence Learning for Grounded Video Description","authors":[{"name":"Kai Shen","slug":"kai-shen","given":"Kai","family":"Shen","affiliation":"Zhejiang University"},{"name":"Lingfei Wu","slug":"lingfei-wu","given":"Lingfei","family":"Wu","affiliation":"IBM Research"},{"name":"Fangli Xu","slug":"fangli-xu","given":"Fangli","family":"Xu","affiliation":"Squirrel AI Learning"},{"name":"Siliang Tang","slug":"siliang-tang","given":"Siliang","family":"Tang","affiliation":"Zhejiang University"},{"name":"Jun Xiao","slug":"jun-xiao","given":"Jun","family":"Xiao","affiliation":"Zhejiang University"},{"name":"Yueting Zhuang","slug":"yueting-zhuang","given":"Yueting","family":"Zhuang","affiliation":"Zhejiang University"}],"abstract":"The task of Grounded Video Description~(GVD) is to generate sentences whose objects can be grounded with the bounding boxes in the video frames. Existing works often fail to exploit structural information both in modeling the relationships among the region proposals and in attending them for text generation. To address these issues, we cast the GVD task as a spatial-temporal Graph-to-Sequence learning problem, where we model video frames as spatial-temporal sequence graph in order to better capture implicit structural relationships. In particular, we exploit two ways to construct a sequence graph that captures spatial-temporal correlations among different objects in each frame and further present a novel graph topology refinement technique to discover optimal underlying graph structure. In addition, we also present hierarchical attention mechanism to attend sequence graph in different resolution levels for better generating the sentences. Our extensive experiments demonstrate the effectiveness of our proposed method compared to state-of-the-art methods.","doi":"10.24963/ijcai.2020/131","url":"https://ijcai.org/proceedings/2020/131","pdf":"https://ijcai.org/proceedings/2020/0131.pdf","markdown":"https://ijcai.org/proceedings/2020/131.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/131","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":131,"pages":{"first":941,"last":947},"keywords":["Computer Vision: Video: Events, Activities and Surveillance","Computer Vision: Language and Vision"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25361/file_25361.m3u8","length_sec":299,"provider":"hls"}],"references":[{"position":1,"raw":"[Anderson et al., 2018] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for im- age captioning and visual question answering. In Proceed- ings of the IEEE Conference on Computer Vision and Pat- tern Recognition, pages 6077-6086, 2018.","title":"Bottom-up and top-down attention for image captioning and visual question answering","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2019] Yu Chen, Lingfei Wu, and Mohammed J Zaki. Deep iterative and adaptive learning for graph neural networks. arXiv preprint arXiv:1912.07832, 2019.","title":"Deep iterative and adaptive learning for graph neural networks","authors":"Yu Chen, Lingfei Wu, Mohammed J Zaki","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2020] Yu Chen, Lingfei Wu, and Mohammed J Zaki. Reinforcement learning based graph-to-sequence model for natural question generation. In ICLR, 2020.","title":"Reinforcement learning based graph-to-sequence model for natural question generation","authors":"Yu Chen, Lingfei Wu, Mohammed J Zaki","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gao et al., 2019] Yuyang Gao, Lingfei Wu, Houman Homayoun, and Liang Zhao. Dyngraph2seq: Dynamic- graph-to-sequence interpretable learning for health stage prediction in online health forums. In ICDM, 2019.","title":"Dyngraph2seq: Dynamicgraph-to-sequence interpretable learning for health stage prediction in online health forums","authors":"Yuyang Gao, Lingfei Wu, Houman Homayoun, Liang Zhao","year":"2019","venue":"ICDM","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Krishna et al., 2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Inter- national Journal of Computer Vision, 123(1):32-73, 2017.","title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","authors":"Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma","year":"2017","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Li et al., 2019a] Juncheng Li, Xin Wang, Siliang Tang, Haizhou Shi, Fei Wu, Yueting Zhuang, and William Yang Wang. Unsupervised reinforcement learning of transfer- able meta-skills for embodied navigation. arXiv preprint arXiv:1911.07450, 2019.","title":"Unsupervised reinforcement learning of transferable meta-skills for embodied navigation","authors":"Li","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2019b] Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu. Relation-aware graph attention network for visual question answering. arXiv preprint arXiv:1903.12314, 2019.","title":"Relation-aware graph attention network for visual question answering","authors":"Li","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Liu et al., 2016] An-An Liu, Yu-Ting Su, Wei-Zhi Nie, and Mohan Kankanhalli. Hierarchical clustering multi-task learning for joint human action grouping and recognition. IEEE transactions on pattern analysis and machine intel- ligence, 39(1):102-114, 2016.","title":"Hierarchical clustering multi-task learning for joint human action grouping and recognition","authors":"Liu","year":"2016","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2018] Anan Liu, Ning Xu, Hanwang Zhang, Weizhi Nie, Yuting Su, and Yongdong Zhang. Multi-level policy and reward reinforcement learning for image cap- tioning. In IJCAI, pages 821-827, 2018.","title":"Multi-level policy and reward reinforcement learning for image captioning","authors":"Anan Liu, Ning Xu, Hanwang Zhang, Weizhi Nie, Yuting Su, Yongdong Zhang","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/114"},{"position":11,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time ob- ject detection with region proposal networks. In Advances in neural information processing systems, pages 91-99, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Tomei et al., 2019] Matteo Tomei, Lorenzo Baraldi, Si- mone Calderara, Simone Bronzin, and Rita Cucchiara. Stage: Spatio-temporal attention on graph entities for video action detection. arXiv preprint arXiv:1912.04316, 2019.","title":"Stage: Spatio-temporal attention on graph entities for video action detection","authors":"Matteo Tomei, Lorenzo Baraldi, Simone Calderara, Simone Bronzin, Rita Cucchiara","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Tran et al., 2015] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on com- puter vision, pages 4489-4497, 2015.","title":"Learning spatiotemporal features with 3d convolutional networks","authors":"Tran","year":"2015","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing sys- tems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Venugopalan et al., 2015] Subhashini Venugopalan, Mar- cus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence- video to text. In Proceedings of the IEEE international conference on computer vision, pages 4534-4542, 2015.","title":"Sequence to sequencevideo to text","authors":"Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, Kate Saenko","year":"2015","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wang and Gupta, 2018] Xiaolong Wang and Abhinav Gupta. Videos as space-time region graphs. In Proceed- ings of the European Conference on Computer Vision (ECCV), pages 399-417, 2018.","title":"Videos as space-time region graphs","authors":"Wang, Gupta, Xiaolong Wang, Abhinav Gupta","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Xu et al., 2018a] Kun Xu, Lingfei Wu, Zhiguo Wang, Yansong Feng, Michael Witbrock, and Vadim Sheinin. Graph2seq: Graph to sequence learning with attention- based neural networks. arXiv preprint arXiv:1804.00823, 2018.","title":"Graph2seq: Graph to sequence learning with attentionbased neural networks","authors":"Xu","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Xu et al., 2018b] Ning Xu, An-An Liu, Yongkang Wong, Yongdong Zhang, Weizhi Nie, Yuting Su, and Mohan Kankanhalli. Dual-stream recurrent neural network for video captioning. IEEE Transactions on Circuits and Sys- tems for Video Technology, 29(8):2482-2493, 2018.","title":"Dual-stream recurrent neural network for video captioning","authors":"Xu","year":"2018","venue":"IEEE Transactions on Circuits and Systems for Video Technology","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Yang et al., 2019] Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai. Auto-encoding scene graphs for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10685- 10694, 2019.","title":"Auto-encoding scene graphs for image captioning","authors":"Xu Yang, Kaihua Tang, Hanwang Zhang, Jianfei Cai","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yao et al., 2018] Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Exploring visual relationship for image captioning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 684-699, 2018.","title":"Exploring visual relationship for image captioning","authors":"Ting Yao, Yingwei Pan, Yehao Li, Tao Mei","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang and Peng, 2019] Junchao Zhang and Yuxin Peng. Object-aware aggregation with bidirectional temporal graph for video captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8327-8336, 2019.","title":"Object-aware aggregation with bidirectional temporal graph for video captioning","authors":"Peng Zhang, Junchao Zhang, Yuxin Peng","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhou et al., 2018] Luowei Zhou, Yingbo Zhou, Jason J Corso, Richard Socher, and Caiming Xiong. End-to-end dense video captioning with masked transformer. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8739-8748, 2018.","title":"End-to-end dense video captioning with masked transformer","authors":"Zhou","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhou et al., 2019] Luowei Zhou, Yannis Kalantidis, Xinlei Chen, Jason J Corso, and Marcus Rohrbach. Grounded video description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6578- 6587, 2019.","title":"Grounded video description","authors":"Zhou","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}