{"type":"ScholarlyArticle","title":"SBAT: Video Captioning with Sparse Boundary-Aware Transformer","authors":[{"name":"Tao Jin","slug":"tao-jin","given":"Tao","family":"Jin","affiliation":"College of Information Science & Electronic Engineering , Zhejiang University , China"},{"name":"Siyu Huang","slug":"siyu-huang","given":"Siyu","family":"Huang","affiliation":"Baidu Research , China"},{"name":"Ming Chen","slug":"ming-chen","given":"Ming","family":"Chen","affiliation":"Alibaba Group , China"},{"name":"Yingming Li","slug":"yingming-li","given":"Yingming","family":"Li","affiliation":"College of Information Science & Electronic Engineering , Zhejiang University , China"},{"name":"Zhongfei Zhang","slug":"zhongfei-zhang","given":"Zhongfei","family":"Zhang","affiliation":"Department of Computer Science , Binghamton University , USA"}],"abstract":"In this paper, we focus on the problem of applying the transformer structure to video captioning effectively. The vanilla transformer is proposed for uni-modal language generation task such as machine translation. However, video captioning is a multimodal learning problem, and the video features have much redundancy between different time steps. Based on these concerns, we propose a novel method called sparse boundary-aware transformer (SBAT) to reduce the redundancy in video representation. SBAT employs boundary-aware pooling operation for scores from multihead attention and selects diverse features from different scenarios. Also, SBAT includes a local correlation scheme to compensate for the local information loss brought by sparse operation. Based on SBAT, we further propose an aligned cross-modal encoding scheme to boost the multimodal interaction. Experimental results on two benchmark datasets show that SBAT outperforms the state-of-the-art methods under most of the metrics.","doi":"10.24963/ijcai.2020/88","url":"https://ijcai.org/proceedings/2020/88","pdf":"https://ijcai.org/proceedings/2020/0088.pdf","markdown":"https://ijcai.org/proceedings/2020/88.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/88","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":88,"pages":{"first":630,"last":636},"keywords":["Computer Vision: Video: Events, Activities and Surveillance","Computer Vision: Language and Vision"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24064/file_24064.m3u8","length_sec":339,"provider":"hls"}],"references":[{"position":1,"raw":"[Aafaq et al., 2019] Nayyer Aafaq, Naveed Akhtar, Wei Liu, Syed Zulqarnain Gilani, and Ajmal Mian. Spatio-temporal dynamics and semantic attribute enriched visual encoding for video captioning. In CVPR, 2019.","title":"Spatio-temporal dynamics and semantic attribute enriched visual encoding for video captioning","authors":"Nayyer Aafaq, Naveed Akhtar, Wei Liu, Ajmal Syed Zulqarnain Gilani, Mian","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Ji- asen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015.","title":"Vqa: Visual question answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Carreira and Zisserman, 2017] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017.","title":"Quo vadis, action recognition? a new model and the kinetics dataset","authors":"Joao Carreira, Andrew Zisserman","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen and Dolan, 2011] David L Chen and William B Dolan. Collecting highly parallel data for paraphrase eval- uation. In ACL, 2011.","title":"Collecting highly parallel data for paraphrase evaluation","authors":"L David, William B Chen, Dolan","year":"2011","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen and Jiang, 2019] Shaoxiang Chen and Yu-Gang Jiang. Motion guided spatial attention for video captioning, 2019.","title":"Motion guided spatial attention for video captioning","authors":"Chen, Jiang, Shaoxiang Chen, Yu-Gang Jiang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2018] Ming Chen, Yingming Li, Zhongfei Zhang, and Siyu Huang. Tvt: Two-view transformer net- work for video captioning. In ACML, 2018.","title":"Tvt: Two-view transformer network for video captioning","authors":"Ming Chen, Yingming Li, Zhongfei Zhang, Siyu Huang","year":"2018","venue":"ACML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gan et al., 2017] Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, and Li Deng. Semantic compositional networks for visual captioning. In CVPR, 2017.","title":"Semantic compositional networks for visual captioning","authors":"Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, Li Deng","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hori et al., 2017] Chiori Hori, Takaaki Hori, Teng-Yok Lee, Ziming Zhang, Bret Harsham, John R Hershey, Tim K Marks, and Kazuhiko Sumi. Attention-based multimodal fusion for video description. In ICCV, 2017.","title":"Attention-based multimodal fusion for video description","authors":"Chiori Hori, Takaaki Hori, Teng-Yok Lee, Ziming Zhang, Bret Harsham, John R Hershey, Tim K Marks, Kazuhiko Sumi","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jin et al., 2019a] Tao Jin, Siyu Huang, Yingming Li, and Zhongfei Zhang. Low-rank hoca: Efficient high-order cross-modal attention for video captioning. arXiv preprint arXiv:1911.00212, 2019.","title":"Low-rank hoca: Efficient high-order cross-modal attention for video captioning","authors":"Jin","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jin et al., 2019b] Tao Jin, Yingming Li, and Zhongfei Zhang. Recurrent convolutional video captioning with global and local attention. Neurocomputing, 2019.","title":"Recurrent convolutional video captioning with global and local attention","authors":"Jin","year":"2019","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2019] Xiangpeng Li, Jingkuan Song, Lianli Gao, Xianglong Liu, Wenbing Huang, Xiangnan He, and Chuang Gan. Beyond rnns: Positional self-attention with co-attention for video question answering. In AAAI, 2019.","title":"Beyond rnns: Positional self-attention with co-attention for video question answering","authors":"Li","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Long et al., 2018] Xiang Long, Chuang Gan, and Gerard de Melo. Video captioning with multi-faceted attention. TACL, 2018.","title":"Video captioning with multi-faceted attention","authors":"Xiang Long, Chuang Gan, Gerard De, Melo","year":"2018","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pan et al., 2017] Yingwei Pan, Ting Yao, Houqiang Li, and Tao Mei. Video captioning with transferred semantic at- tributes. In CVPR, 2017.","title":"Video captioning with transferred semantic attributes","authors":"Pan","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pei et al., 2019] Wenjie Pei, Jiyuan Zhang, Xiangrong Wang, Lei Ke, Xiaoyong Shen, and Yu-Wing Tai. Memory-attended recurrent network for video captioning. In CVPR, 2019.","title":"Memory-attended recurrent network for video captioning","authors":"Wenjie Pei, Jiyuan Zhang, Xiangrong Wang, Lei Ke, Xiaoyong Shen, Yu-Wing Tai","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Shen et al., 2017] Zhiqiang Shen, Jianguo Li, Zhou Su, Minjun Li, Yurong Chen, Yu-Gang Jiang, and Xiangyang Xue. Weakly supervised dense video captioning. In CVPR, 2017.","title":"Weakly supervised dense video captioning","authors":"Zhiqiang Shen, Jianguo Li, Zhou Su, Minjun Li, Yurong Chen, Yu-Gang Jiang, Xiangyang Xue","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Szegedy et al., 2017] Christian Szegedy, Sergey Ioffe, Vin- cent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In AAAI, 2017.","title":"Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning","authors":"Szegedy","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Venugopalan et al., 2015] Subhashini Venugopalan, Mar- cus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence- video to text. In ICCV, 2015.","title":"Sequence to sequencevideo to text","authors":"Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, Kate Saenko","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2019] Bairui Wang, Lin Ma, Wei Zhang, Wen- hao Jiang, Jingwen Wang, and Wei Liu. Controllable video captioning with pos sequence guidance based on gated fu- sion network. In ICCV, 2019.","title":"Controllable video captioning with pos sequence guidance based on gated fusion network","authors":"Bairui Wang, Lin Ma, Wei Zhang, Wenhao Jiang, Jingwen Wang, Wei Liu","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xu et al., 2016] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016.","title":"Msr-vtt: A large video description dataset for bridging video and language","authors":"Jun Xu, Tao Mei, Ting Yao, Yong Rui","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yao et al., 2015] Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, and Aaron Courville. Describing videos by exploiting tempo- ral structure. In ICCV, 2015.","title":"Describing videos by exploiting temporal structure","authors":"Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, Aaron Courville","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[You et al., 2016] Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In CVPR, 2016.","title":"Image captioning with semantic attention","authors":"You","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhou et al., 2018] Luowei Zhou, Yingbo Zhou, Jason J Corso, Richard Socher, and Caiming Xiong. End-to- end dense video captioning with masked transformer. In CVPR, 2018.","title":"End-toend dense video captioning with masked transformer","authors":"Zhou","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}