{"type":"ScholarlyArticle","title":"Multi-modal Circulant Fusion for Video-to-Language and Backward","authors":[{"name":"Aming Wu","slug":"aming-wu","given":"Aming","family":"Wu","affiliation":"School of Computer Science and Technology , Tianjin University , Tianjin , China"},{"name":"Yahong Han","slug":"yahong-han","given":"Yahong","family":"Han","affiliation":"School of Computer Science and Technology , Tianjin University , Tianjin , China"}],"abstract":"Multi-modal fusion has been widely involved in focuses of the modern artificial intelligence research, e.g., from visual content to languages and backward. Common-used multi-modal fusion methods mainly include element-wise product, element-wise sum, or even simply concatenation between different types of features, which are somewhat straightforward but lack in-depth analysis. Recent studies have shown fully exploiting interactions among elements of multi-modal features will lead to a further performance gain. In this paper, we put forward a new approach of multi-modal fusion, namely Multi-modal Circulant Fusion (MCF). Particularly, after reshaping feature vectors into circulant matrices, we define two types of interaction operations between vectors and matrices. As each row of the circulant matrix shifts one elements, with newly-defined interaction operations, we almost explore all possible interactions between vectors of different modalities. Moreover, as only regular operations are involved and defined a priori, MCF avoids increasing parameters or computational costs for multi-modal fusion. We evaluate MCF with tasks of video captioning and temporal activity localization via language (TALL). Experiments on MSVD and MSRVTT show our method obtains the state-of-the-art for video captioning. For TALL, by plugging into MCF, we achieve a performance gain of roughly 4.2% on TACoS.","doi":"10.24963/ijcai.2018/143","url":"https://ijcai.org/proceedings/2018/143","pdf":"https://ijcai.org/proceedings/2018/0143.pdf","markdown":"https://ijcai.org/proceedings/2018/143.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/143","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":143,"pages":{"first":1029,"last":1035},"keywords":["Computer Vision: Video: Events, Activities and Surveillance","Computer Vision: Language and Vision","Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation"],"videos":[],"references":[{"position":1,"raw":"[Ballas et al., 2016] Nicolas Ballas, Li Yao, Chris Pal, and Aaron Courville. Delving deeper into convolutional net- works for learning video representations. ICLR, 2016.","title":"Delving deeper into convolutional networks for learning video representations","authors":"Nicolas Ballas, Li Yao, Chris Pal, Aaron Courville","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baraldi et al., 2017] Lorenzo Baraldi, Costantino Grana, and Rita Cucchiara. Hierarchical boundary-aware neural encoder for video captioning. CVPR, 2017.","title":"Hierarchical boundary-aware neural encoder for video captioning","authors":"Lorenzo Baraldi, Costantino Grana, Rita Cucchiara","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen and Dolan, 2011] David L Chen and William B Dolan. Collecting highly parallel data for paraphrase eval- uation. In ACL, pages 190-200, 2011.","title":"Collecting highly parallel data for paraphrase evaluation","authors":"L David, William B Chen, Dolan","year":"2011","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2017] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convo- lutional nets, atrous convolution, and fully connected crfs. TPAMI, 2017.","title":"Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs","authors":"Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan L Yuille","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fukui et al., 2016] Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for vi- sual question answering and visual grounding. In EMNLP, pages 457-468, 2016.","title":"Multimodal compact bilinear pooling for visual question answering and visual grounding","authors":"Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, Marcus Rohrbach","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gao et al., 2017] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In ICCV, Oct 2017.","title":"Tall: Temporal activity localization via language query","authors":"Jiyang Gao, Chen Sun, Zhenheng Yang, Ram Nevatia","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, June 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Karpathy and Fei-Fei, 2015] Andrej Karpathy and Li Fei- Fei. Deep visual-semantic alignments for generating im- age descriptions. In CVPR, pages 3128-3137, 2015.","title":"Deep visual-semantic alignments for generating image descriptions","authors":"Fei-Fei ; Karpathy, Andrej Karpathy, Li Fei-Fei","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kiros et al., 2015] Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In NIPS, pages 3294-3302. 2015.","title":"Skip-thought vectors","authors":"Ryan Kiros, Yukun Zhu, R Ruslan, Richard Salakhutdinov, Raquel Zemel, Antonio Urtasun, Sanja Torralba, Fidler","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2017] Xuelong Li, Bin Zhao, and Xiaoqiang Lu. Mam-rnn: Multi-level attention model based rnn for video captioning. In IJCAI, 2017.","title":"Mam-rnn: Multi-level attention model based rnn for video captioning","authors":"Xuelong Li, Bin Zhao, Xiaoqiang Lu","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/307"},{"position":11,"raw":"[Pan et al., 2016a] Pingbo Pan, Zhongwen Xu, Yi Yang, Fei Wu, and Yueting Zhuang. Hierarchical recurrent neural encoder for video representation with application to cap- tioning. In CVPR, pages 1029-1038, 2016.","title":"Hierarchical recurrent neural encoder for video representation with application to captioning","authors":"Pan","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Pan et al., 2016b] Yingwei Pan, Tao Mei, Ting Yao, Houqiang Li, and Yong Rui. Jointly modeling embedding and translation to bridge video and language. In CVPR, pages 4594-4602, 2016.","title":"Jointly modeling embedding and translation to bridge video and language","authors":"Pan","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Peng et al., 2018] Yuxin Peng, Jinwei Qi, Xin Huang, and Yuxin Yuan. Ccl: Cross-modal correlation learning with multigrained fusion by hierarchical network. IEEE Trans- actions on Multimedia, 20(2):405-420, 2018.","title":"Ccl: Cross-modal correlation learning with multigrained fusion by hierarchical network","authors":"Yuxin Peng, Jinwei Qi, Xin Huang, Yuxin Yuan","year":"2018","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Regneri et al., 2013] Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Man- fred Pinkal. Grounding action descriptions in videos. TACL, 1:25-36, 2013.","title":"Grounding action descriptions in videos","authors":"Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, Manfred Pinkal","year":"2013","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Song et al., 2017] Jingkuan Song, Zhao Guo, Lianli Gao, Wu Liu, Dongxiang Zhang, and Heng Tao Shen. Hier- archical lstm with adjusted temporal attention for video captioning. arXiv preprint arXiv:1706.01231, 2017.","title":"Hierarchical lstm with adjusted temporal attention for video captioning","authors":"Song","year":"2017","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2017/381"},{"position":16,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabi- novich. Going deeper with convolutions. In CVPR, pages 1-9, 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tenenbaum and Freeman, 1997] Joshua B Tenenbaum and William T Freeman. Separating style and content. In NIPS, pages 662-668, 1997.","title":"Separating style and content","authors":"B Joshua, William T Tenenbaum, Freeman","year":"1997","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Venugopalan et al., 2014] Subhashini Venugopalan, Hui- juan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, and Kate Saenko. Translating videos to natural language using deep recurrent neural networks. NAACL, 2014.","title":"Translating videos to natural language using deep recurrent neural networks","authors":"Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, Kate Saenko","year":"2014","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Venugopalan et al., 2015] Subhashini Venugopalan, Mar- cus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence- video to text. In ICCV, pages 4534-4542, 2015.","title":"Sequence to sequencevideo to text","authors":"Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, Kate Saenko","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xu et al., 2016] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, pages 5288-5296, 2016.","title":"Msr-vtt: A large video description dataset for bridging video and language","authors":"Jun Xu, Tao Mei, Ting Yao, Yong Rui","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Xu et al., 2017] Jun Xu, Ting Yao, Yongdong Zhang, and Tao Mei. Learning multimodal attention lstm networks for video captioning. In ACM MM, pages 537-545. ACM, 2017.","title":"Learning multimodal attention lstm networks for video captioning","authors":"Jun Xu, Ting Yao, Yongdong Zhang, Tao Mei","year":"2017","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yang et al., 2017] Ziwei Yang, Yahong Han, and Zheng Wang. Catching the temporal regions-of-interest for video captioning. In Proceedings of the 2017 ACM on Multime- dia Conference, pages 146-153. ACM, 2017.","title":"Catching the temporal regions-of-interest for video captioning","authors":"Ziwei Yang, Yahong Han, Zheng Wang","year":"2017","venue":"Proceedings of the 2017 ACM on Multimedia Conference","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yao et al., 2015] Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, and Aaron Courville. Describing videos by exploiting tempo- ral structure. In ICCV, pages 4507-4515, 2015.","title":"Describing videos by exploiting temporal structure","authors":"Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, Aaron Courville","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yu et al., 2016] Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, and Wei Xu. Video paragraph captioning using hierarchical recurrent neural networks. In CVPR, pages 4584-4593, 2016.","title":"Video paragraph captioning using hierarchical recurrent neural networks","authors":"Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, Wei Xu","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yu et al., 2017] Zhou Yu, Jun Yu, Jianping Fan, and Dacheng Tao. Multi-modal factorized bilinear pooling with co-attention learning for visual question answering. In ICCV, Oct 2017.","title":"Multi-modal factorized bilinear pooling with co-attention learning for visual question answering","authors":"Zhou Yu, Jun Yu, Jianping Fan, Dacheng Tao","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2016] Yuting Zhang, Kibok Lee, and Honglak Lee. Augmenting supervised neural networks with unsu- pervised objectives for large-scale image classification. In ICML, pages 612-621, 2016.","title":"Augmenting supervised neural networks with unsupervised objectives for large-scale image classification","authors":"Yuting Zhang, Kibok Lee, Honglak Lee","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2018] Hanwang Zhang, Yulei Niu, and Shih- Fu Chang. Grounding referring expressions in images by variational context. In CVPR, 2018.","title":"Grounding referring expressions in images by variational context","authors":"Hanwang Zhang, Yulei Niu, Shih-Fu Chang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhu et al., 2017] Linchao Zhu, Zhongwen Xu, and Yi Yang. Bidirectional multirate reconstruction for temporal model- ing in videos. In CVPR, July 2017.","title":"Bidirectional multirate reconstruction for temporal modeling in videos","authors":"Zhu","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}