{"type":"ScholarlyArticle","title":"MAT: A Multimodal Attentive Translator for Image Captioning","authors":[{"name":"Chang Liu","slug":"chang-liu","given":"Chang","family":"Liu","affiliation":"Department of Computer Science , Tsinghua University; Cognitive Science & Computer Science , Johns Hopkins University"},{"name":"Fuchun Sun","slug":"fuchun-sun","given":"Fuchun","family":"Sun","affiliation":"Department of Computer Science , Tsinghua University"},{"name":"Changhu Wang","slug":"changhu-wang","given":"Changhu","family":"Wang","affiliation":"Toutiao AI Lab ,"},{"name":"Feng Wang","slug":"feng-wang","given":"Feng","family":"Wang","affiliation":"Department of Electronic Engineering , UESTC; Cognitive Science & Computer Science , Johns Hopkins University"},{"name":"Alan Yuille","slug":"alan-yuille","given":"Alan","family":"Yuille","affiliation":"Cognitive Science & Computer Science , Johns Hopkins University"}],"abstract":"In this work we formulate the problem of image captioning as a multimodal translation task. Analogous to machine translation, we present a sequence-to-sequence recurrent neural networks (RNN) model for image caption generation. Different from most existing work where the whole image is represented by convolutional neural network (CNN) feature, we propose to represent the input image as a sequence of detected objects which feeds as the source sequence of the RNN model. In this way, the sequential representation of an image can be naturally translated to a sequence of words, as the target sequence of the RNN model. To represent the image in a sequential way, we extract the objects features in the image and arrange them in a order using convolutional neural networks. To further leverage the visual information from the encoded objects, a sequential attention layer is introduced to selectively attend to the objects that are related to generate corresponding words in the sentences. Extensive experiments are conducted to validate the proposed approach on popular benchmark dataset, i.e., MS COCO, and the proposed model surpasses the state-of-the-art methods in all metrics following the dataset splits of previous work. The proposed approach is also evaluated by the evaluation server of MS COCO captioning challenge, and achieves very competitive results, e.g., a CIDEr of 1.029 (c5) and 1.064 (c40).","doi":"10.24963/ijcai.2017/563","url":"https://ijcai.org/proceedings/2017/563","pdf":"https://ijcai.org/proceedings/2017/0563.pdf","markdown":"https://ijcai.org/proceedings/2017/563.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/563","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Natural Language Processing","track":"Main track","number":563,"pages":{"first":4033,"last":4039},"keywords":["Machine Learning: Deep Learning","Natural Language Processing: Natural Language Generation"],"videos":[],"references":[{"position":1,"raw":"GT: A cat that is eating some kind of banana. B1: A couple of cats laying on top of a bed.. B2: A cat laying next to a pair of scissors. MAT: A close up of a cat eating a banana. GT: A large white dog is sitting on a bench beside an elderly man. B1: A dog standing next to a person on a skateboard. B2: A couple of dogs sitting on a bench. MAT: A man sitting on a bench with a dog. GT: A baseball player preparing to throw a pitch during a game. B1: A baseball player swinging a bat at a ball. B2: A baseball player standing on top of a field. MAT: A baseball player pitching a ball on a field. GT: A man on a bicycle riding next to a train. B1: A man standing in front of a train station. B2: A person riding a bike next to a train. MAT: A man riding a bike next to a train. GT: Two kayaks one pink the other yellow on bank of water. B1: A yellow surfboard sitting on top of a sandy beach. B2: A couple of boats parked next to each other. MAT: Two boats are docked in a body of water. References","title":"of a cat eating a banana. GT: A large white dog is sitting on a bench beside an elderly man. B1: A dog standing next to a person on a skateboard. B2: A couple of dogs sitting on a bench","authors":"Gt","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anderson et al., 2016] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In ECCV, 2016.","title":"Spice: Semantic propositional image caption evaluation","authors":"Peter Anderson, Basura Fernando, Mark Johnson, Stephen Gould","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. ICLR, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Banerjee and Lavie, 2005] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In ACL workshop, 2005.","title":"Meteor: An automatic metric for mt evaluation with improved correlation with human judgments","authors":"Lavie Banerjee, Satanjeev Banerjee, Alon Lavie","year":"2005","venue":"ACL workshop","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen and Lawrence Zitnick, 2015] Xinlei Chen and C. Lawrence Zitnick. Mind's eye: A recurrent visual representation for image caption generation. In CVPR, 2015.","title":"Mind's eye: A recurrent visual representation for image caption generation","authors":"Chen, Lawrence Zitnick, ; Xinlei, Chen, C Lawrence Zitnick","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Devlin et al., 2015] Jacob Devlin, Hao Cheng, Hao Fang, and et al. Language models for image captioning: The quirks and what works. In ACL, 2015.","title":"Language models for image captioning: The quirks and what works","authors":"Jacob Devlin, Hao Cheng, Hao Fang","year":"2015","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Donahue et al., 2015] Jeffrey Donahue, Lisa Anne Hen- dricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long- term recurrent convolutional networks for visual recogni- tion and description. In CVPR, 2015.","title":"Longterm recurrent convolutional networks for visual recognition and description","authors":"Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, Trevor Darrell","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Elliott and Keller, 2013] Desmond Elliott and Frank Keller. Image description using visual dependency representa- tions. In EMNLP, 2013.","title":"Image description using visual dependency representations","authors":"Desmond Elliott, Frank Keller","year":"2013","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fang et al., 2015] Hao Fang, Saurabh Gupta, Forrest Ian- dola, and et al. From captions to visual concepts and back. In CVPR, 2015.","title":"From captions to visual concepts and back","authors":"Saurabh Hao Fang, Forrest Gupta, Iandola","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gong et al., 2014] Yunchao Gong, Liwei Wang, Micah Ho- dosh, Julia Hockenmaier, and Svetlana Lazebnik. Improv- ing image-sentence embeddings using large weakly anno- tated photo collections. In ECCV. 2014.","title":"Improving image-sentence embeddings using large weakly annotated photo collections","authors":"Gong","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1997.","title":"Long short-term memory","authors":"Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hochreiter et al., 2001] Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, and Jürgen Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term de- pendencies, 2001.","title":"Gradient flow in recurrent nets: the difficulty of learning long-term dependencies","authors":"Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jifeng et al., 2016] Dai Jifeng, Yi Li, Kaiming He, and Jian Sun. R-fcn: Object detection via region-based fully con- volutional networks. In NIPS, 2016.","title":"R-fcn: Object detection via region-based fully convolutional networks","authors":"Dai Jifeng, Yi Li, Kaiming He, Jian Sun","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Karpathy and Fei-Fei, 2015] Andrej Karpathy and Li Fei- Fei. Deep visual-semantic alignments for generating im- age descriptions. In CVPR, 2015.","title":"Deep visual-semantic alignments for generating image descriptions","authors":"Fei-Fei ; Karpathy, Andrej Karpathy, Li Fei-Fei","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kuznetsova et al., 2014] Polina Kuznetsova, Vicente Or- donez, Tamara L Berg, and Yejin Choi. Treetalk: Com- position and compression of trees for image descriptions. ACL, 2014.","title":"Treetalk: Composition and compression of trees for image descriptions","authors":"Polina Kuznetsova, Vicente Ordonez, Tamara L Berg, Yejin Choi","year":"2014","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In ECCV. 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lin, 2004] Chin-Yew Lin. Rouge: A package for auto- matic evaluation of summaries. In Text summarization branches out: Proceedings of the ACL-04 workshop, vol- ume 8. Barcelona, Spain, 2004.","title":"Rouge: A package for automatic evaluation of summaries","authors":"Chin-Yew Lin","year":"2004","venue":"Text summarization branches out: Proceedings of the ACL-04 workshop","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2016] Chenxi Liu, Junhua Mao, Fei Sha, and Alan Yuille. Attention correctness in neural image cap- tioning. AAAI, 2016.","title":"Attention correctness in neural image captioning","authors":"Chenxi Liu, Junhua Mao, Fei Sha, Alan Yuille","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"Luong et al., ] Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention-based neural machine translation. In EMNLP.","title":"Effective approaches to attention-based neural machine translation","authors":"Luong","year":null,"venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mao et al., 2015] Junhua Mao, Xu Wei, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan L. Yuille. Learning like a child: Fast novel visual concept learning from sentence descriptions of images. In ICCV, 2015.","title":"Learning like a child: Fast novel visual concept learning from sentence descriptions of images","authors":"Junhua Mao, Xu Wei, Yi Yang, Jiang Wang, Zhiheng Huang, Alan L Yuille","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mitchell and et al., 2012] Margaret Mitchell and et al. Midge: Generating image descriptions from computer vi- sion detections. In EACL, 2012.","title":"Midge: Generating image descriptions from computer vision detections","authors":"Mitchell","year":"2012","venue":"EACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In ACL, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. Sequence to sequence learning with neu- ral networks. In NIPS, 2014.","title":"Quoc VV Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vedantam et al., 2015] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In CVPR, 2015.","title":"Cider: Consensusbased image description evaluation","authors":"C Lawrence Ramakrishna Vedantam, Devi Zitnick, Parikh","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Venugopalan et al., 2015] Subhashini Venugopalan, Mar- cus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence -video to text. In The IEEE International Conference on Computer Vision (ICCV), December 2015.","title":"Sequence to sequence -video to text","authors":"Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, Kate Saenko","year":"2015","venue":"The IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vinyals et al., 2015a] Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. Grammar as a foreign language. In NIPS, 2015.","title":"Grammar as a foreign language","authors":"Vinyals","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Vinyals et al., 2015b] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neu- ral image caption generator. In CVPR, 2015.","title":"Show and tell: A neural image caption generator","authors":"Vinyals","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wu et al., 2016] Qi Wu, Chunhua Shen, Lingqiao Liu, An- thony Dick, and Anton van den Hengel. What value do ex- plicit high level concepts have in vision to language prob- lems? In CVPR, 2016.","title":"What value do explicit high level concepts have in vision to language problems?","authors":"Qi Wu, Chunhua Shen, Lingqiao Liu, Anthony Dick, Anton Van Den, Hengel","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, and et al. Show, attend and tell: Neural image caption gen- eration with visual attention. In ICML, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yang et al., 2011] Yezhou Yang, Ching Lik Teo, Hal Daumé III, and Yiannis Aloimonos. Corpus-guided sen- tence generation of natural images. In EMNLP, 2011.","title":"Corpus-guided sentence generation of natural images","authors":"Yang","year":"2011","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang et al., 2016] Zhilin Yang, Ye Yuan, Yuexin Wu, William W Cohen, and Ruslan R Salakhutdinov. Review networks for caption generation. In NIPS, 2016.","title":"Review networks for caption generation","authors":"Zhilin Yang, Ye Yuan, Yuexin Wu, William W Cohen, Ruslan R Salakhutdinov","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[You et al., 2016] Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In CVPR, 2016.","title":"Image captioning with semantic attention","authors":"You","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}