{"type":"ScholarlyArticle","title":"A Multi-task Learning Approach for Image Captioning","authors":[{"name":"Wei Zhao","slug":"wei-zhao","given":"Wei","family":"Zhao","affiliation":"Shenzhen Institutes of Advanced Technology , Chinese Academy of Sciences"},{"name":"Benyou Wang","slug":"benyou-wang","given":"Benyou","family":"Wang","affiliation":"Tencent Inc."},{"name":"Jianbo Ye","slug":"jianbo-ye","given":"Jianbo","family":"Ye","affiliation":"Pennsylvania State University"},{"name":"Min Yang","slug":"min-yang","given":"Min","family":"Yang","affiliation":"Shenzhen Institutes of Advanced Technology , Chinese Academy of Sciences"},{"name":"Zhou Zhao","slug":"zhou-zhao","given":"Zhou","family":"Zhao","affiliation":"Zhejiang University"},{"name":"Ruotian Luo","slug":"ruotian-luo","given":"Ruotian","family":"Luo","affiliation":"Toyota Technological Institute at Chicago"},{"name":"Yu Qiao","slug":"yu-qiao","given":"Yu","family":"Qiao","affiliation":"Shenzhen Institutes of Advanced Technology , Chinese Academy of Sciences"}],"abstract":"In this paper, we propose a Multi-task Learning Approach for Image Captioning (MLAIC ), motivated by the fact that humans have no difficulty performing such task because they possess capabilities of multiple domains. Specifically, MLAIC consists of three key components: (i) A multi-object classification model that learns rich category-aware image representations using a CNN image encoder; (ii) A syntax generation model that learns better syntax-aware LSTM based decoder; (iii) An image captioning model that generates image descriptions in text, sharing its CNN encoder and LSTM decoder with the object classification task and the syntax generation task, respectively. In particular, the image captioning model can benefit from the additional object categorization and syntax knowledge. To verify the effectiveness of our approach, we conduct extensive experiments on MS-COCO dataset. The experimental results demonstrate that our model achieves impressive results compared to other strong competitors.","doi":"10.24963/ijcai.2018/168","url":"https://ijcai.org/proceedings/2018/168","pdf":"https://ijcai.org/proceedings/2018/0168.pdf","markdown":"https://ijcai.org/proceedings/2018/168.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/168","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":168,"pages":{"first":1205,"last":1211},"keywords":["Humans and AI: Cognitive Modeling","Machine Learning: Transfer, Adaptation, Multi-task Learning","Computer Vision: Language and Vision"],"videos":[],"references":[{"position":1,"raw":"[Anderson et al., 2017] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for im- age captioning and vqa. arXiv preprint arXiv:1707.07998, 2017.","title":"Bottom-up and top-down attention for image captioning and vqa","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bernardi et al., 2016] Raffaella Bernardi, Ruket Cakici, Desmond Elliott, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis, Frank Keller, Adrian Muscat, and Barbara Plank. Automatic description generation from images: A survey of models, datasets, and evaluation measures. JAIR, 55:409-442, 2016.","title":"Automatic description generation from images: A survey of models, datasets, and evaluation measures","authors":"Raffaella Bernardi, Ruket Cakici, Desmond Elliott, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis, Frank Keller, Adrian Muscat, Barbara Plank","year":"2016","venue":"JAIR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Caruana, 1998] Rich Caruana. Multitask learning. In Learning to learn, pages 95-133. Springer, 1998.","title":"Multitask learning","authors":"Rich Caruana","year":"1998","venue":"Learning to learn","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2017] Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, and Tat-Seng Chua. Sca- cnn: Spatial and channel-wise attention in convolutional networks for image captioning. In CVPR, 2017.","title":"Scacnn: Spatial and channel-wise attention in convolutional networks for image captioning","authors":"Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, Tat-Seng Chua","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gu et al., 2017a] Jiuxiang Gu, Jianfei Cai, Gang Wang, and Tsuhan Chen. Stack-captioning: Coarse-to-fine learning for image captioning. arXiv preprint arXiv:1709.03376, 2017.","title":"Stack-captioning: Coarse-to-fine learning for image captioning","authors":"Gu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gu et al., 2017b] Jiuxiang Gu, Gang Wang, Jianfei Cai, and Tsuhan Chen. An empirical study of language cnn for im- age captioning. In ICCV, 2017.","title":"An empirical study of language cnn for image captioning","authors":"Gu","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Karpathy and Fei-Fei, 2015] Andrej Karpathy and Li Fei- Fei. Deep visual-semantic alignments for generating im- age descriptions. In CVPR, pages 3128-3137, 2015.","title":"Deep visual-semantic alignments for generating image descriptions","authors":"Fei-Fei ; Karpathy, Andrej Karpathy, Li Fei-Fei","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2017] Yuncheng Li, Yale Song, and Jiebo Luo. Im- proving pairwise ranking for multi-label image classifica- tion. arXiv preprint arXiv:1704.03135, 2017.","title":"Improving pairwise ranking for multi-label image classification","authors":"Yuncheng Li, Yale Song, Jiebo Luo","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Pi- otr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2016] Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. Optimization of image description metrics using policy gradient methods. arXiv preprint arXiv:1612.00370, 2016.","title":"Optimization of image description metrics using policy gradient methods","authors":"Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, Kevin Murphy","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lu et al., 2016] Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. arXiv preprint arXiv:1612.01887, 2016.","title":"Knowing when to look: Adaptive attention via a visual sentinel for image captioning","authors":"Lu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Nadejde et al., 2017] Maria Nadejde, Siva Reddy, Rico Sennrich, Tomasz Dwojak, Marcin Junczys-Dowmunt, Philipp Koehn, and Alexandra Birch. Predicting target language ccg supertags improves neural machine transla- tion. In Proceedings of the Second Conference on Machine Translation, pages 68-79, 2017.","title":"Predicting target language ccg supertags improves neural machine translation","authors":"Maria Nadejde, Siva Reddy, Rico Sennrich, Tomasz Dwojak, Marcin Junczys-Dowmunt, Philipp Koehn, Alexandra Birch","year":"2017","venue":"Proceedings of the Second Conference on Machine Translation","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pasunuru and Bansal, 2017] Ramakanth Pasunuru and Mo- hit Bansal. Multi-task video captioning with video and entailment generation. arXiv preprint arXiv:1704.07489, 2017.","title":"Ramakanth Pasunuru and Mohit Bansal. Multi-task video captioning with video and entailment generation","authors":"Bansal Pasunuru","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pu et al., 2016] Yunchen Pu, Zhe Gan, Ricardo Henao, Xin Yuan, Chunyuan Li, Andrew Stevens, and Lawrence Carin. Variational autoencoder for deep learning of im- ages, labels and captions. In NIPS, pages 2352-2360, 2016.","title":"Variational autoencoder for deep learning of images, labels and captions","authors":"Pu","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rennie et al., 2016] Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. Self- critical sequence training for image captioning. arXiv preprint arXiv:1612.00563, 2016.","title":"Selfcritical sequence training for image captioning","authors":"Etienne Steven J Rennie, Youssef Marcheret, Jarret Mroueh, Vaibhava Ross, Goel","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vedantam et al., 2015] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In CVPR, pages 4566-4575, 2015.","title":"Cider: Consensusbased image description evaluation","authors":"Ramakrishna Vedantam, Lawrence Zitnick, Devi Parikh","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Venugopalan et al., 2016] Subhashini Venugopalan, Lisa Anne Hendricks, Raymond Mooney, and Kate Saenko. Improving lstm-based video description with linguistic knowledge mined from text. In arXiv preprint arXiv:1604.01729, 2016.","title":"Improving lstm-based video description with linguistic knowledge mined from text","authors":"Subhashini Venugopalan, Lisa Anne Hendricks, Raymond Mooney, Kate Saenko","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neu- ral image caption generator. In CVPR, pages 3156-3164, 2015.","title":"Show and tell: A neural image caption generator","authors":"Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vinyals et al., 2017] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: Lessons learned from the 2015 mscoco image captioning chal- lenge. TPAMI, 39(4):652-663, 2017.","title":"Show and tell: Lessons learned from the 2015 mscoco image captioning challenge","authors":"Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan","year":"2017","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine Learning, 8:229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2016] Qi Wu, Chunhua Shen, Lingqiao Liu, An- thony Dick, and Anton van den Hengel. What value do ex- plicit high level concepts have in vision to language prob- lems? In CVPR, pages 203-212, 2016.","title":"What value do explicit high level concepts have in vision to language problems?","authors":"Qi Wu, Chunhua Shen, Lingqiao Liu, Anthony Dick, Anton Van Den, Hengel","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, pages 2048-2057, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, Yoshua Bengio","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhao et al., 2017] Wei Zhao, Wei Xu, Min Yang, Jianbo Ye, Zhou Zhao, Yabing Feng, and Yu Qiao. Dual learning for cross-domain image captioning. In CIKM, pages 29-38, 2017.","title":"Dual learning for cross-domain image captioning","authors":"Wei Zhao, Wei Xu, Min Yang, Jianbo Ye, Zhou Zhao, Yabing Feng, Yu Qiao","year":"2017","venue":"CIKM","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}