{"type":"ScholarlyArticle","title":"Image Captioning with Compositional Neural Module Networks","authors":[{"name":"Junjiao Tian","slug":"junjiao-tian","given":"Junjiao","family":"Tian","affiliation":"Carnegie Mellon University"},{"name":"Jean Oh","slug":"jean-oh","given":"Jean","family":"Oh","affiliation":"Carnegie Mellon University"}],"abstract":"In image captioning where fluency is an important factor in evaluation, n-gram metrics, sequential models are commonly used; however, sequential models generally result in overgeneralized expressions that lack the details that may be present in an input image. Inspired by the idea of the compositional neural module networks in the visual question answering task, we introduce a hierarchical framework for image captioning that explores both compositionality and sequentiality of natural language. Our algorithm learns to compose a detail-rich sentence by selectively attending to different modules corresponding to unique aspects of each object detected in an input image to include specific descriptions such as counts and color. In a set of experiments on the MSCOCO dataset, the proposed model outperforms a state-of-the art model across multiple evaluation metrics, more importantly, presenting visually interpretable results. Furthermore, the breakdown of subcategories f-scores of the SPICE metric and human evaluation on Amazon Mechanical Turk  show that our compositional module networks effectively generate  accurate and detailed captions.","doi":"10.24963/ijcai.2019/496","url":"https://ijcai.org/proceedings/2019/496","pdf":"https://ijcai.org/proceedings/2019/0496.pdf","markdown":"https://ijcai.org/proceedings/2019/496.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/496","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":496,"pages":{"first":3576,"last":3584},"keywords":["Computer Vision: Language and Vision","Machine Learning: Interpretability","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Anderson et al., 2016] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In European Con- ference on Computer Vision, pages 382-398. Springer, 2016.","title":"Spice: Semantic propositional image caption evaluation","authors":"Anderson","year":"2016","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anderson et al., 2017] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for im- age captioning and vqa. arXiv preprint arXiv:1707.07998, 2017.","title":"Bottom-up and top-down attention for image captioning and vqa","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Andreas et al., 2016] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learning to compose neural networks for question answering. arXiv preprint arXiv:1601.01705, 2016.","title":"Learning to compose neural networks for question answering","authors":"Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Dan Klein","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dai et al., 2017] Bo Dai, Yuqi Zhang, and Dahua Lin. De- tecting visual relationships with deep relational networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3298-3308. IEEE, 2017.","title":"Detecting visual relationships with deep relational networks","authors":"Bo Dai, Yuqi Zhang, Dahua Lin","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dai et al., 2018] Bo Dai, Sanja Fidler, and Dahua Lin. A neural compositional paradigm for image captioning. In Advances in Neural Information Processing Systems, pages 656-666, 2018.","title":"A neural compositional paradigm for image captioning","authors":"Bo Dai, Sanja Fidler, Dahua Lin","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Denkowski and Lavie, 2014] Michael Denkowski and Alon Lavie. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the ninth workshop on statistical machine translation, pages 376-380, 2014.","title":"Meteor universal: Language specific translation evaluation for any target language","authors":"Lavie Denkowski, Michael Denkowski, Alon Lavie","year":"2014","venue":"Proceedings of the ninth workshop on statistical machine translation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fang et al., 2015] Hao Fang, Saurabh Gupta, Forrest Ian- dola, Rupesh K Srivastava, Li Deng, Piotr Dollár, Jian- feng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. From captions to visual concepts and back. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 1473-1482, 2015.","title":"From captions to visual concepts and back","authors":"Saurabh Hao Fang, Forrest Gupta, Iandola, K Rupesh, Li Srivastava, Piotr Deng, Jianfeng Dollár, Xiaodong Gao, Margaret He, John C Mitchell, Platt","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ilse et al., 2018] Maximilian Ilse, Jakub M Tomczak, and Max Welling. Attention-based deep multiple instance learning. arXiv preprint arXiv:1802.04712, 2018.","title":"Attention-based deep multiple instance learning","authors":"Maximilian Ilse, Jakub M Tomczak, Max Welling","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Karpathy and Fei-Fei, 2015] Andrej Karpathy and Li Fei- Fei. Deep visual-semantic alignments for generating im- age descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128- 3137, 2015.","title":"Deep visual-semantic alignments for generating image descriptions","authors":"Fei-Fei ; Karpathy, Andrej Karpathy, Li Fei-Fei","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krishna et al., 2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Inter- national Journal of Computer Vision, 123(1):32-73, 2017.","title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","authors":"Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma","year":"2017","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Com- mon objects in context. In European conference on com- puter vision, pages 740-755. Springer, 2014.","title":"Microsoft COCO: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin, 2004] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out, 2004.","title":"Rouge: A package for automatic evaluation of summaries","authors":"Lin, Chin-Yew Lin","year":"2004","venue":"Text Summarization Branches Out","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lu et al., 2017] Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 375-383, 2017.","title":"Knowing when to look: Adaptive attention via a visual sentinel for image captioning","authors":"Lu","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for auto- matic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics, 2002.","title":"Bleu: a method for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu","year":"2002","venue":"Proceedings of the 40th annual meeting on association for computational linguistics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher Manning","year":"2014","venue":"Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Ren et al., ] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural in- formation processing systems, pages 91-99.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":null,"venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vedantam et al., 2015] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015.","title":"Cider: Consensusbased image description evaluation","authors":"Ramakrishna Vedantam, Lawrence Zitnick, Devi Parikh","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on, pages 3156-3164. IEEE, 2015.","title":"Show and tell: A neural image caption generator","authors":"Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan","year":"2015","venue":"Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2017] Yufei Wang, Zhe Lin, Xiaohui Shen, Scott Cohen, and Garrison W Cottrell. Skeleton key: Im- age captioning by skeleton-attribute decomposition. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, pages 7378-7387. IEEE, 2017.","title":"Skeleton key: Image captioning by skeleton-attribute decomposition","authors":"Yufei Wang, Zhe Lin, Xiaohui Shen, Scott Cohen, Garrison, Cottrell","year":"2017","venue":"Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yao et al., 2017] Ting Yao, Yingwei Pan, Yehao Li, Zhao- fan Qiu, and Tao Mei. Boosting image captioning with attributes. In Proceedings of the IEEE International Con- ference on Computer Vision, pages 4894-4902, 2017.","title":"Boosting image captioning with attributes","authors":"Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qiu, Tao Mei","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yao et al., 2018] Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Exploring visual relationship for image captioning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 684-699, 2018.","title":"Exploring visual relationship for image captioning","authors":"Ting Yao, Yingwei Pan, Yehao Li, Tao Mei","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[You et al., 2016] Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 4651-4659, 2016.","title":"Image captioning with semantic attention","authors":"You","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2006] Cha Zhang, John C. Platt, and Paul A. Viola. Multiple instance boosting for object detection. In Y. Weiss, B. Schölkopf, and J. C. Platt, editors, Advances in Neural Information Processing Systems, pages 1417- 1424. MIT Press, 2006.","title":"Multiple instance boosting for object detection","authors":"Cha Zhang, John C Platt, Paul A Viola","year":"2006","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}