{"type":"ScholarlyArticle","title":"Swell-and-Shrink: Decomposing Image Captioning by Transformation and Summarization","authors":[{"name":"Hanzhang Wang","slug":"hanzhang-wang","given":"Hanzhang","family":"Wang","affiliation":"Department of Computer Science and Technology , Tongji University , Shanghai , P. R. China"},{"name":"Hanli Wang","slug":"hanli-wang","given":"Hanli","family":"Wang","affiliation":"Department of Computer Science and Technology , Tongji University , Shanghai , P. R. China"},{"name":"Kaisheng Xu","slug":"kaisheng-xu","given":"Kaisheng","family":"Xu","affiliation":"Department of Computer Science and Technology , Tongji University , Shanghai , P. R. China"}],"abstract":"Image captioning is currently viewed as a problem analogous to machine translation. However, it always suffers from poor interpretability, coarse or even incorrect descriptions on regional details. Moreover, information abstraction and compression, as essential characteristics of captioning, are always overlooked and seldom discussed. To overcome the shortcomings, a swell-shrink method is proposed to redefine image captioning as a compositional task which consists of two separated modules: modality transformation and text compression. The former is guaranteed to accurately transform adequate visual content into textual form while the latter consists of a hierarchical LSTM which particularly emphasizes on removing the redundancy among multiple phrases and organizing the final abstractive caption. Additionally, the order and quality of region of interest and modality processing are studied to give insights of better understanding the influence of regional visual cues on language forming. Experiments demonstrate the effectiveness of the proposed method.","doi":"10.24963/ijcai.2019/726","url":"https://ijcai.org/proceedings/2019/726","pdf":"https://ijcai.org/proceedings/2019/0726.pdf","markdown":"https://ijcai.org/proceedings/2019/726.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/726","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":726,"pages":{"first":5226,"last":5232},"keywords":["Natural Language Processing: Natural Language Summarization","Computer Vision: Language and Vision"],"videos":[],"references":[{"position":1,"raw":"[Anderson et al., 2018] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang. Bottom- up and top-down attention for image captioning and visu- al question answering. In CVPR, pages 6077-6086, Jun. 2018.","title":"Bottomup and top-down attention for image captioning and visual question answering","authors":"P Anderson, X He, C Buehler, D Teney, M Johnson, S Gould, L Zhang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aneja et al., 2018] J. Aneja, A. Deshpande, and A. G Schwing. Convolutional image captioning. In CVPR, pages 5561-5570, Jun. 2018.","title":"Convolutional image captioning","authors":"J Aneja, A Deshpande, A G Schwing","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bahdanau et al., 2015] D. Bahdanau, K. Cho, and Y. Ben- gio. Neural machine translation by jointly learning to align and translate. In ICLR, May 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"D Bahdanau, K Cho, Y Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cheng and Lapata, 2016] J. Cheng and M. Lapata. Neural summarization by extracting sentences and words. In ACL, pages 484-494, Aug. 2016.","title":"Neural summarization by extracting sentences and words","authors":"J Cheng, M Lapata","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Donahue et al., 2015] J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional net- works for visual recognition and description. In CVPR, pages 2625-2634, Jun. 2015.","title":"Long-term recurrent convolutional networks for visual recognition and description","authors":"J Donahue, L Hendricks, S Guadarrama, M Rohrbach, S Venugopalan, K Saenko, T Darrell","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jin et al., 2017] J. Jin, K. Fu, R. Cui, F. Sha, and C. Zhang. Aligning where to see and what to tell: Image captioning with region-based attention and scene-specific contexts. IEEE Trans. PAMI, 39(12):2321-2334, Dec. 2017.","title":"Aligning where to see and what to tell: Image captioning with region-based attention and scene-specific contexts","authors":"J Jin, K Fu, R Cui, F Sha, C Zhang","year":"2017","venue":"IEEE Trans. PAMI","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Johnson et al., 2016] J. Johnson, A. Karpathy, and L. Fei- Fei. Densecap: Fully convolutional localization networks for dense captioning. In CVPR, pages 4565-4574, Jun. 2016.","title":"Densecap: Fully convolutional localization networks for dense captioning","authors":"J Johnson, A Karpathy, L Fei-Fei","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Karpathy and Fei-Fei, 2015] A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image de- scriptions. In CVPR, pages 3128-3137, Jun. 2015.","title":"Deep visual-semantic alignments for generating image descriptions","authors":"Fei-Fei Karpathy, ; A Karpathy, L Fei-Fei","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Krause et al., 2017] J. Krause, J. Johnson, R. Krishna, and L. Fei-Fei. A hierarchical approach for generating descrip- tive image paragraphs. In CVPR, pages 3337-3345, Jul. 2017.","title":"A hierarchical approach for generating descriptive image paragraphs","authors":"J Krause, J Johnson, R Krishna, L Fei-Fei","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krishna et al., 2017] R. Krishna, Y. Zhu, O. Groth, J. John- son, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L. Li, D. A. Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsourced dense image anno- tations. IJCV, 123(1):32-73, 2017.","title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","authors":"R Krishna, Y Zhu, O Groth, J Johnson, K Hata, J Kravitz, S Chen, Y Kalantidis, L Li, D A Shamma","year":"2017","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lavie and Denkowski, 2009] A. Lavie and M. J. Denkows- ki. The meteor metric for automatic evaluation of machine translation. Machine Translation, 23:105-115, Sep. 2009.","title":"The meteor metric for automatic evaluation of machine translation","authors":"A Lavie, M J Denkowski","year":"2009","venue":"Machine Translation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2017a] L. Li, S. Tang, L. Deng, Y. Zhang, and Q. Tian. Image caption with global-local attention. In AAAI, pages 4133-4139, Feb. 2017.","title":"Image caption with global-local attention","authors":"Li","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2017b] Y. Li, W. Ouyang, B. Zhou, K. Wang, and X. Wang. Scene graph generation from objects, phras- es and region captions. In ICCV, pages 1261-1270, Oct. 2017.","title":"Scene graph generation from objects, phrases and region captions","authors":"Li","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liang et al., 2017] X. Liang, Z. Hu, H. Zhang, C. Gan, and E. P. Xing. Recurrent topic-transition GAN for visual para- graph generation. In ICCV, pages 3362-3371, Oct. 2017.","title":"Recurrent topic-transition GAN for visual paragraph generation","authors":"Liang","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin and Och, 2004] C. Lin and F. J. Och. Automatic evalua- tion of machine translation quality using longest common subsequence and skip-bigram statistics. In ACL, pages 605-612, Jul. 2004.","title":"Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics","authors":"C Lin, F J Och","year":"2004","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2014] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Mi- crosoft COCO: Common objects in context. In ECCV, pages 740-755, Sep. 2014.","title":"Microsoft COCO: Common objects in context","authors":"T Lin, M Maire, S Belongie, J Hays, P Perona, D Ramanan, P Dollár, C L Zitnick","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Miao and Blunsom, 2016] Y. Miao and P. Blunsom. Lan- guage as a latent variable: Discrete generative models for sentence compression. In EMNLP, pages 319-328, Nov. 2016.","title":"Language as a latent variable: Discrete generative models for sentence compression","authors":"Y Miao, P Blunsom","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nallapati et al., 2016] R. Nallapati, B. Zhou, C. dos Santos, C. Gulcehre, and B. Xiang. Abstractive text summariza- tion using sequence-to-sequence RNNs and beyond. In SIGNLL CCNLL, pages 280-290, Aug. 2016.","title":"Abstractive text summarization using sequence-to-sequence RNNs and beyond","authors":"R Nallapati, B Zhou, C Santos, C Gulcehre, B Xiang","year":"2016","venue":"SIGNLL CCNLL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nallapati et al., 2017] R. Nallapati, F. Zhai, and B. Zhou. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In AAAI, pages 3075-3081, Feb. 2017.","title":"Summarunner: A recurrent neural network based sequence model for extractive summarization of documents","authors":"R Nallapati, F Zhai, B Zhou","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Papineni et al., 2002] K. Papineni, S. Roukos, T. Ward, and W. Zhu. BLEU: a method for automatic evaluation of ma- chine translation. In ACL, pages 311-318, Jul. 2002.","title":"BLEU: a method for automatic evaluation of machine translation","authors":"K Papineni, S Roukos, T Ward, W Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rush et al., 2015] A. M. Rush, S. Chopra, and J. Weston. A neural attention model for abstractive sentence summa- rization. In EMNLP, pages 379-389, Sep. 2015.","title":"A neural attention model for abstractive sentence summarization","authors":"A M Rush, S Chopra, J Weston","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vedantam et al., 2015] R. Vedantam, C. Lawrence Zitnick, and D. Parikh. CIDEr: Consensus-based image descrip- tion evaluation. In CVPR, pages 4566-4575, Jun. 2015.","title":"CIDEr: Consensus-based image description evaluation","authors":"R Vedantam, C Lawrence Zitnick, D Parikh","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vinyals et al., 2015] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption genera- tor. In CVPR, pages 3156-3164, Jun. 2015.","title":"Show and tell: A neural image caption generator","authors":"O Vinyals, A Toshev, S Bengio, D Erhan","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu and Hu, 2018] Y. Wu and B. Hu. Learning to extract co- herent summary via deep reinforcement learning. In AAAI, pages 5602-5609, Feb. 2018.","title":"Learning to extract coherent summary via deep reinforcement learning","authors":"Y Wu, B Hu","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wu et al., 2016] Q. Wu, C. Shen, A. Liu, L. Dick, and A. van den Hengel. What value do explicit high level con- cepts have in vision to language problems? In CVPR, pages 203-212, Jun. 2016.","title":"What value do explicit high level concepts have in vision to language problems?","authors":"Q Wu, C Shen, A Liu, L Dick, A Van Den, Hengel","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xu et al., 2015] K. Xu, J. Lei Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. S. Zemel, and Y. Ben- gio. Show, attend and tell: Neural image caption genera- tion with visual attention. In ICML, pages 2048-2057, Jul. 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"K Xu, J Lei, R Ba, K Kiros, A Cho, R Courville, R S Salakhudinov, Y Zemel, Bengio","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yang et al., 2017] L. Yang, K. D. Tang, J. Yang, and L. Li. Dense captioning with joint inference and visual context. In CVPR, pages 1978-1987, Jul. 2017.","title":"Dense captioning with joint inference and visual context","authors":"L Yang, K D Tang, J Yang, L Li","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yin and Pei, 2015] W. Yin and Y. Pei. Optimizing sentence modeling and selection for document summarization. In IJCAI, pages 1383-1389, Aug. 2015.","title":"Optimizing sentence modeling and selection for document summarization","authors":"W Yin, Y Pei","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/199"},{"position":29,"raw":"[Zeiler and Fergus, 2014] M.D. Zeiler and R. Fergus. Visu- alizing and understanding convolutional networks. In EC- CV, pages 818-833, Sep. 2014.","title":"Visualizing and understanding convolutional networks","authors":"M D Zeiler, R Fergus","year":"2014","venue":"EC-CV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}