{"type":"ScholarlyArticle","title":"An Attention-based Regression Model for Grounding Textual Phrases in Images","authors":[{"name":"Ko Endo","slug":"ko-endo","given":"Ko","family":"Endo","affiliation":"Toyohashi University of Technology"},{"name":"Masaki Aono","slug":"masaki-aono","given":"Masaki","family":"Aono","affiliation":"Toyohashi University of Technology"},{"name":"Eric Nichols","slug":"eric-nichols","given":"Eric","family":"Nichols","affiliation":"Honda Research Institute Japan"},{"name":"Kotaro Funakoshi","slug":"kotaro-funakoshi","given":"Kotaro","family":"Funakoshi","affiliation":"Honda Research Institute Japan"}],"abstract":"Grounding, or localizing, a textual phrase in an image is a challenging problem that is integral to visual language understanding. Previous approaches to this task typically make use of candidate region proposals, where end performance depends on that of the region proposal method and additional computational costs are incurred. In this paper, we treat grounding as a regression problem and propose a method to directly identify the region referred to by a textual phrase, eliminating the need for external candidate region prediction. Our approach uses deep neural networks to combine image and text representations and refines the target region with attention models over both image subregions and words in the textual phrase. Despite the challenging nature of this task and sparsity of available data, in evaluation on the ReferIt dataset, our proposed method achieves a new state-of-the-art in performance of 37.26% accuracy, surpassing the previously reported best by over 5 percentage points. We find that combining image and text attention models and an image attention area-sensitive loss function contribute to substantial improvements.","doi":"10.24963/ijcai.2017/558","url":"https://ijcai.org/proceedings/2017/558","pdf":"https://ijcai.org/proceedings/2017/0558.pdf","markdown":"https://ijcai.org/proceedings/2017/558.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/558","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Natural Language Processing","track":"Main track","number":558,"pages":{"first":3995,"last":4001},"keywords":["Natural Language Processing: Natural Language Semantics","Natural Language Processing: Information Retrieval","Machine Learning: Deep Learning","Robotics and Vision: Robotics and Vision"],"videos":[],"references":[{"position":1,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Ji- asen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question An- swering. In International Conference on Computer Vision (ICCV), 2015.","title":"VQA: Visual Question Answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh","year":"2015","venue":"International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. International Conference on Learning Representationaa(ICLR), 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"International Conference on Learning Representationaa","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Blaschko and Lampert, 2008] Matthew B. Blaschko and Christoph H. Lampert. Learning to localize objects with structured output regression. In Proceedings of the 10th European Conference on Computer Vision: Part I, ECCV '08, pages 2-15, Berlin, Heidelberg, 2008. Springer- Verlag.","title":"Learning to localize objects with structured output regression","authors":"Matthew B Blaschko, Christoph H Lampert","year":"2008","venue":"Proceedings of the 10th European Conference on Computer Vision: Part I, ECCV '08","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Escalante et al., 2010] Hugo Jair Escalante, Carlos A. Hernández, Jesus A. Gonzalez, A. López-López, Manuel Montes, Eduardo F. Morales, L. Enrique Sucar, Luis Vil- laseñor, and Michael Grubinger. The segmented and an- notated IAPR TC-12 benchmark. Comput. Vis. Image Un- derst., 114(4):419-428, April 2010.","title":"The segmented and annotated IAPR TC-12 benchmark","authors":"Escalante","year":"2010","venue":"Comput. Vis. Image Underst","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Everingham et al., 2005] Mark Everingham, Andrew Zis- serman, Chris Williams, Luc Van Gool, and Moray Al- lan. The 2005 PASCAL visual object classes challenge. First PASCAL Machine Learning Challenges Workshop (MLCW 05), pages 117-176, 2005.","title":"The 2005 PASCAL visual object classes challenge","authors":"Mark Everingham, Andrew Zisserman, Chris Williams, Luc Van Gool, Moray Allan","year":"2005","venue":"First PASCAL Machine Learning Challenges Workshop (MLCW 05","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fukui et al., 2016] Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Mar- cus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv:1606.01847, 2016.","title":"Multimodal compact bilinear pooling for visual question answering and visual grounding","authors":"Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, Marcus Rohrbach","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Grubinger et al., 2006] M. Grubinger, P. Clough, H. Muller, and T. Deselaers. The IAPR TC-12 benchmark: A new evaluation resource for visual information systems. In In- ternational Workshop OntoImage, 2006.","title":"The IAPR TC-12 benchmark: A new evaluation resource for visual information systems","authors":"M Grubinger, P Clough, H Muller, T Deselaers","year":"2006","venue":"International Workshop OntoImage","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2016a] Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. Segmentation from natural language ex- pressions. Proceedings of the European Conference on Computer Vision (ECCV), 2016.","title":"Segmentation from natural language expressions","authors":"Hu","year":"2016","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2016b] Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. Natural language object retrieval. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.","title":"Natural language object retrieval","authors":"Hu","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jaderberg et al., 2016] Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Reading text in the wild with convolutional neural networks. International Journal on Computer Vision(IJCV), 2016.","title":"Reading text in the wild with convolutional neural networks","authors":"Jaderberg","year":"2016","venue":"International Journal on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kazemzadeh et al., 2014] Sahar Kazemzadeh, Vicente Or- donez, Mark Matten, and Tamara L. Berg. ReferIt game: Referring to objects in photographs of natural scenes. In EMNLP, 2014.","title":"ReferIt game: Referring to objects in photographs of natural scenes","authors":"Kazemzadeh","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kingma and Ba, 2014] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kumar et al., 2015] Ankit Kumar, Ozan Irsoy, Jonathan Su, James Bradbury, Robert English, Brian Pierce, Peter On- druska, Ishaan Gulrajani, and Richard Socher. Ask me anything: Dynamic memory networks for natural language processing. CoRR, abs/1506.07285, 2015.","title":"Ask me anything: Dynamic memory networks for natural language processing","authors":"Ankit Kumar, Ozan Irsoy, Jonathan Su, James Bradbury, Robert English, Brian Pierce, Peter Ondruska, Ishaan Gulrajani, Richard Socher","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Com- mon objects in context. In European Conference on Com- puter Vision, pages 740-755. Springer, 2014.","title":"Microsoft COCO: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lu et al., 2016] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. CoRR, abs/1606.00061, 2016.","title":"Hierarchical question-image co-attention for visual question answering","authors":"Lu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Luo and Shakhnarovich, 2017] Ruotian Luo and Gregory Shakhnarovich. Comprehension-guided referring expres- sions. arXiv:1071.03439, 2017.","title":"Ruotian Luo and Gregory Shakhnarovich. Comprehension-guided referring expressions","authors":"Shakhnarovich Luo","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Infor- mation Processing Systems 26, pages 3111-3119.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Curran Associates, Inc., 2013.","title":null,"authors":"Curran Associates, Inc","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. GloVe: Global vectors for word representation. In Empirical Meth- ods in Natural Language Processing (EMNLP), pages 1532-1543, 2014.","title":"GloVe: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rohrbach et al., 2016] A. Rohrbach, M. Rohrbach, R. Hu, T. Darrell, and Bernt Schiele. Grounding of textual phrases in images by reconstruction. In ECCV2016, 2016.","title":"Grounding of textual phrases in images by reconstruction","authors":"A Rohrbach, M Rohrbach, R Hu, T Darrell, Bernt Schiele","year":"2016","venue":"ECCV2016","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Simonyan and Zisserman, 2014] K. Simonyan and A. Zis- serman. Very deep convolutional networks for large-scale image recognition. arXiv.org, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"K Simonyan, A Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual atten- tion. arXiv preprint arXiv:1502.03044, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2016] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention net- works for image question answering. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.","title":"Stacked attention networks for image question answering","authors":"Yang","year":"2016","venue":"The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zitnick and Dollár, 2014] C. Lawrence Zitnick and Piotr Dollár. Edge boxes: Locating object proposals from edges. In ECCV, 2014.","title":"Edge boxes: Locating object proposals from edges","authors":"C, Lawrence Zitnick, Piotr Dollár","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}