{"type":"ScholarlyArticle","title":"Position Focused Attention Network for Image-Text Matching","authors":[{"name":"Yaxiong Wang","slug":"yaxiong-wang","given":"Yaxiong","family":"Wang","affiliation":""},{"name":"Hao Yang","slug":"hao-yang","given":"Hao","family":"Yang","affiliation":""},{"name":"Xueming Qian","slug":"xueming-qian","given":"Xueming","family":"Qian","affiliation":""},{"name":"Lin Ma","slug":"lin-ma","given":"Lin","family":"Ma","affiliation":""},{"name":"Jing Lu","slug":"jing-lu","given":"Jing","family":"Lu","affiliation":""},{"name":"Biao Li","slug":"biao-li","given":"Biao","family":"Li","affiliation":""},{"name":"Xin Fan","slug":"xin-fan","given":"Xin","family":"Fan","affiliation":""}],"abstract":"Image-text matching tasks\r\nhave recently attracted a lot of attention in the computer vision field. The\r\nkey point of this cross-domain problem is how to accurately measure the\r\nsimilarity between the visual and the textual contents, which demands a fine\r\nunderstanding of both modalities. In this paper, we propose a novel position\r\nfocused attention network (PFAN) to investigate the relation between the visual\r\nand the textual views. In this work, we integrate the object position clue to\r\nenhance the visual-text joint-embedding learning. We first split the images into blocks, by which we\r\ninfer the relative position of region in the image. Then, an attention\r\nmechanism is proposed to model the relations between the image region and\r\nblocks and generate the valuable position feature, which will be further\r\nutilized to enhance the region expression and model a more reliable\r\nrelationship between the visual image and the textual sentence. Experiments\r\non the popular datasets Flickr30K and MS-COCO show the effectiveness of the\r\nproposed method. Besides the public datasets, we also conduct experiments on\r\nour collected practical news dataset (Tencent-News) to validate the practical\r\napplication value of proposed method. As far as we know, this is the first\r\nattempt to test the performance on the practical application. Our method can achieve\r\nthe state-of-art performance on all of these three datasets.","doi":"10.24963/ijcai.2019/526","url":"https://ijcai.org/proceedings/2019/526","pdf":"https://ijcai.org/proceedings/2019/0526.pdf","markdown":"https://ijcai.org/proceedings/2019/526.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/526","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":526,"pages":{"first":3792,"last":3798},"keywords":["Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Machine Learning: Deep Learning","Computer Vision: Language and Vision"],"videos":[],"references":[{"position":1,"raw":"[Plummer et al., 2018] Bryan Plummer, Paige Kordas, M Kiapour, Shuai Zheng, Robinso Piramuthu, and Svetlana Lazebnik. Conditional Image-Text Embedding Networks. In ECCV, pages 258-274, 2018.","title":"Conditional Image-Text Embedding Networks","authors":"Bryan Plummer, Paige Kordas, M Kiapour, Shuai Zheng, Robinso Piramuthu, Svetlana Lazebnik","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Lee et al., 2018] Kuang-Huei Lee, Xi Chen, Gang, Hua, Houdong Hu, and Xiaodong He. Stacked Cross Attention for Image-Text Matching. In ECCV, pages 212-218, 2018.","title":"Stacked Cross Attention for Image-Text Matching","authors":"Kuang-Huei Lee, Xi Chen, Gang, Houdong Hua, Xiaodong Hu, He","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Hu et al., 2016] Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. Natural language object retrieval. In CVPR, 2016.","title":"Natural language object retrieval","authors":"Hu","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gu et al., 2018] Jiuxiang Gu, Jianfei Cai, Shafiq R.Joty, Li Niu, and Gang Wang. Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models. In CVPR, 2018.","title":"Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models","authors":"Jiuxiang Gu, Jianfei Cai, R Shafiq, Li Joty, Gang Niu, Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Yan et al., 2015] Fei Yan, and Krystian Mikolajczyk. Deep Correlation for Matching Images and Text. In CVPR, 2015.","title":"Deep Correlation for Matching Images and Text","authors":"Fei Yan, Krystian Mikolajczyk","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Eisenschtat et al., 2017] Aviv Eisenschtat, and Lior Wolf. Linking Image and Text with 2-way Nets. In CVPR, 2017.","title":"Linking Image and Text with 2-way Nets","authors":"Eisenschtat","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Xu, et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courvile, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, pp. 2048-2057, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courvile, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kiros et al., 2014] Ryan Kiros, Ruslan Salakhutdinov, and Richard Zemel. Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models. In arXiv/1141.2539, 2014.","title":"Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models","authors":"Ryan Kiros, Ruslan Salakhutdinov, Richard Zemel","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Vinyals, et al., 2017] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: Lessons learned from the 2015 MSCOCO image captioning challenge. IEEE TPAMI, 39(4):652-663, 2017.","title":"Show and tell: Lessons learned from the 2015 MSCOCO image captioning challenge","authors":"Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan","year":"2017","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ma et al., 2015] Lin Ma, Zhengdong Lu, Lifeng Shang, and Hang Li. Multimodal convolutional neural networks for matching image and sentence. In ICCV, 2015.","title":"Multimodal convolutional neural networks for matching image and sentence","authors":"Lin Ma, Zhengdong Lu, Lifeng Shang, Hang Li","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Figure. 6: The visualization figures of attending image region to each word Zitnick, and Devi Parikh. VQA: visual question answering. In ICCV, pages 2425-2433, 2015.","title":"Lawrence Figure. 6: The visualization figures of attending image region to each word Zitnick, and Devi Parikh. VQA: visual question answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Chang et al., 2018] Xiaobin Chang, Tao Xiang, and Timothy Hospedales. Scalable and Effective Deep CCA via Soft Decorrelation. In CVPR, pages 1488-1497, 2018.","title":"Scalable and Effective Deep CCA via Soft Decorrelation","authors":"Xiaobin Chang, Tao Xiang, Timothy Hospedales","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Wang et al., 2018] Liwei Wang, Yin Li, Jing Huang, and Svetlana Lazbnik. Learning Two-Branch Neural Networks for Image-Text Matching Tasks. IEEE TPAMI, 41(2):394-407, 2018.","title":"Learning Two-Branch Neural Networks for Image-Text Matching Tasks","authors":"Liwei Wang, Yin Li, Jing Huang, Svetlana Lazbnik","year":"2018","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Klein et al., 2015] Benjamin Klein, Guy Lev, Gil Sadeh, and Lior Wolf. Fisher vectors derived from hybrid gaussian- laplacian mixture models for image annotation. In CVPR, 2015.","title":"Fisher vectors derived from hybrid gaussianlaplacian mixture models for image annotation","authors":"Benjamin Klein, Guy Lev, Gil Sadeh, Lior Wolf","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2016] Xiao Lin, and Devi Parikh. Leveraging visual question answering for image-caption rank. In ECCV, pages 261-277, 2016.","title":"Leveraging visual question answering for image-caption rank","authors":"Xiao Lin, Devi Parikh","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Zhang et al., 2018] Ying Zhang, and Huchuan Lu. Deep Cross-Modal Projection Learning for Image-Text Matching. In ECCV, pages 707-723, 2018.","title":"Deep Cross-Modal Projection Learning for Image-Text Matching","authors":"Ying Zhang, Huchuan Lu","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nam et al., 2017] Hyeonseob Nam, Jung-Woo Ha, and Jeonghee Kim. Dual Attention Networks for Multimodal Reasoning and Matching. In CVPR, pages. 2156-2164, 2017.","title":"Dual Attention Networks for Multimodal Reasoning and Matching","authors":"Nam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Huang et al., 2018] Yan Huang, Qi Wu, Chunfeng Song, and Liang Wang. Learning Semantic Concepts and Order for Image and Sentence Matching. In CVPR, 2018.","title":"Learning Semantic Concepts and Order for Image and Sentence Matching","authors":"Yan Huang, Qi Wu, Chunfeng Song, Liang Wang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zheng et al., 2018] Zhedong Zheng, Liang Zheng, Michael Garrett, Yi Yang, and Yi-Dong Shen. Dual-Path Convolutional Image-Text Embedding with Instance Loss. In CVPR, 2018.","title":"Dual-Path Convolutional Image-Text Embedding with Instance Loss","authors":"Zheng","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Faghri et al., 2018] Fartash Faghri, David Fleet, Jamie Kiros, and Sanja Fidler. VSE++: Improving Visual-Semantic Embeddings with Hard Negatives. In BMVC, 2018.","title":"VSE++: Improving Visual-Semantic Embeddings with Hard Negatives","authors":"Faghri","year":"2018","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Huang et al., 2017] Yan Huang, Wei Wang, and Liang Wang. Instance-Aware Image and Sentence Matching with Selection Multimodal LSTM. In CVPR, pages 7254- 7262, 2017.","title":"Instance-Aware Image and Sentence Matching with Selection Multimodal LSTM","authors":"Yan Huang, Wei Wang, Liang Wang","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Niu et al.,2017] Zhenxing Niu, Mo Zhou, Le Wang, Xinbo Gao, and Gang Hua. Hierarchical Multimodal LSTM for Dense Visual-Semantic Embedding. In ICCV, pages. 1899-1907, 2017.","title":"Hierarchical Multimodal LSTM for Dense Visual-Semantic Embedding","authors":"Zhenxing Niu, Mo Zhou, Le Wang, Xinbo Gao, Gang Hua","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Karpathy et al., 2015] Andrej Karpathy, and Li Fei-Fei. Deep Visual-Semantic Alignments for Generating image descriptions. In CVPR, pages 3128-3138, 2015.","title":"Deep Visual-Semantic Alignments for Generating image descriptions","authors":"Andrej Karpathy, Li Fei-Fei","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Anderson et al., 2018] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lie Zhang. Bottom-Up and Top-Down Attention for Image Caption and VQA. In CVPR, 2018.","title":"Bottom-Up and Top-Down Attention for Image Caption and VQA","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lie Zhang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ma et al., 2019] Lin Ma, Wenhao Jiang, Zequn Jie, Yugang Jiang, and Wei Liu. Matching Image and Sentence with Multi-faceted Representation. early access, IEEE TCSVT, 2019.","title":"Matching Image and Sentence with Multi-faceted Representation. early access","authors":"Lin Ma, Wenhao Jiang, Zequn Jie, Yugang Jiang, Wei Liu","year":"2019","venue":"IEEE TCSVT","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2018] Shuhui Wang, Yangyu Chen, Junbao Zhuo, Qingming Huang, and Qi Tian. Joint Global and Co-Attentive Representation Learning for Image- Sentence Retrieval. In ACM Multimedia, pages 1398- 1406, 2018.","title":"Joint Global and Co-Attentive Representation Learning for Image-Sentence Retrieval","authors":"Shuhui Wang, Yangyu Chen, Junbao Zhuo, Qingming Huang, Qi Tian","year":"2018","venue":"ACM Multimedia","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ma et al., 2019] Lin Ma, Wenhao Jiang, Zequn Jie, and Xu Wang. Bidirectional image-sentence retrieval by local and global deep matching. Neurocomputing, 345:36-44, 2019.","title":"Bidirectional image-sentence retrieval by local and global deep matching","authors":"Lin Ma, Wenhao Jiang, Zequn Jie, Xu Wang","year":"2019","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Song et al., 2017] Guoli Song, Shuhui Wang, Qingming Huang, and Qi Tian. Multimodal Similarity Gaussian Process Latent Variable Model. IEEE TIP, 26(9):4168- 4181, 2017.","title":"Multimodal Similarity Gaussian Process Latent Variable Model","authors":"Guoli Song, Shuhui Wang, Qingming Huang, Qi Tian","year":"2017","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2016] Zihao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alexander Smola. Stacked Attention Networks for Image Question Answering. In CVPR, pages 21-29, 2016.","title":"Stacked Attention Networks for Image Question Answering","authors":"Yang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Shih et al., 2016] Kevin Shih, Saurabh Singh, and Derek Hoiem. Where to look: Focus regions for visual question answering. In CVPR, pages 4613-4621, 2016.","title":"Where to look: Focus regions for visual question answering","authors":"Kevin Shih, Saurabh Singh, Derek Hoiem","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, pages 770-778, 2016.","title":"Deep Residual Learning for Image Recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Ren et al., 2017] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE TPAMI, 39(6):1137-1149, 2017.","title":"Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks","authors":"Ren","year":"2017","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Krishna et al., 2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image. In IJCV, 123(1): 32-73, 2017.","title":"Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image","authors":"Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, Michael Bernstein, Li Fei-Fei","year":"2017","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Kingma et al., 2015] Diederik Kingma, and Jimmy Ba. Adam: A Method for Stochastic Optimization. In ICLR, 2015.","title":"Adam: A Method for Stochastic Optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Qian et al., 2017] Xueming. Qian, Dan. Lu, Yaxiong. Wang, Li. Zhu, Yuanyan Tang, and Meng. Wang. Image Re- Ranking Based on Topic Diversity. IEEE TIP, 26(8):2724-2747, 2017.","title":"Image Re-Ranking Based on Topic Diversity","authors":"Xueming, Dan Qian, Yaxiong Lu, Li Wang, Yuanyan Zhu, Meng Tang, Wang","year":"2017","venue":"IEEE TIP","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wang et al., 2018] Yaxiong Wang, Li Zhu, Xueming Qian, and Junwei. Han. Joint Hypergraph Learning for Tag- Based Image Retrieval. IEEE Trans on Image Processing, 27(9): 4437-4451, 2018.","title":"Joint Hypergraph Learning for Tag-Based Image Retrieval","authors":"Wang","year":"2018","venue":"IEEE Trans on Image Processing","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Kim et al., 2017] Jin-Hwa Kim, Kyoung Woon On, Woosang Lim, Jeonghee Kim, JungWoo Ha, and Byoung-Tak Zhang. Hadamard Product for Low-Rank Bilinear Pooling. In ICLR, 2017.","title":"Hadamard Product for Low-Rank Bilinear Pooling","authors":"Jin-Hwa Kim, Kyoung Woon On, Woosang Lim, Jeonghee Kim, Jungwoo Ha, Byoung-Tak Zhang","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"Zhu et al. 2017] Linchao Zhu, Zhongwen Xu, and Yi Yang. Bidirectional multirate reconstruction for temporal modeling in videos. In CVPR, pages 1339-1348, 2017.","title":"Bidirectional multirate reconstruction for temporal modeling in videos","authors":"Zhu","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhao et al., 2017] Handong Zhao, Zhengming Ding, and Yun Fu. Multi-View Clustering via Deep Matrix Factorization. In AAAI, pages 2921-2927, 2017.","title":"Multi-View Clustering via Deep Matrix Factorization","authors":"Handong Zhao, Zhengming Ding, Yun Fu","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}