{"type":"ScholarlyArticle","title":"Cross-media Multi-level Alignment with Relation Attention Network","authors":[{"name":"Jinwei Qi","slug":"jinwei-qi","given":"Jinwei","family":"Qi","affiliation":"Institute of Computer Science and Technology , Peking University , Beijing 100871 , China"},{"name":"Yuxin Peng","slug":"yuxin-peng","given":"Yuxin","family":"Peng","affiliation":"Institute of Computer Science and Technology , Peking University , Beijing 100871 , China"},{"name":"Yuxin Yuan","slug":"yuxin-yuan","given":"Yuxin","family":"Yuan","affiliation":"Institute of Computer Science and Technology , Peking University , Beijing 100871 , China"}],"abstract":"With the rapid growth of multimedia data, such as image and text, it is a highly challenging problem to effectively correlate and retrieve the data of different media types. Naturally, when correlating an image with textual description, people focus on not only the alignment between discriminative image regions and key words, but also the relations lying in the visual and textual context. Relation understanding is essential for cross-media correlation learning, which is ignored by prior cross-media retrieval works. To address the above issue, we propose Cross-media Relation Attention Network (CRAN) with multi-level alignment. First, we propose visual-language relation attention model to explore both fine-grained patches and their relations of different media types. We aim to not only exploit cross-media fine-grained local information, but also capture the intrinsic relation information, which can provide complementary hints for correlation learning. Second, we propose cross-media multi-level alignment to explore global, local and relation alignments across different media types, which can mutually boost to learn more precise cross-media correlation. We conduct experiments on 2 cross-media datasets, and compare with 10 state-of-the-art methods to verify the effectiveness of proposed approach.","doi":"10.24963/ijcai.2018/124","url":"https://ijcai.org/proceedings/2018/124","pdf":"https://ijcai.org/proceedings/2018/0124.pdf","markdown":"https://ijcai.org/proceedings/2018/124.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/124","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":124,"pages":{"first":892,"last":898},"keywords":["Machine Learning: Relational Learning","Computer Vision: Language and Vision"],"videos":[],"references":[{"position":1,"raw":"[Andrew et al., 2013] Galen Andrew, Raman Arora, Jeff A. Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International Conference on Machine Learn- ing (ICML), pages 1247-1255, 2013.","title":"Deep canonical correlation analysis","authors":"Galen Andrew, Raman Arora, Jeff A Bilmes, Karen Livescu","year":"2013","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Feng et al., 2014] Fangxiang Feng, Xiaojie Wang, and Ruifan Li. Cross-modal retrieval with correspondence autoencoder. In ACM Conference on Multimedia (ACM- MM), pages 7-16, 2014.","title":"Cross-modal retrieval with correspondence autoencoder","authors":"Feng","year":"2014","venue":"ACM Conference on Multimedia (ACM-MM)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Gong et al., 2014] Yunchao Gong, Qifa Ke, Michael Isard, and Svetlana Lazebnik. A multi-view embedding space for modeling internet images, tags, and their semantics. Inter- national Journal of Computer Vision (IJCV), 106(2):210- 233, 2014.","title":"A multi-view embedding space for modeling internet images, tags, and their semantics","authors":"Gong","year":"2014","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hardoon et al., 2004] David R. Hardoon, Sándor Szedmák, and John Shawe-Taylor. Canonical correlation analysis: An overview with application to learning methods. Neural Computation, 16(12):2639-2664, 2004.","title":"Canonical correlation analysis: An overview with application to learning methods","authors":"David R Hardoon, Sándor Szedmák, John Shawe-Taylor","year":"2004","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hotelling, 1936] Harold Hotelling. Relations between two sets of variates. Biometrika, pages 321-377, 1936.","title":"Harold Hotelling. Relations between two sets of variates","authors":"Hotelling","year":"1936","venue":"Biometrika","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Klein et al., 2015] Benjamin Klein, Guy Lev, Gil Sadeh, and Lior Wolf. Associating neural word embeddings with deep image representations using fisher vectors. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 4437-4446, 2015.","title":"Associating neural word embeddings with deep image representations using fisher vectors","authors":"Benjamin Klein, Guy Lev, Gil Sadeh, Lior Wolf","year":"2015","venue":"Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Krizhevsky A, 2012] Hinton G Krizhevsky A, Sutskever I. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS), 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"A Hinton G Krizhevsky, I Sutskever","year":"2012","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Li et al., 2003] Dongge Li, Nevenka Dimitrova, Mingkun Li, and Ishwar K Sethi. Multimedia content processing through cross-modal association. In ACM Conference on Multimedia (ACM-MM), pages 604-611, 2003.","title":"Multimedia content processing through cross-modal association","authors":"Li","year":"2003","venue":"ACM Conference on Multimedia (ACM-MM)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge J. Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In European Conference on Com- puter Vision (ECCV), pages 740-755, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge J Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence Zitnick","year":"2014","venue":"European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ngiam et al., 2011] Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multi- modal deep learning. In International Conference on Ma- chine Learning (ICML), pages 689-696, 2011.","title":"Multimodal deep learning","authors":"Ngiam","year":"2011","venue":"International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Peng et al., 2016a] Yuxin Peng, Xin Huang, and Jinwei Qi. Cross-media shared representation by hierarchical learn- ing with multiple deep networks. In International Joint Conference on Artificial Intelligence (IJCAI), pages 3846- 3853, 2016.","title":"Cross-media shared representation by hierarchical learning with multiple deep networks","authors":"Peng","year":"2016","venue":"International Joint Conference on Artificial Intelligence (IJCAI)","doi":null,"url":null,"internal_url":"/proceedings/2016/541"},{"position":12,"raw":"[Peng et al., 2016b] Yuxin Peng, Xiaohua Zhai, Yunzhen Zhao, and Xin Huang. Semi-supervised cross-media fea- ture learning with unified patch graph regularization. IEEE Transactions on Circuits and Systems for Video Technol- ogy (TCSVT), 26(3):583-596, 2016.","title":"Semi-supervised cross-media feature learning with unified patch graph regularization","authors":"Peng","year":"2016","venue":"IEEE Transactions on Circuits and Systems for Video Technology (TCSVT)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Peng et al., 2017] Yuxin Peng, Jinwei Qi, Xin Huang, and Yuxin Yuan. CCL: Cross-modal correlation learning with multi-grained fusion by hierarchical network. IEEE Trans- actions on Multimedia (TMM), 2017.","title":"CCL: Cross-modal correlation learning with multi-grained fusion by hierarchical network","authors":"Yuxin Peng, Jinwei Qi, Xin Huang, Yuxin Yuan","year":"2017","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Plummer et al., 2015] Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hocken- maier, and Svetlana Lazebnik. Flickr30k entities: Collect- ing region-to-phrase correspondences for richer image-to- sentence models. In IEEE International Conference on Computer Vision (ICCV), pages 2641-2649, 2015.","title":"Flickr30k entities: Collecting region-to-phrase correspondences for richer image-tosentence models","authors":"Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, Svetlana Lazebnik","year":"2015","venue":"IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ranjan et al., 2015] Viresh Ranjan, Nikhil Rasiwasia, and C. V. Jawahar. Multi-label cross-modal retrieval. In IEEE International Conference on Computer Vision (ICCV), pages 4094-4102, 2015.","title":"Multi-label cross-modal retrieval","authors":"Nikhil Viresh Ranjan, C V Rasiwasia, Jawahar","year":"2015","venue":"IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rasiwasia et al., 2010] Nikhil Rasiwasia, Jose Costa Pereira, Emanuele Coviello, Gabriel Doyle, Gert RG Lanckriet, Roger Levy, and Nuno Vasconcelos. A new approach to cross-modal multimedia retrieval. In ACM Conference on Multimedia (ACM-MM), pages 251-260, 2010.","title":"A new approach to cross-modal multimedia retrieval","authors":"Nikhil Rasiwasia, Jose Costa Pereira, Emanuele Coviello, Gabriel Doyle, Gert Rg Lanckriet, Roger Levy, Nuno Vasconcelos","year":"2010","venue":"ACM Conference on Multimedia (ACM-MM)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross B. Gir- shick, and Jian Sun. Faster R-CNN: towards real-time ob- ject detection with region proposal networks. In Advances in Neural Information Processing Systems (NIPS), pages 91-99, 2015.","title":"Faster R-CNN: towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Confer- ence on Learning Representations (ICLR), 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":"International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Srivastava and Salakhutdinov, 2012] Nitish Srivastava and Ruslan Salakhutdinov. Learning representations for multi- modal data with deep belief nets. In International Confer- ence on Machine Learning (ICML) Workshop, 2012.","title":"Learning representations for multimodal data with deep belief nets","authors":"Nitish Srivastava, Ruslan Salakhutdinov","year":"2012","venue":"International Conference on Machine Learning (ICML) Workshop","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tran et al., 2016] Thi Quynh Nhi Tran, Hervé Le Borgne, and Michel Crucianu. Aggregating image and text quan- tized correlated components. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 2046- 2054, 2016.","title":"Aggregating image and text quantized correlated components","authors":"Thi Quynh, Nhi Tran, Hervé Le Borgne, Michel Crucianu","year":"2016","venue":"Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yan and Mikolajczyk, 2015] Fei Yan and Krystian Mikola- jczyk. Deep correlation for matching images and text. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 3441-3450, 2015.","title":"Deep correlation for matching images and text","authors":"Yan, Mikolajczyk ;, Fei Yan, Krystian Mikolajczyk","year":"2015","venue":"Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Young et al., 2014] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics (TACL), 2:67-78, 2014.","title":"From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions","authors":"Peter Young, Alice Lai, Micah Hodosh, Julia Hockenmaier","year":"2014","venue":"Transactions of the Association for Computational Linguistics (TACL)","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhai et al., 2014] Xiaohua Zhai, YuXin Peng, and Jianguo Xiao. Learning cross-media joint representation with sparse and semi-supervised regularization. IEEE Trans- actions on Circuits and Systems for Video Technology (TCSVT), 24:965-978, 2014.","title":"Learning cross-media joint representation with sparse and semi-supervised regularization","authors":"Xiaohua Zhai, Yuxin Peng, Jianguo Xiao","year":"2014","venue":"IEEE Transactions on Circuits and Systems for Video Technology (TCSVT)","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2015] Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in Neural Information Pro- cessing Systems (NIPS), pages 649-657, 2015.","title":"Character-level convolutional networks for text classification","authors":"Xiang Zhang, Junbo, Jake Zhao, Yann Lecun","year":"2015","venue":"Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zheng et al., 2017] Zhedong Zheng, Liang Zheng, Michael Garrett, Yi Yang, and Yi-Dong Shen. Dual-path convolutional image-text embedding. arXiv preprint arXiv:1711.05535, 2017.","title":"Dual-path convolutional image-text embedding","authors":"Zhedong Zheng, Liang Zheng, Michael Garrett, Yi Yang, Yi-Dong Shen","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}