{"type":"ScholarlyArticle","title":"Image-embodied Knowledge Representation Learning","authors":[{"name":"Ruobing Xie","slug":"ruobing-xie","given":"Ruobing","family":"Xie","affiliation":"Department of Computer Science and Technology , State Key Lab on Intelligent Technology and Systems ,"},{"name":"Zhiyuan Liu","slug":"zhiyuan-liu","given":"Zhiyuan","family":"Liu","affiliation":"Department of Computer Science and Technology , State Key Lab on Intelligent Technology and Systems ,; Jiangsu Collaborative Innovation Center for Language Ability , Jiangsu Normal University , China"},{"name":"Huanbo Luan","slug":"huanbo-luan","given":"Huanbo","family":"Luan","affiliation":"Department of Computer Science and Technology , State Key Lab on Intelligent Technology and Systems ,; National Lab for Information Science and Technology , Tsinghua University , China"},{"name":"Maosong Sun","slug":"maosong-sun","given":"Maosong","family":"Sun","affiliation":"Department of Computer Science and Technology , State Key Lab on Intelligent Technology and Systems ,; Jiangsu Collaborative Innovation Center for Language Ability , Jiangsu Normal University , China"}],"abstract":"Entity images could provide significant visual information for knowledge representation learning. Most conventional methods learn knowledge representations merely from structured triples, ignoring rich visual information extracted from entity images. In this paper, we propose a novel Image-embodied Knowledge Representation Learning model (IKRL), where knowledge representations are learned with both triple facts and images. More specifically, we first construct representations for all images of an entity with a neural image encoder. These image representations are then integrated into an aggregated image-based representation via an attention-based method. We evaluate our IKRL models on knowledge graph completion and triple classification. Experimental results demonstrate that our models outperform all baselines on both tasks, which indicates the significance of visual information for knowledge representations and the capability of our models in learning knowledge representations with images.","doi":"10.24963/ijcai.2017/438","url":"https://ijcai.org/proceedings/2017/438","pdf":"https://ijcai.org/proceedings/2017/0438.pdf","markdown":"https://ijcai.org/proceedings/2017/438.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/438","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":438,"pages":{"first":3140,"last":3146},"keywords":["Natural Language Processing: Natural Language Semantics","Natural Language Processing: Information Extraction","Machine Learning: Data Mining"],"videos":[],"references":[{"position":1,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Ji- asen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Z- itnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of ICCV, pages 2425-2433, 2015.","title":"Vqa: Visual question answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh","year":"2015","venue":"Proceedings of ICCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"Proceedings of ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bordes et al., 2012] Antoine Bordes, Xavier Glorot, Jason Weston, and Yoshua Bengio. Joint learning of words and meaning representations for open-text semantic parsing. In Proceedings of AISTATS, pages 127-135, 2012.","title":"Joint learning of words and meaning representations for open-text semantic parsing","authors":"Antoine Bordes, Xavier Glorot, Jason Weston, Yoshua Bengio","year":"2012","venue":"Proceedings of AISTATS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bordes et al., 2013] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi- relational data. In Proceedings of NIPS, pages 2787-2795, 2013.","title":"Translating embeddings for modeling multirelational data","authors":"Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, Oksana Yakhnenko","year":"2013","venue":"Proceedings of NIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bordes et al., 2014] Antoine Bordes, Xavier Glorot, Jason Weston, and Yoshua Bengio. A semantic matching energy function for learning with multi-relational data. Machine Learning, 94(2):233-259, 2014.","title":"A semantic matching energy function for learning with multi-relational data","authors":"Antoine Bordes, Xavier Glorot, Jason Weston, Yoshua Bengio","year":"2014","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In Proceedings of CVPR, pages 248-255, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ji et al., 2015] Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. Knowledge graph embedding via dy- namic mapping matrix. In Proceedings of ACL, pages 687-696, 2015.","title":"Knowledge graph embedding via dynamic mapping matrix","authors":"Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, Jun Zhao","year":"2015","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jia et al., 2014] Yangqing Jia, Evan Shelhamer, Jeff Don- ahue, Sergey Karayev, Jonathan Long, Ross Girshick, Ser- gio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of ACM MM, pages 675-678, 2014.","title":"Caffe: Convolutional architecture for fast feature embedding","authors":"Jia","year":"2014","venue":"Proceedings of ACM MM","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kiros et al., 2014] Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic embeddings with multimodal neural language models. In Proceedings of NIPS, 2014.","title":"Unifying visual-semantic embeddings with multimodal neural language models","authors":"Ryan Kiros, Ruslan Salakhutdinov, Richard S Zemel","year":"2014","venue":"Proceedings of NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Proceedings of NIPS, pages 1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Proceedings of NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lin et al., 2015] Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation em- beddings for knowledge graph completion. In Proceedings of AAAI, 2015.","title":"Learning entity and relation embeddings for knowledge graph completion","authors":"Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, Xuan Zhu","year":"2015","venue":"Proceedings of AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of HLT- NAACL, 2013.","title":"Linguistic regularities in continuous space word representations","authors":"Mikolov","year":"2013","venue":"Proceedings of HLT-NAACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Miller, 1995] George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 1995.","title":"Wordnet: a lexical database for english","authors":"George A Miller","year":"1995","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih et al., 2014] Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In Proceedings of NIPS, pages 2204-2212, 2014.","title":"Recurrent models of visual attention","authors":"Mnih","year":"2014","venue":"Proceedings of NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rush et al., 2015] Alexander M Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. In Proceedings of EMNLP, 2015.","title":"A neural attention model for abstractive sentence summarization","authors":"Sumit Alexander M Rush, Jason Chopra, Weston","year":"2015","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Shutova et al., 2016] Ekaterina Shutova, Douwe Kiela, and Jean Maillard. Black holes and white rabbits: Metaphor identification with visual features. In Proceedings of NAA- CL, 2016.","title":"Black holes and white rabbits: Metaphor identification with visual features","authors":"Ekaterina Shutova, Douwe Kiela, Jean Maillard","year":"2016","venue":"Proceedings of NAA-CL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Socher et al., 2013] Richard Socher, Danqi Chen, Christo- pher D Manning, and Andrew Ng. Reasoning with neural tensor networks for knowledge base completion. In Pro- ceedings of NIPS, pages 926-934, 2013.","title":"Reasoning with neural tensor networks for knowledge base completion","authors":"Richard Socher, Danqi Chen, Christopher D Manning, Andrew Ng","year":"2013","venue":"Proceedings of NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wang et al., 2014a] Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph and text jointly embedding. In Proceedings of EMNLP, pages 1591-1601, 2014.","title":"Knowledge graph and text jointly embedding","authors":"Wang","year":"2014","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2014b] Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by translating on hyperplanes. In Proceedings of AAAI, pages 1112-1119, 2014.","title":"Knowledge graph embedding by translating on hyperplanes","authors":"Wang","year":"2014","venue":"Proceedings of AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xie et al., 2016] Ruobing Xie, Zhiyuan Liu, Jia Jia, Huan- bo Luan, and Maosong Sun. Representation learning of knowledge graphs with entity descriptions. In Proceed- ings of AAAI, 2016.","title":"Representation learning of knowledge graphs with entity descriptions","authors":"Ruobing Xie, Zhiyuan Liu, Jia Jia, Huanbo Luan, Maosong Sun","year":"2016","venue":"Proceedings of AAAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang et al., 2014] Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. In Proceedings of ICLR, 2014.","title":"Embedding entities and relations for learning and inference in knowledge bases","authors":"Bishan Yang, Wen-Tau Yih, Xiaodong He, Jianfeng Gao, Li Deng","year":"2014","venue":"Proceedings of ICLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yin et al., 2016] Jun Yin, Xin Jiang, Zhengdong Lu, Lifeng Shang, Hang Li, and Xiaoming Li. Neural generative question answering. In Proceedings of IJCAI, 2016.","title":"Neural generative question answering","authors":"Jun Yin, Xin Jiang, Zhengdong Lu, Lifeng Shang, Hang Li, Xiaoming Li","year":"2016","venue":"Proceedings of IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/422"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}