{"type":"ScholarlyArticle","title":"Representation Learning for Scene Graph Completion via Jointly Structural and Visual Embedding","authors":[{"name":"Hai Wan","slug":"hai-wan","given":"Hai","family":"Wan","affiliation":"School of Data and Computer Science , Sun Yat-sen University , Guangzhou , China; Guangdong Key Laboratory of Big Data Analysis and Processing , Guangzhou , China; Key Laboratory of Machine Intelligence and Advanced Computing (Sun Yat- sen University) , Ministry of Education , China"},{"name":"Yonghao Luo","slug":"yonghao-luo","given":"Yonghao","family":"Luo","affiliation":"School of Data and Computer Science , Sun Yat-sen University , Guangzhou , China"},{"name":"Bo Peng","slug":"bo-peng","given":"Bo","family":"Peng","affiliation":"School of Data and Computer Science , Sun Yat-sen University , Guangzhou , China"},{"name":"Wei-Shi Zheng","slug":"wei-shi-zheng","given":"Wei-Shi","family":"Zheng","affiliation":"School of Data and Computer Science , Sun Yat-sen University , Guangzhou , China; Key Laboratory of Machine Intelligence and Advanced Computing (Sun Yat- sen University) , Ministry of Education , China"}],"abstract":"This paper focuses on scene graph completion which aims at predicting new relations between two entities utilizing existing scene graphs and images. By comparing with the well-known knowledge graph, we first identify that each scene graph is associated with an image and each entity of a visual triple in a scene graph is composed of its entity type with attributes and grounded with a bounding box in its corresponding image. We then propose an end-to-end model named Representation Learning via Jointly Structural and Visual Embedding (RLSV) to take advantages of structural and visual information in scene graphs. In RLSV model, we provide a fully-convolutional module to extract the visual embeddings of a visual triple and apply hierarchical projection to combine the structural and visual embeddings of a visual triple. In experiments, we evaluate our model on two scene graph completion tasks: link prediction and visual triple classification, and further analyze by case studies. Experimental results demonstrate that our model outperforms all baselines in both tasks, which justifies the significance of combining structural and visual information for scene graph completion.","doi":"10.24963/ijcai.2018/132","url":"https://ijcai.org/proceedings/2018/132","pdf":"https://ijcai.org/proceedings/2018/0132.pdf","markdown":"https://ijcai.org/proceedings/2018/132.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/132","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":132,"pages":{"first":949,"last":956},"keywords":["Computer Vision: Structural and Model-Based Approaches, Knowledge Representation and Reasoning","Knowledge Representation and Reasoning: Knowledge Representation Languages"],"videos":[],"references":[{"position":1,"raw":"[Anderson et al., 2016] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. SPICE: semantic propositional image caption evaluation. In Proceedings of ECCV, pages 382-398, 2016.","title":"SPICE: semantic propositional image caption evaluation","authors":"Anderson","year":"2016","venue":"Proceedings of ECCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bordes et al., 2013] Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi- relational data. In Proceedings of NIPS, pages 2787-2795, 2013.","title":"Translating embeddings for modeling multirelational data","authors":"Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston, Oksana Yakhnenko","year":"2013","venue":"Proceedings of NIPS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dai et al., 2017] Bo Dai, Yuqi Zhang, and Dahua Lin. De- tecting visual relationships with deep relational networks. In Proceedings of CVPR, pages 3298-3308, 2017.","title":"Detecting visual relationships with deep relational networks","authors":"Bo Dai, Yuqi Zhang, Dahua Lin","year":"2017","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Elhoseiny et al., 2017] Mohamed Elhoseiny, Scott Cohen, Walter Chang, Brian L. Price, and Ahmed M. Elgammal. Sherlock: Scalable fact learning in images. In Proceedings of AAAI, pages 4016-4024, 2017.","title":"Sherlock: Scalable fact learning in images","authors":"Mohamed Elhoseiny, Scott Cohen, Walter Chang, Brian L Price, Ahmed M Elgammal","year":"2017","venue":"Proceedings of AAAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Girshick, 2015] Ross B. Girshick. Fast R-CNN. In Pro- ceedings of ICCV, pages 1440-1448, 2015.","title":"Fast R-CNN","authors":"Ross B Girshick","year":"2015","venue":"Proceedings of ICCV","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Glorot and Bengio, 2010] Xavier Glorot and Yoshua Ben- gio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of AISTATS, pages 249-256, 2010.","title":"Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks","authors":"Bengio Glorot","year":"2010","venue":"Proceedings of AISTATS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2015] Shizhu He, Kang Liu, Guoliang Ji, and Jun Zhao. Learning to represent knowledge graphs with gaus- sian embedding. In Proceedings of CIKM, pages 623-632, 2015.","title":"Learning to represent knowledge graphs with gaussian embedding","authors":"He","year":"2015","venue":"Proceedings of CIKM","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jaderberg et al., 2015] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. In Proceedings of NIPS, pages 2017-2025, 2015.","title":"Spatial transformer networks","authors":"Jaderberg","year":"2015","venue":"Proceedings of NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ji et al., 2015] Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. Knowledge graph embedding via dy- namic mapping matrix. In Proceedings of ACL, pages 687-696, 2015.","title":"Knowledge graph embedding via dynamic mapping matrix","authors":"Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, Jun Zhao","year":"2015","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ji et al., 2016] Guoliang Ji, Kang Liu, Shizhu He, and Jun Zhao. Knowledge graph completion with adaptive sparse transfer matrix. In Proceedings of AAAI, pages 985-991, 2016.","title":"Knowledge graph completion with adaptive sparse transfer matrix","authors":"Guoliang Ji, Kang Liu, Shizhu He, Jun Zhao","year":"2016","venue":"Proceedings of AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Johnson et al., 2015] Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Image retrieval using scene graphs. In Proceedings of CVPR, pages 3668-3678, 2015.","title":"Image retrieval using scene graphs","authors":"Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David A Shamma, Michael S Bernstein, Fei-Fei Li","year":"2015","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Johnson et al., 2016] Justin Johnson, Andrej Karpathy, and Li Fei-Fei. Densecap: Fully convolutional localization networks for dense captioning. In Processings of CVPR, pages 4565-4574, 2016.","title":"Densecap: Fully convolutional localization networks for dense captioning","authors":"Justin Johnson, Andrej Karpathy, Li Fei-Fei","year":"2016","venue":"Processings of CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Process- ings of ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"Processings of ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Krishna et al., 2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowd- sourced dense image annotations. International Journal of Computer Vision, 123(1):32-73, 2017.","title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","authors":"Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael S Bernstein, Li Fei-Fei","year":"2017","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2017] Yikang Li, Wanli Ouyang, Bolei Zhou, Kun Wang, and Xiaogang Wang. Scene graph generation from objects, phrases and region captions. In Proceedings of ICCV, pages 1270-1279, 2017.","title":"Scene graph generation from objects, phrases and region captions","authors":"Li","year":"2017","venue":"Proceedings of ICCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liang et al., 2017] Xiaodan Liang, Lisa Lee, and Eric P. Xing. Deep variation-structured reinforcement learning for visual relationship and attribute detection. In Proceed- ings of CVPR, pages 4408-4417, 2017.","title":"Deep variation-structured reinforcement learning for visual relationship and attribute detection","authors":"Xiaodan Liang, Lisa Lee, Eric P Xing","year":"2017","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liao et al., 2017] Wentong Liao, Shuai Lin, Bodo Rosen- hahn, and Michael Ying Yang. Natural language guided vi- sual relationship detection. CoRR, abs/1711.06032, 2017.","title":"Natural language guided visual relationship detection","authors":"Liao","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lin et al., 2014] Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. In Processings of ICLR, 2014.","title":"Network in network","authors":"Lin","year":"2014","venue":"Processings of ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lin et al., 2015] Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation em- beddings for knowledge graph completion. In Proceedings of AAAI, pages 2181-2187, 2015.","title":"Learning entity and relation embeddings for knowledge graph completion","authors":"Lin","year":"2015","venue":"Proceedings of AAAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lu et al., 2016] Cewu Lu, Ranjay Krishna, Michael S. Bernstein, and Fei-Fei Li. Visual relationship detection with language priors. In Proceedings of ECCV, pages 852- 869, 2016.","title":"Visual relationship detection with language priors","authors":"Lu","year":"2016","venue":"Proceedings of ECCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Maaten and Hinton, 2008] Laurens Van Der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(2605):2579-2605, 2008.","title":"Visualizing data using t-sne","authors":"Hinton Maaten, Laurens Van, Der Maaten, Geoffrey Hinton","year":"2008","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Marino et al., 2017] Kenneth Marino, Ruslan Salakhutdi- nov, and Abhinav Gupta. The more you know: Using knowledge graphs for image classification. In Proceedings of CVPR, pages 20-28, 2017.","title":"The more you know: Using knowledge graphs for image classification","authors":"Kenneth Marino, Ruslan Salakhutdinov, Abhinav Gupta","year":"2017","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Socher et al., 2013] Richard Socher, Danqi Chen, Christo- pher D. Manning, and Andrew Y. Ng. Reasoning with neural tensor networks for knowledge base completion. In Proceedings of NIPS, pages 926-934, 2013.","title":"Reasoning with neural tensor networks for knowledge base completion","authors":"Richard Socher, Danqi Chen, Christopher D Manning, Andrew Y Ng","year":"2013","venue":"Proceedings of NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2014] Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by translating on hyperplanes. In Proceedings of AAAI, pages 1112-1119, 2014.","title":"Knowledge graph embedding by translating on hyperplanes","authors":"Zhen Wang, Jianwen Zhang, Jianlin Feng, Zheng Chen","year":"2014","venue":"Proceedings of AAAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xiao et al., 2016] Han Xiao, Minlie Huang, and Xiaoyan Zhu. TransG: A generative model for knowledge graph embedding. In Proceedings of ACL, pages 2316-2325, 2016.","title":"TransG: A generative model for knowledge graph embedding","authors":"Han Xiao, Minlie Huang, Xiaoyan Zhu","year":"2016","venue":"Proceedings of ACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xu et al., 2017] Danfei Xu, Yuke Zhu, Christopher Choy, and Li Fei-Fei. Scene graph generation by iterative mes- sage passing. In Proceedings of CVPR, pages 5410-5419, 2017.","title":"Scene graph generation by iterative message passing","authors":"Danfei Xu, Yuke Zhu, Christopher Choy, Li Fei-Fei","year":"2017","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yu et al., 2017] Ruichi Yu, Ang Li, Vlad I. Morariu, and Larry S. Davis. Visual relationship detection with internal and external linguistic knowledge distillation. In Proceed- ings of ICCV, pages 1068-1076, 2017.","title":"Visual relationship detection with internal and external linguistic knowledge distillation","authors":"Ruichi Yu, Ang Li, Vlad I Morariu, Larry S Davis","year":"2017","venue":"Proceedings of ICCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2017a] Hanwang Zhang, Zawlin Kyaw, Shih- Fu Chang, and Tat-Seng Chua. Visual translation embed- ding network for visual relation detection. In Proceedings of CVPR, pages 3107-3115, 2017.","title":"Visual translation embedding network for visual relation detection","authors":"Zhang","year":"2017","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2017b] Hanwang Zhang, Zawlin Kyaw, Jinyang Yu, and Shih-Fu Chang. PPR-FCN: weakly supervised visual relation detection via parallel pairwise R-FCN. In Proceedings of ICCV, pages 4243-4251, 2017.","title":"PPR-FCN: weakly supervised visual relation detection via parallel pairwise R-FCN","authors":"Zhang","year":"2017","venue":"Proceedings of ICCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhu et al., 2017] Yuke Zhu, Joseph J Lim, and Li Fei-Fei. Knowledge acquisition for visual question answering via iterative querying. In Proceedings of CVPR, pages 1154- 1163, 2017.","title":"Knowledge acquisition for visual question answering via iterative querying","authors":"Yuke Zhu, Joseph J Lim, Li Fei-Fei","year":"2017","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}