{"type":"ScholarlyArticle","title":"Deep Reasoning with Knowledge Graph for Social Relationship Understanding","authors":[{"name":"Zhouxia Wang","slug":"zhouxia-wang","given":"Zhouxia","family":"Wang","affiliation":"School of Data and Computer Science , Sun Yat-sen University , China; SenseTime Research , China"},{"name":"Tianshui Chen","slug":"tianshui-chen","given":"Tianshui","family":"Chen","affiliation":"School of Data and Computer Science , Sun Yat-sen University , China"},{"name":"Jimmy Ren","slug":"jimmy-ren","given":"Jimmy","family":"Ren","affiliation":"SenseTime Research , China"},{"name":"Weihao Yu","slug":"weihao-yu","given":"Weihao","family":"Yu","affiliation":"School of Data and Computer Science , Sun Yat-sen University , China"},{"name":"Hui Cheng","slug":"hui-cheng","given":"Hui","family":"Cheng","affiliation":"School of Data and Computer Science , Sun Yat-sen University , China; SenseTime Research , China"},{"name":"Liang Lin","slug":"liang-lin","given":"Liang","family":"Lin","affiliation":"School of Data and Computer Science , Sun Yat-sen University , China; SenseTime Research , China"}],"abstract":"Social relationships (e.g., friends, couple etc.) form the basis of the social network in our daily life. Automatically interpreting such relationships bears a great potential for the intelligent systems to understand human behavior in depth and to better interact with people at a social level. Human beings interpret the social relationships within a group not only based on the people alone, and the interplay between such social relationships and the contextual information around the people also plays a significant role. However, these additional cues are largely overlooked by the previous studies. We found that the interplay between these two factors can be effectively modeled by a novel structured knowledge graph with proper message propagation and attention. And this structured knowledge can be efficiently integrated into the deep neural network architecture to promote social relationship understanding by an end-to-end trainable Graph Reasoning Model (GRM), in which a propagation mechanism is learned to propagate node message through the graph to explore the interaction between persons of interest and the contextual objects. Meanwhile, a graph attentional mechanism is introduced to explicitly reason about the discriminative objects to promote recognition. Extensive experiments on the public benchmarks demonstrate the superiority of our method over the existing leading competitors.","doi":"10.24963/ijcai.2018/142","url":"https://ijcai.org/proceedings/2018/142","pdf":"https://ijcai.org/proceedings/2018/0142.pdf","markdown":"https://ijcai.org/proceedings/2018/142.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/142","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":142,"pages":{"first":1021,"last":1028},"keywords":["Computer Vision: Computer Vision","Computer Vision: Structural and Model-Based Approaches, Knowledge Representation and Reasoning"],"videos":[],"references":[{"position":1,"raw":"[Alahi et al., 2016] Alexandre Alahi, Kratarth Goel, Vignesh Ra- manathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social lstm: Human trajectory prediction in crowded spaces. In CVPR, pages 961-971, 2016.","title":"Social lstm: Human trajectory prediction in crowded spaces","authors":"Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, Silvio Savarese","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bugental, 2000] Daphne Blunt Bugental. Acquisition of the algo- rithms of social life: A domain-based approach. Psychological bulletin, 126(2):187, 2000.","title":"Acquisition of the algorithms of social life: A domain-based approach","authors":"Daphne Blunt Bugental, Bugental","year":"2000","venue":"Psychological bulletin","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2012] Yan-Ying Chen, Winston H Hsu, and Hong- Yuan Mark Liao. Discovering informative social subgraphs and predicting pairwise relationships from group photos. In ACM MM, pages 669-678, 2012.","title":"Discovering informative social subgraphs and predicting pairwise relationships from group photos","authors":"Yan-Ying Chen, Winston H Hsu, Hong-Yuan Mark Liao","year":"2012","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2018] Tianshui Chen, Zhouxia Wang, Guanbin Li, and Liang Lin. Recurrent attentional reinforcement learning for multi-label image recognition. In AAAI, pages 6730-6737, 2018.","title":"Recurrent attentional reinforcement learning for multi-label image recognition","authors":"Chen","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Choi and Savarese, 2012] Wongun Choi and Silvio Savarese. A unified framework for multi-target tracking and collective activity recognition. ECCV, pages 215-230, 2012.","title":"A unified framework for multi-target tracking and collective activity recognition","authors":"Savarese Choi, Wongun Choi, Silvio Savarese","year":"2012","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Deng et al., 2016] Zhiwei Deng, Arash Vahdat, Hexiang Hu, and Greg Mori. Structure inference machines: Recurrent neural net- works for analyzing relations in group activity recognition. In CVPR, pages 4772-4781, 2016.","title":"Structure inference machines: Recurrent neural networks for analyzing relations in group activity recognition","authors":"Zhiwei Deng, Arash Vahdat, Hexiang Hu, Greg Mori","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dibeklioglu et al., 2013] Hamdi Dibeklioglu, Albert Ali Salah, and Theo Gevers. Like father, like son: Facial expression dynam- ics for kinship verification. In ICCV, pages 1497-1504, 2013.","title":"Like father, like son: Facial expression dynamics for kinship verification","authors":"Hamdi Dibeklioglu, Albert Ali Salah, Theo Gevers","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Duvenaud et al., 2015] David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. In NIPS, pages 2224- 2232, 2015.","title":"Convolutional networks on graphs for learning molecular fingerprints","authors":"Dougal David K Duvenaud, Jorge Maclaurin, Rafael Iparraguirre, Timothy Bombarell, Alán Hirzel, Ryan P Aspuru-Guzik, Adams","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Fairclough, 2003] Norman Fairclough. Analysing discourse: Tex- tual analysis for social research. Psychology Press, 2003.","title":"Norman Fairclough. Analysing discourse: Textual analysis for social research","authors":"Fairclough","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Fiske, 1992] Alan P Fiske. The four elementary forms of sociality: framework for a unified theory of social relations. Psychological review, 99(4):689, 1992.","title":"The four elementary forms of sociality: framework for a unified theory of social relations","authors":"Alan P Fiske","year":"1992","venue":"Psychological review","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huang et al., 2016] Chen Huang, Chen Change Loy, and Xiaoou Tang. Unsupervised learning of discriminative attributes and vi- sual representations. In CVPR, pages 5175-5184, 2016.","title":"Unsupervised learning of discriminative attributes and visual representations","authors":"Chen Huang, Chen Change Loy, Xiaoou Tang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kim et al., 2016] Jin-Hwa Kim, Kyoung-Woon On, Woosang Lim, Jeonghee Kim, Jung-Woo Ha, and Byoung-Tak Zhang. Hadamard product for low-rank bilinear pooling. arXiv preprint arXiv:1610.04325, 2016.","title":"Hadamard product for low-rank bilinear pooling","authors":"Jin-Hwa Kim, Kyoung-Woon On, Woosang Lim, Jeonghee Kim, Jung-Woo Ha, Byoung-Tak Zhang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kipf and Welling, 2016] Thomas N Kipf and Max Welling. Semi- supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.","title":"Semisupervised classification with graph convolutional networks","authors":"N Thomas, Max Kipf, Welling","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lan et al., 2012] Tian Lan, Leonid Sigal, and Greg Mori. Social roles in hierarchical models for human activity recognition. In CVPR, pages 1354-1361, 2012.","title":"Social roles in hierarchical models for human activity recognition","authors":"Tian Lan, Leonid Sigal, Greg Mori","year":"2012","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li et al., 2015a] Li-Jia Li, David A Shamma, Xiangnan Kong, Sina Jafarpour, Roelof Van Zwol, and Xuanhui Wang. Celebri- tynet: A social network constructed from large-scale online celebrity images. TOMM, 12(1):3, 2015.","title":"Celebritynet: A social network constructed from large-scale online celebrity images","authors":"Li","year":"2015","venue":"TOMM","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2015b] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493, 2015.","title":"Gated graph sequence neural networks","authors":"Li","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2017] Junnan Li, Yongkang Wong, Qi Zhao, and Mo- han S Kankanhalli. Dual-glance model for deciphering social relationships. In ICCV, pages 2650-2659, 2017.","title":"Dual-glance model for deciphering social relationships","authors":"Li","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liang et al., 2016] Xiaodan Liang, Xiaohui Shen, Jiashi Feng, Liang Lin, and Shuicheng Yan. Semantic object parsing with graph lstm. In ECCV, pages 125-143, 2016.","title":"Semantic object parsing with graph lstm","authors":"Xiaodan Liang, Xiaohui Shen, Jiashi Feng, Liang Lin, Shuicheng Yan","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liang et al., 2017] Xiaodan Liang, Liang Lin, Xiaohui Shen, Ji- ashi Feng, Shuicheng Yan, and Eric P Xing. Interpretable structure-evolving lstm. In CVPR, pages 2175-2184, 2017.","title":"Interpretable structure-evolving lstm","authors":"Xiaodan Liang, Liang Lin, Xiaohui Shen, Jiashi Feng, Shuicheng Yan, Eric P Xing","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in con- text. In ECCV, pages 740-755, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Lin et al., 2017] Liang Lin, Lili Huang, Tianshui Chen, Yukang Gan, and Hui Cheng. Knowledge-guided recurrent neural net- work learning for task-oriented action prediction. In ICME, pages 625-630, 2017.","title":"Knowledge-guided recurrent neural network learning for task-oriented action prediction","authors":"Liang Lin, Lili Huang, Tianshui Chen, Yukang Gan, Hui Cheng","year":"2017","venue":"ICME","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Lu et al., 2016] Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Visual relationship detection with language priors. In ECCV, pages 852-869, 2016.","title":"Visual relationship detection with language priors","authors":"Lu","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Malisiewicz and Efros, 2009] Tomasz Malisiewicz and Alyosha Efros. Beyond categories: The visual memex model for reason- ing about object relationships. In NIPS, pages 1222-1230, 2009.","title":"Beyond categories: The visual memex model for reasoning about object relationships","authors":"Tomasz Malisiewicz, Alyosha Efros","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Marino et al., 2016] Kenneth Marino, Ruslan Salakhutdinov, and Abhinav Gupta. The more you know: Using knowledge graphs for image classification. arXiv preprint arXiv:1612.04844, 2016.","title":"The more you know: Using knowledge graphs for image classification","authors":"Kenneth Marino, Ruslan Salakhutdinov, Abhinav Gupta","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Qi et al., 2017] Xiaojuan Qi, Renjie Liao, Jiaya Jia, Sanja Fidler, and Raquel Urtasun. 3d graph neural networks for rgbd semantic segmentation. In CVPR, pages 5199-5208, 2017.","title":"3d graph neural networks for rgbd semantic segmentation","authors":"Xiaojuan Qi, Renjie Liao, Jiaya Jia, Sanja Fidler, Raquel Urtasun","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, pages 91-99, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Schlichtkrull et al., 2017] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. arXiv preprint arXiv:1703.06103, 2017.","title":"Modeling relational data with graph convolutional networks","authors":"Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den, Ivan Berg, Max Titov, Welling","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale im- age recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Sun et al., 2017] Qianru Sun, Bernt Schiele, and Mario Fritz. A domain based approach to social relation recognition. In ICCV, pages 435-444, 2017.","title":"A domain based approach to social relation recognition","authors":"Sun","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Teney et al., 2017] Damien Teney, Lingqiao Liu, and Anton van den Hengel. Graph-structured representations for visual question answering. In CVPR, pages 3233-3241, 2017.","title":"Graph-structured representations for visual question answering","authors":"Damien Teney, Lingqiao Liu, Anton Van Den, Hengel","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Wang et al., 2010] Gang Wang, Andrew Gallagher, Jiebo Luo, and David Forsyth. Seeing people in social context: Recognizing people and social relationships. ECCV, pages 169-182, 2010.","title":"Seeing people in social context: Recognizing people and social relationships","authors":"Wang","year":"2010","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Wang et al., 2017] Zhouxia Wang, Tianshui Chen, Guanbin Li, Ruijia Xu, and Liang Lin. Multi-label image recognition by re- currently discovering attentional regions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 464-472, 2017.","title":"Multi-label image recognition by recurrently discovering attentional regions","authors":"Wang","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Xia et al., 2012] Siyu Xia, Ming Shao, Jiebo Luo, and Yun Fu. Un- derstanding kin relationships in a photo. TMM, 14(4):1046-1056, 2012.","title":"Understanding kin relationships in a photo","authors":"Siyu Xia, Ming Shao, Jiebo Luo, Yun Fu","year":"2012","venue":"TMM","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2015] Ning Zhang, Manohar Paluri, Yaniv Taigman, Rob Fergus, and Lubomir Bourdev. Beyond frontal faces: Im- proving person recognition using multiple cues. In CVPR, pages 4804-4813, 2015.","title":"Beyond frontal faces: Improving person recognition using multiple cues","authors":"Ning Zhang, Manohar Paluri, Yaniv Taigman, Rob Fergus, Lubomir Bourdev","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhu et al., 2014] Yuke Zhu, Alireza Fathi, and Li Fei-Fei. Reason- ing about object affordances in a knowledge base representation. In ECCV, pages 408-424, 2014.","title":"Reasoning about object affordances in a knowledge base representation","authors":"Yuke Zhu, Alireza Fathi, Li Fei-Fei","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}