{"type":"ScholarlyArticle","title":"A Graph-based Interactive Reasoning for Human-Object Interaction Detection","authors":[{"name":"Dongming Yang","slug":"dongming-yang","given":"Dongming","family":"Yang","affiliation":"School of ECE , Peking University , Shenzhen , China"},{"name":"Yuexian Zou","slug":"yuexian-zou","given":"Yuexian","family":"Zou","affiliation":"School of ECE , Peking University , Shenzhen , China; Peng Cheng Laboratory , Shenzhen , China"}],"abstract":"Human-Object Interaction (HOI) detection devotes to learn how humans interact with surrounding objects via inferring triplets of < human, verb, object >. However, recent HOI detection methods mostly rely on additional annotations (e.g., human pose) and neglect powerful interactive reasoning beyond convolutions. In this paper, we present a novel graph-based interactive reasoning model called Interactive Graph (abbr. in-Graph) to infer HOIs, in which interactive semantics implied among visual targets are efficiently exploited. The proposed model consists of a project function that maps related targets from convolution space to a graph-based semantic space, a message passing process propagating semantics among all nodes and an update function transforming the reasoned nodes back to convolution space. Furthermore, we construct a new framework to assemble in-Graph models for detecting HOIs, namely in-GraphNet. Beyond inferring HOIs using instance features respectively, the framework dynamically parses pairwise interactive semantics among visual targets by integrating two-level in-Graphs, i.e., scene-wide and instance-wide in-Graphs. Our framework is end-to-end trainable and free from costly annotations like human pose. Extensive experiments show that our proposed framework outperforms existing HOI detection methods on both V-COCO and HICO-DET benchmarks and improves the baseline about 9.4% and 15% relatively, validating its efficacy in detecting HOIs.","doi":"10.24963/ijcai.2020/155","url":"https://ijcai.org/proceedings/2020/155","pdf":"https://ijcai.org/proceedings/2020/0155.pdf","markdown":"https://ijcai.org/proceedings/2020/155.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/155","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":155,"pages":{"first":1111,"last":1117},"keywords":["Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Computer Vision: 2D and 3D Computer Vision","Computer Vision: Structural and Model-Based Approaches, Knowledge Representation and Reasoning","Computer Vision: Action Recognition"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24122/file_24122.m3u8","length_sec":278,"provider":"hls"}],"references":[{"position":1,"raw":"[Chao et al., 2018] Yuwei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human- object interactions. workshop on applications of computer vision, pages 381-389, 2018.","title":"Learning to detect humanobject interactions","authors":"Yuwei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, Jia Deng","year":"2018","venue":"workshop on applications of computer vision","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Fang et al., 2017] Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. Rmpe: Regional multi-person pose estima- tion. In Proceedings of the IEEE International Conference on Computer Vision, pages 2334-2343, 2017.","title":"Rmpe: Regional multi-person pose estimation","authors":"Shuqin Hao-Shu Fang, Yu-Wing Xie, Cewu Tai, Lu","year":"2017","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fang et al., 2018] Hao-Shu Fang, Jinkun Cao, Yu-Wing Tai, and Cewu Lu. Pairwise body-part attention for recogniz- ing human-object interactions. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV), pages 51- 67, 2018.","title":"Pairwise body-part attention for recognizing human-object interactions","authors":"Jinkun Hao-Shu Fang, Yu-Wing Cao, Cewu Tai, Lu","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gao et al., 2018] Chen Gao, Yuliang Zou, and Jiabin Huang. ican: Instance-centric attention network for human-object interaction detection. british machine vision conference, page 41, 2018.","title":"ican: Instance-centric attention network for human-object interaction detection","authors":"Chen Gao, Yuliang Zou, Jiabin Huang","year":"2018","venue":"british machine vision conference","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gilmer et al., 2017] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1263-1272. JMLR. org, 2017.","title":"Neural message passing for quantum chemistry","authors":"Justin Gilmer, S Samuel, Patrick F Schoenholz, Oriol Riley, George E Vinyals, Dahl","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gkioxari et al., 2015] Georgia Gkioxari, Ross B Girshick, and Jitendra Malik. Contextual action recognition with r*cnn. international conference on computer vision, pages 1080-1088, 2015.","title":"Contextual action recognition with r*cnn. international conference on computer vision","authors":"Gkioxari","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gkioxari et al., 2018] Georgia Gkioxari, Ross B Girshick, Piotr Dollar, and Kaiming He. Detecting and recognizing human-object interactions. computer vision and pattern recognition, pages 8359-8367, 2018.","title":"Detecting and recognizing human-object interactions. computer vision and pattern recognition","authors":"Gkioxari","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.","title":"Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn","authors":"He","year":"2017","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Idrees et al., 2017] Haroon Idrees, Amir R Zamir, Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. The thumos challenge on action recogni- tion for videos \"in the wild\". Computer Vision and Image Understanding, 155:1-23, 2017.","title":"The thumos challenge on action recognition for videos \"in the wild","authors":"Haroon Idrees, Yu-Gang Amir R Zamir, Alex Jiang, Ivan Gorban, Rahul Laptev, Mubarak Sukthankar, Shah","year":"2017","venue":"Computer Vision and Image Understanding","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kolesnikov et al., 2018] Alexander Kolesnikov, Christoph H Lampert, and Vittorio Ferrari. Detect- ing visual relationships using box attention. arXiv: Computer Vision and Pattern Recognition, 2018.","title":"Detecting visual relationships using box attention","authors":"Alexander Kolesnikov, Christoph H Lampert, Vittorio Ferrari","year":"2018","venue":"arXiv: Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2018] Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Thirty-Second AAAI Confer- ence on Artificial Intelligence, 2018.","title":"Deeper insights into graph convolutional networks for semi-supervised learning","authors":"Li","year":"2018","venue":"Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2019] Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang, Yanfeng Wang, and Cewu Lu. Transferable interactiveness knowledge for human-object interaction detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 3585-3594, 2019.","title":"Transferable interactiveness knowledge for human-object interaction detection","authors":"Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang, Yanfeng Wang, Cewu Lu","year":"2019","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Qi et al., 2018] Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, and Songchun Zhu. Learning human- object interactions by graph parsing neural networks. eu- ropean conference on computer vision, pages 407-423, 2018.","title":"Learning humanobject interactions by graph parsing neural networks","authors":"Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, Songchun Zhu","year":"2018","venue":"european conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ren et al., 2017] Shaoqing Ren, Kaiming He, Ross B Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time ob- ject detection with region proposal networks. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 39(6):1137-1149, 2017.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2017","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sharma et al., 2015] Shikhar Sharma, Ryan Kiros, and Rus- lan Salakhutdinov. Action recognition using visual atten- tion. international conference on learning representations, 2015.","title":"Action recognition using visual attention","authors":"Shikhar Sharma, Ryan Kiros, Ruslan Salakhutdinov","year":"2015","venue":"international conference on learning representations","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shen et al., 2018] Liyue Shen, Serena Yeung, Judy Hoff- man, Greg Mori, and Li Feifei. Scaling human-object interaction recognition through zero-shot learning. work- shop on applications of computer vision, pages 1568- 1576, 2018.","title":"Scaling human-object interaction recognition through zero-shot learning. workshop on applications of computer vision","authors":"Shen","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Wan et al., 2019] Bo Wan, Desen Zhou, Yongfei Liu, Rongjie Li, and Xuming He. Pose-aware multi-level feature network for human object interaction detection. In Proceedings of the IEEE International Conference on Computer Vision, pages 9469-9478, 2019.","title":"Pose-aware multi-level feature network for human object interaction detection","authors":"Bo Wan, Desen Zhou, Yongfei Liu, Rongjie Li, Xuming He","year":"2019","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wang et al., 2019] Tiancai Wang, Rao Muhammad Anwer, Muhammad Haris Khan, Fahad Shahbaz Khan, Yanwei Pang, Ling Shao, and Jorma Laaksonen. Deep contextual attention for human-object interaction detection. In Pro- ceedings of the IEEE International Conference on Com- puter Vision, pages 5694-5702, 2019.","title":"Deep contextual attention for human-object interaction detection","authors":"Wang","year":"2019","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xu et al., 2019] Bingjie Xu, Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S Kankanhalli. Interact as you in- tend: Intention-driven human-object interaction detection. IEEE Transactions on Multimedia, 2019.","title":"Interact as you intend: Intention-driven human-object interaction detection","authors":"Xu","year":"2019","venue":"IEEE Transactions on Multimedia","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang et al., 2018] Maoke Yang, Kun Yu, Chi Zhang, Zhi- wei Li, and Kuiyuan Yang. Denseaspp for semantic seg- mentation in street scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3684-3692, 2018.","title":"Denseaspp for semantic segmentation in street scenes","authors":"Yang","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yatskar et al., 2016] Mark Yatskar, Luke Zettlemoyer, and Ali Farhadi. Situation recognition: Visual semantic role labeling for image understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 5534-5542, 2016.","title":"Situation recognition: Visual semantic role labeling for image understanding","authors":"Mark Yatskar, Luke Zettlemoyer, Ali Farhadi","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhou and Chi, 2019] Penghao Zhou and Mingmin Chi. Re- lation parsing neural network for human-object interaction detection. In Proceedings of the IEEE International Con- ference on Computer Vision, pages 843-851, 2019.","title":"Penghao Zhou and Mingmin Chi. Relation parsing neural network for human-object interaction detection","authors":"Chi Zhou","year":"2019","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}