{"type":"ScholarlyArticle","title":"Action-Guided Attention Mining and Relation Reasoning Network for Human-Object Interaction Detection","authors":[{"name":"Xue Lin","slug":"xue-lin","given":"Xue","family":"Lin","affiliation":"Beijing Key Laboratory of Traffic Data Analysis and Mining , Beijing Jiaotong University , China"},{"name":"Qi Zou","slug":"qi-zou","given":"Qi","family":"Zou","affiliation":"Beijing Key Laboratory of Traffic Data Analysis and Mining , Beijing Jiaotong University , China"},{"name":"Xixia Xu","slug":"xixia-xu","given":"Xixia","family":"Xu","affiliation":"Beijing Key Laboratory of Traffic Data Analysis and Mining , Beijing Jiaotong University , China"}],"abstract":"Human-object interaction (HOI) detection is important to understand human-centric scenes and is challenging due to subtle difference between fine-grained actions, and multiple co-occurring interactions. Most approaches tackle the problems by considering the multi-stream information and even introducing extra knowledge, which suffer from a huge combination space and the non-interactive pair domination problem. In this paper, we propose an Action-Guided attention mining and Relation Reasoning (AGRR) network to solve the problems. Relation reasoning on human-object pairs is performed by exploiting contextual compatibility consistency among pairs to filter out the non-interactive combinations. To better discriminate the subtle difference between fine-grained actions, an action-aware attention based on class activation map is proposed to mine the most relevant features for recognizing HOIs. Extensive experiments on V-COCO and HICO-DET datasets demonstrate the effectiveness of the proposed model compared with the state-of-the-art approaches.","doi":"10.24963/ijcai.2020/154","url":"https://ijcai.org/proceedings/2020/154","pdf":"https://ijcai.org/proceedings/2020/0154.pdf","markdown":"https://ijcai.org/proceedings/2020/154.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/154","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":154,"pages":{"first":1104,"last":1110},"keywords":["Machine Learning: Relational Learning","Machine Learning: Deep Learning: Convolutional networks","Computer Vision: Action Recognition","Computer Vision: Structural and Model-Based Approaches, Knowledge Representation and Reasoning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24159/file_24159.m3u8","length_sec":263,"provider":"hls"}],"references":[{"position":1,"raw":"[Chao et al., 2018] Yuwei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human- object interactions. In WACV, pages 381-389, 2018.","title":"Learning to detect humanobject interactions","authors":"Yuwei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, Jia Deng","year":"2018","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Fang et al., 2018] Haoshu Fang, Jinkun Cao, Yuwing Tai, and Cewu Lu. Pairwise body-part attention for recognizing human-object interactions. In ECCV, pages 52-68, 2018.","title":"Pairwise body-part attention for recognizing human-object interactions","authors":"Fang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Fukui et al., 2019] Hiroshi Fukui, Tsubasa Hirakawa, Takayoshi Yamashita, and Hironobu Fujiyoshi. Attention branch network: Learning of attention mechanism for visual explanation. In CVPR, pages 10705-10714, 2019.","title":"Attention branch network: Learning of attention mechanism for visual explanation","authors":"Hiroshi Fukui, Tsubasa Hirakawa, Takayoshi Yamashita, Hironobu Fujiyoshi","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gao et al., 2018] Chen Gao, Yuliang Zou, and Jiabin Huang. ican: Instance-centric attention network for human-object interaction detection. In BMVC, 2018.","title":"ican: Instance-centric attention network for human-object interaction detection","authors":"Chen Gao, Yuliang Zou, Jiabin Huang","year":"2018","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Girshick et al., 2018] Ross Girshick, Ilija Radosavovic, Georgia Gkioxari, Piotr Dollár, and Kaiming He. De- tectron. https://github.com/facebookresearch/detectron, 2018.","title":"Detectron","authors":"Ross Girshick, Ilija Radosavovic, Georgia Gkioxari, Piotr Dollár, Kaiming He","year":null,"venue":null,"doi":null,"url":"https://github.com/facebookresearch/detectron,2018","internal_url":null},{"position":6,"raw":"[Gkioxari et al., 2018] Georgia Gkioxari, Ross Girshick, Pi- otr Dollar, and Kaiming He. Detecting and recognizing human-object interactions. In CVPR, pages 8359-8367, 2018.","title":"Georgia Gkioxari, Ross Girshick, Piotr Dollar, and Kaiming He. Detecting and recognizing human-object interactions","authors":"Gkioxari","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gupta and Malik, 2015] Saurabh Gupta and Jitendra Malik. Visual semantic role labeling. arXiv:1505.04474, 2015.","title":"Visual semantic role labeling","authors":"Saurabh Gupta, Jitendra Malik","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Heilbron et al., 2015] Fabian Caba Heilbron, Victor Escor- cia, Bernard Ghanem, and Juan Carlos Niebles. Activi- tynet: A large-scale video benchmark for human activity understanding. In CVPR, pages 961-970, 2015.","title":"Activitynet: A large-scale video benchmark for human activity understanding","authors":"Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, Juan Carlos Niebles","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kim et al., 2019] Daesik Kim, Gyuejeong Lee, Jisoo Jeong, and Nojun Kwak. Tell me what they're holding: Weakly- supervised object detection with transferable knowledge from human-object interaction. arXiv:1911.08141, 2019.","title":"Tell me what they're holding: Weaklysupervised object detection with transferable knowledge from human-object interaction","authors":"Kim","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kipf and Welling, 2017] Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.","title":"Semi-supervised classification with graph convolutional networks","authors":"Thomas Kipf, Max Welling","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2018] Kunpeng Li, Ziyan Wu, Kuanchuan Peng, Jan Ernst, and Yun Fu. Tell me where to look: Guided attention inference network. In CVPR, pages 9215-9223, 2018.","title":"Tell me where to look: Guided attention inference network","authors":"Kunpeng Li, Ziyan Wu, Kuanchuan Peng, Jan Ernst, Yun Fu","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2019a] Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu. Visual semantic reasoning for image-text matching. In ICCV, 2019.","title":"Visual semantic reasoning for image-text matching","authors":"Li","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2019b] Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu. Relation-aware graph attention network for visual question answering. In ICCV, 2019.","title":"Relation-aware graph attention network for visual question answering","authors":"Li","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2019c] Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang, Yan-Feng Wang, and Cewu Lu. Transferable interactiveness prior for human- object interaction detection. In CVPR, 2019.","title":"Transferable interactiveness prior for humanobject interaction detection","authors":"Li","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2014] Tsungyi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In ECCV, pages 740-755, 2014.","title":"Microsoft coco: Common objects in context","authors":"Lin","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin et al., 2017] Tsungyi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Fea- ture pyramid networks for object detection. In CVPR, pages 936-944, 2017.","title":"Feature pyramid networks for object detection","authors":"Lin","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mallya and Lazebnik, 2016] Arun Mallya and Svetlana Lazebnik. Learning models for actions and person-object interactions with transfer to question answering. In ECCV, pages 414-428, 2016.","title":"Learning models for actions and person-object interactions with transfer to question answering","authors":"Lazebnik Mallya, Arun Mallya, Svetlana Lazebnik","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Qi et al., 2018] Siyuan Qi, Wenguan Wang, Baoxiong Ji- a, Jianbing Shen, and Songchun Zhu. Learning human- object interactions by graph parsing neural networks. In ECCV, pages 407-423, 2018.","title":"Learning humanobject interactions by graph parsing neural networks","authors":"Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, Songchun Zhu","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: towards real-time ob- ject detection with region proposal networks. 2015:91-99, 2015.","title":"Faster r-cnn: towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Selvaraju et al., 2017a] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanation- s from deep networks via gradient-based localization. In ICCV, pages 618-626, 2017.","title":"Grad-cam: Visual explanations from deep networks via gradient-based localization","authors":"Selvaraju","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Selvaraju et al., 2017b] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanation- s from deep networks via gradient-based localization. In ICCV, pages 618-626, 2017.","title":"Grad-cam: Visual explanations from deep networks via gradient-based localization","authors":"Selvaraju","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Velickovic et al., 2018] Petar Velickovic, Guillem Cucurul- l, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.","title":"Graph attention networks","authors":"Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wan et al., 2019] Bo Wan, Desen Zhou, Yongfei Liu, Rongjie Li, and Xuming He. Pose-aware multi-level fea- ture network for human object interaction detection. In ICCV, 2019.","title":"Pose-aware multi-level feature network for human object interaction detection","authors":"Bo Wan, Desen Zhou, Yongfei Liu, Rongjie Li, Xuming He","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2019] Tiancai Wang, Rao Muhammad Anwer, Muhammad Haris Khan, Fahad Shahbaz Khan, Yanwei Pang, Ling Shao, and Jorma Laaksonen. Deep contextual attention for human-object interaction detection. In ICCV, 2019.","title":"Deep contextual attention for human-object interaction detection","authors":"Wang","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2019] Bingjie Xu, Yongkang Wong, Junnan Li, Qi Zhao, and Mohan S Kankanhalli. Learning to detec- t human-object interactions with knowledge. In CVPR, pages 2019-2028, 2019.","title":"Learning to detect human-object interactions with knowledge","authors":"Xu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhou and Chi, 2019] Penghao Zhou and Mingmin Chi. Re- lation parsing neural network for human-object interaction detection. In ICCV, pages 843-851, 2019.","title":"Penghao Zhou and Mingmin Chi. Relation parsing neural network for human-object interaction detection","authors":"Chi Zhou","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhou et al., 2016a] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, pages 2921-2929, 2016.","title":"Learning deep features for discriminative localization","authors":"Zhou","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhou et al., 2016b] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, pages 2921-2929, 2016.","title":"Learning deep features for discriminative localization","authors":"Zhou","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}