{"type":"ScholarlyArticle","title":"Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answering","authors":[{"name":"Zihao Zhu","slug":"zihao-zhu","given":"Zihao","family":"Zhu","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences , Beijing , China; School of Cyber Security , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Jing Yu","slug":"jing-yu","given":"Jing","family":"Yu","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences , Beijing , China; School of Cyber Security , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Yujing Wang","slug":"yujing-wang","given":"Yujing","family":"Wang","affiliation":"Microsoft Research Asia , Beijing , China"},{"name":"Yajing Sun","slug":"yajing-sun","given":"Yajing","family":"Sun","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences , Beijing , China; School of Cyber Security , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Yue Hu","slug":"yue-hu","given":"Yue","family":"Hu","affiliation":"Institute of Information Engineering , Chinese Academy of Sciences , Beijing , China; School of Cyber Security , University of Chinese Academy of Sciences , Beijing , China"},{"name":"Qi Wu","slug":"qi-wu","given":"Qi","family":"Wu","affiliation":"University of Adelaide , Australia"}],"abstract":"Fact-based Visual Question Answering (FVQA) requires external knowledge beyond the visible content to answer questions about an image. This ability is challenging but indispensable to achieve general VQA. One limitation of existing FVQA solutions is that they jointly embed all kinds of information without fine-grained selection, which introduces unexpected noises for reasoning the final answer. How to capture the question-oriented and information-complementary evidence remains a key challenge to solve the problem. In this paper, we depict an image by a multi-modal heterogeneous graph, which contains multiple layers of information corresponding to the visual, semantic and factual features. On top of the multi-layer graph representations, we propose a modality-aware heterogeneous graph convolutional network to capture evidence from different layers that is most relevant to the given question. Specifically, the intra-modal graph convolution selects evidence from each modality and cross-modal graph convolution aggregates relevant information across different graph layers. By stacking this process multiple times, our model performs iterative reasoning across three modalities and predicts the optimal answer by analyzing all question-oriented evidence. We achieve a new state-of-the-art performance on the FVQA task and demonstrate the effectiveness and interpretability of our model with extensive experiments.","doi":"10.24963/ijcai.2020/153","url":"https://ijcai.org/proceedings/2020/153","pdf":"https://ijcai.org/proceedings/2020/0153.pdf","markdown":"https://ijcai.org/proceedings/2020/153.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/153","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":153,"pages":{"first":1097,"last":1103},"keywords":["Computer Vision: Language and Vision","Machine Learning: Knowledge-based Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24821/file_24821.m3u8","length_sec":281,"provider":"hls"}],"references":[{"position":1,"raw":"[Anderson et al., 2016] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In ECCV, pages 382-398, 2016.","title":"Spice: Semantic propositional image caption evaluation","authors":"Anderson","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anderson et al., 2018] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for im- age captioning and visual question answering. In CVPR, pages 6319-6328, 2018.","title":"Bottom-up and top-down attention for image captioning and visual question answering","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Ji- asen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In ICCV, pages 2425-2433, 2015.","title":"Vqa: Visual question answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Battaglia et al., 2018] Peter W Battaglia, Jessica B Ham- rick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261, 2018.","title":"Relational inductive biases, deep learning, and graph networks","authors":"Jessica B Peter W Battaglia, Victor Hamrick, Alvaro Bapst, Vinicius Sanchez-Gonzalez, Mateusz Zambaldi, Andrea Malinowski, David Tacchetti, Adam Raposo, Ryan Santoro, Faulkner","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ben-Younes et al., 2019] Hedi Ben-Younes, Rémi Cadene, Nicolas Thome, and Matthieu Cord. Block: Bilinear su- perdiagonal fusion for visual question answering and vi- sual relationship detection. In AAAI, pages 8102-8109, 2019.","title":"Block: Bilinear superdiagonal fusion for visual question answering and visual relationship detection","authors":"Ben-Younes","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cadene et al., 2019] Remi Cadene, Hedi Ben-Younes, Matthieu Cord, and Nicolas Thome. Murel: Multimodal relational reasoning for visual question answering. In CVPR, pages 1989-1998, 2019.","title":"Murel: Multimodal relational reasoning for visual question answering","authors":"Cadene","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gilmer et al., 2017] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In ICML, pages 1263-1272, 2017.","title":"Neural message passing for quantum chemistry","authors":"Justin Gilmer, S Samuel, Patrick F Schoenholz, Oriol Riley, George E Vinyals, Dahl","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2019a] Linmei Hu, Tianchi Yang, Chuan Shi, Houye Ji, and Xiaoli Li. Heterogeneous graph attention networks for semi-supervised short text classification. In EMNLP, pages 4823-4832, 2019.","title":"Heterogeneous graph attention networks for semi-supervised short text classification","authors":"Hu","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2019b] Ronghang Hu, Anna Rohrbach, Trevor Darrell, and Kate Saenko. Language-conditioned graph networks for relational reasoning. In ICCV, pages 10294- 10303, 2019.","title":"Language-conditioned graph networks for relational reasoning","authors":"Hu","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jiang et al., 2020] Xiaoze Jiang, Jing Yu, Zengchang Qin, Yingying Zhuang, Xingxing Zhang, Yue Hu, and Qi Wu. Dualvd: An adaptive dual encoding model for deep visual understanding in visual dialogue. In AAAI, 2020.","title":"Dualvd: An adaptive dual encoding model for deep visual understanding in visual dialogue","authors":"Xiaoze Jiang, Jing Yu, Zengchang Qin, Yingying Zhuang, Xingxing Zhang, Yue Hu, Qi Wu","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Johnson et al., 2016] Justin Johnson, Andrej Karpathy, and Li Fei-Fei. Densecap: Fully convolutional localization networks for dense captioning. In CVPR, pages 4565- 4574, 2016.","title":"Densecap: Fully convolutional localization networks for dense captioning","authors":"Justin Johnson, Andrej Karpathy, Li Fei-Fei","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2019a] Hui Li, Peng Wang, Chunhua Shen, and Anton van den Hengel. Visual question answering as read- ing comprehension. In CVPR, pages 6319-6328, 2019.","title":"Visual question answering as reading comprehension","authors":"Li","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2019b] Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu. Relation-aware graph attention network for visual question answering. In ICCV, pages 10313-10322, 2019.","title":"Relation-aware graph attention network for visual question answering","authors":"Li","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lu et al., 2016] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. In NeurIPS, pages 289-297, 2016.","title":"Hierarchical question-image co-attention for visual question answering","authors":"Lu","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Malinowski et al., 2015] Mateusz Malinowski, Marcus Rohrbach, and Mario Fritz. Ask your neurons: A neural- based approach to answering questions about images. In ICCV, pages 1-9, 2015.","title":"Ask your neurons: A neuralbased approach to answering questions about images","authors":"Mateusz Malinowski, Marcus Rohrbach, Mario Fritz","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Narasimhan and Schwing, 2018] Medhini Narasimhan and Alexander G Schwing. Straight to the facts: Learning knowledge base retrieval for factual visual question an- swering. In ECCV, pages 451-468, 2018.","title":"Straight to the facts: Learning knowledge base retrieval for factual visual question answering","authors":"Schwing Narasimhan, Medhini Narasimhan, Alexander G Schwing","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Narasimhan et al., 2018] Medhini Narasimhan, Svetlana Lazebnik, and Alexander Schwing. Out of the box: Rea- soning with graph convolution nets for factual visual ques- tion answering. In NeurIPS, pages 2654-2665, 2018.","title":"Out of the box: Reasoning with graph convolution nets for factual visual question answering","authors":"Narasimhan","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ren et al., 2017] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time ob- ject detection with region proposal networks. 39(6):1137- 1149, 2017.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schlichtkrull et al., 2018] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convo- lutional networks. In ESWC, pages 593-607, 2018.","title":"Modeling relational data with graph convolutional networks","authors":"Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den, Ivan Berg, Max Titov, Welling","year":"2018","venue":"ESWC","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2017] Peng Wang, Qi Wu, Chunhua Shen, An- thony R. Dick, and Anton van den Hengel. Explicit knowledge-based reasoning for visual question answering. In IJCAI, pages 1290-1296, 2017.","title":"Explicit knowledge-based reasoning for visual question answering","authors":"Peng Wang, Qi Wu, Chunhua Shen, Anthony R Dick, Anton Van Den, Hengel","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/179"},{"position":22,"raw":"[Wang et al., 2018] Peng Wang, Qi Wu, Chunhua Shen, An- thony Dick, and Anton van den Hengel. Fvqa: Fact-based visual question answering. TPAMI, 40(10):2413-2427, 2018.","title":"Fvqa: Fact-based visual question answering","authors":"Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, Anton Van Den, Hengel","year":"2018","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2019a] Peng Wang, Qi Wu, Jiewei Cao, Chun- hua Shen, Lianli Gao, and Anton van den Hengel. Neigh- bourhood watch: Referring expression comprehension via language-guided graph attention networks. In CVPR, pages 1960-1968, 2019.","title":"Neighbourhood watch: Referring expression comprehension via language-guided graph attention networks","authors":"Wang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2019b] Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. Hetero- geneous graph attention network. In WWW, pages 2022- 2032, 2019.","title":"Heterogeneous graph attention network","authors":"Wang","year":"2019","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wu et al., 2019] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S Yu. A comprehensive survey on graph neural networks. arXiv preprint arXiv:1901.00596, 2019.","title":"A comprehensive survey on graph neural networks","authors":"Wu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2016] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention net- works for image question answering. In CVPR, pages 21- 29, 2016.","title":"Stacked attention networks for image question answering","authors":"Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Smola","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}