{"type":"ScholarlyArticle","title":"Explicit Knowledge-based Reasoning for Visual Question Answering","authors":[{"name":"Peng Wang","slug":"peng-wang","given":"Peng","family":"Wang","affiliation":"Northwestern Polytechnical University , China; The University of Adelaide , Australia,"},{"name":"Qi Wu","slug":"qi-wu","given":"Qi","family":"Wu","affiliation":"Australian Centre for Robotic Vision"},{"name":"Chunhua Shen","slug":"chunhua-shen","given":"Chunhua","family":"Shen","affiliation":"The University of Adelaide , Australia,; Australian Centre for Robotic Vision"},{"name":"Anthony Dick","slug":"anthony-dick","given":"Anthony","family":"Dick","affiliation":"The University of Adelaide , Australia,"},{"name":"Anton van den Hengel","slug":"anton-van-den-hengel","given":"Anton","family":"van den Hengel","affiliation":"The University of Adelaide , Australia,; Australian Centre for Robotic Vision"}],"abstract":"We describe a method for visual question answering which is capable of reasoning about an image on the basis of information extracted from a large-scale knowledge base. The method not only answers natural language questions using concepts not contained in the image, but can explain the reasoning by which it developed its answer. It is capable of answering far more complex questions than the predominant long short-term memory-based approach, and outperforms it significantly in testing. We also provide a dataset and a protocol by which to evaluate general visual question answering methods.","doi":"10.24963/ijcai.2017/179","url":"https://ijcai.org/proceedings/2017/179","pdf":"https://ijcai.org/proceedings/2017/0179.pdf","markdown":"https://ijcai.org/proceedings/2017/179.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/179","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Knowledge Representation, Reasoning, and Logic","track":"Main track","number":179,"pages":{"first":1290,"last":1296},"keywords":["Natural Language Processing: Question Answering","Knowledge Representation, Reasoning, and Logic: Reasoning about Knowlege and Belief","Knowledge Representation, Reasoning, and Logic: Common-Sense Reasoning"],"videos":[],"references":[{"position":1,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Ji- asen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. VQA: Visual Question Answering. In Proc. ICCV, 2015.","title":"VQA: Visual Question Answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh","year":"2015","venue":"Proc. ICCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Auer et al., 2007] Sören Auer, Christian Bizer, Georgi Ko- bilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. DBpedia: A nucleus for a web of open data. Springer, 2007.","title":"DBpedia: A nucleus for a web of open data","authors":"Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, Zachary Ives","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Berant et al., 2013] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic Parsing on Freebase from Question-Answer Pairs. In Proc. EMNLP, 2013.","title":"Semantic Parsing on Freebase from Question-Answer Pairs","authors":"Jonathan Berant, Andrew Chou, Roy Frostig, Percy Liang","year":"2013","venue":"Proc. EMNLP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bird et al., 2009] Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python. O'Reilly Media, Inc., 2009.","title":"Natural language processing with Python","authors":"Steven Bird, Ewan Klein, Edward Loper","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bollacker et al., 2008] Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collaboratively created graph database for structuring human knowledge. In ACM SIGMOD, 2008.","title":"Freebase: a collaboratively created graph database for structuring human knowledge","authors":"Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, Jamie Taylor","year":"2008","venue":"ACM SIGMOD","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bordes et al., 2015] Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. Large-scale simple question answering with memory networks. In Proc. ICLR, 2015.","title":"Large-scale simple question answering with memory networks","authors":"Antoine Bordes, Nicolas Usunier, Sumit Chopra, Jason Weston","year":"2015","venue":"Proc. ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cyganiak et al., 2014] Richard Cyganiak, David Wood, and Markus Lanthaler. Rdf 1.1 concepts and abstract syntax, 2014. http://www.w3.org/standards/techs/ rdf.","title":"Rdf 1.1 concepts and abstract syntax","authors":"Richard Cyganiak, David Wood, Markus Lanthaler","year":"2014","venue":null,"doi":null,"url":"http://www.w3.org/standards/techs/rdf","internal_url":null},{"position":8,"raw":"[Etzioni et al., 2011] Oren Etzioni, Anthony Fader, Janara Christensen, Stephen Soderland, and Mausam Mausam. Open Information Extraction: The Second Generation. In Proc. IJCAI, 2011.","title":"Open Information Extraction: The Second Generation","authors":"Oren Etzioni, Anthony Fader, Janara Christensen, Stephen Soderland, Mausam Mausam","year":"2011","venue":"Proc. IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2011/12"},{"position":9,"raw":"[Fukui et al., 2016] Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for vi- sual question answering and visual grounding. In Proc. EMNLP, 2016.","title":"Multimodal compact bilinear pooling for visual question answering and visual grounding","authors":"Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, Marcus Rohrbach","year":"2016","venue":"Proc. EMNLP","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Geman et al., 2015] Donald Geman, Stuart Geman, Neil Hallonquist, and Laurent Younes. Visual Turing test for computer vision systems. Proc. NAS, 112(12):3618-3623, 2015.","title":"Visual Turing test for computer vision systems","authors":"Donald Geman, Stuart Geman, Neil Hallonquist, Laurent Younes","year":"2015","venue":"Proc. NAS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Höffner et al., 2016] Konrad Höffner, Sebastian Walter, Edgard Marx, Ricardo Usbeck, Jens Lehmann, and Axel- Cyrille Ngonga Ngomo. Survey on challenges of question answering in the semantic web. Semantic Web, 2016.","title":"Survey on challenges of question answering in the semantic web","authors":"Konrad Höffner, Sebastian Walter, Edgard Marx, Ricardo Usbeck, Jens Lehmann, Axel-Cyrille Ngonga Ngomo","year":"2016","venue":"Semantic Web","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krishna et al., 2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowd- sourced dense image annotations. Int. J. Comput. Vision, 123(1):32-73, 2017.","title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","authors":"Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, Li Fei-Fei","year":"2017","venue":"Int. J. Comput. Vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Com- mon objects in context. In Proc. ECCV, 2014.","title":"Microsoft COCO: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"Proc. ECCV","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lu et al., 2016] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. In Proc. NIPS, 2016.","title":"Hierarchical question-image co-attention for visual question answering","authors":"Lu","year":"2016","venue":"Proc. NIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mahdisoltani et al., 2015] Farzaneh Mahdisoltani, Joanna Biega, and Fabian Suchanek. YAGO3: A knowledge base from multilingual Wikipedias. In CIDR, 2015.","title":"YAGO3: A knowledge base from multilingual Wikipedias","authors":"Farzaneh Mahdisoltani, Joanna Biega, Fabian Suchanek","year":"2015","venue":"CIDR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Malinowski and Fritz, 2014a] Mateusz Malinowski and Mario Fritz. A multi-world approach to question answer- ing about real-world scenes based on uncertain input. In Proc. NIPS, 2014.","title":"A multi-world approach to question answering about real-world scenes based on uncertain input","authors":"Fritz ; Malinowski, Mateusz Malinowski, Mario Fritz","year":"2014","venue":"Proc. NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Malinowski and Fritz, 2014b] Mateusz Malinowski and Mario Fritz. Towards a Visual Turing Challenge. In NIPS Workshop on Learning Semantics, 2014.","title":"Towards a Visual Turing Challenge","authors":"Fritz ; Malinowski, Mateusz Malinowski, Mario Fritz","year":"2014","venue":"NIPS Workshop on Learning Semantics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Malinowski et al., 2015] Mateusz Malinowski, Marcus Rohrbach, and Mario Fritz. Ask Your Neurons: A Neural-based Approach to Answering Questions about Images. In Proc. ICCV, 2015.","title":"Ask Your Neurons: A Neural-based Approach to Answering Questions about Images","authors":"Mateusz Malinowski, Marcus Rohrbach, Mario Fritz","year":"2015","venue":"Proc. ICCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ren et al., 2015] Mengye Ren, Ryan Kiros, and Richard Zemel. Image Question Answering: A Visual Semantic Embedding Model and a New Dataset. In Proc. NIPS, 2015.","title":"Image Question Answering: A Visual Semantic Embedding Model and a New Dataset","authors":"Ren","year":"2015","venue":"Proc. NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rocktäschel et al., 2016] Tim Rocktäschel, Edward Grefen- stette, Karl Moritz Hermann, Tomáš Kočiskỳ, and Phil Blunsom. Reasoning about Entailment with Neural At- tention. In Proc. ICLR, 2016.","title":"Reasoning about Entailment with Neural Attention","authors":"Rocktäschel","year":"2016","venue":"Proc. ICLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Schmachtenberg et al., 2014] Max Schmachtenberg, C Bizer, and H Paulheim. State of the LOD Cloud 2014, 2014. http://linkeddatacatalog.dws. informatik.unimannheim.de/state.","title":"State of the LOD Cloud 2014","authors":"Max Schmachtenberg, H Bizer, Paulheim","year":"2014","venue":null,"doi":null,"url":"http://linkeddatacatalog.dws.informatik.unimannheim.de/state","internal_url":null},{"position":23,"raw":"[Wang et al., 2017] Peng Wang, Qi Wu, Chunhua Shen, and Anton van den Hengel. The vqa-machine: Learning how to use existing vision algorithms to answer new questions. In Proc. CVPR, 2017.","title":"The vqa-machine: Learning how to use existing vision algorithms to answer new questions","authors":"Peng Wang, Qi Wu, Chunhua Shen, Anton Van Den, Hengel","year":"2017","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wu et al., 2016] Qi Wu, Peng Wang, Chunhua Shen, An- thony Dick, and Anton van den Hengel. Ask Me Anything: Free-form Visual Question Answering Based on Knowl- edge from External Sources. In Proc. CVPR, 2016.","title":"Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources","authors":"Qi Wu, Peng Wang, Chunhua Shen, Anthony Dick, Anton Van Den, Hengel","year":"2016","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu and Saenko, 2016] Huijuan Xu and Kate Saenko. Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering. In Proc. ECCV, 2016.","title":"Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering","authors":"Saenko Xu, Huijuan Xu, Kate Saenko","year":"2016","venue":"Proc. ECCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. In Proc. ICML, 2015.","title":"Show, Attend and Tell: Neural Image Caption Generation with Visual Attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio","year":"2015","venue":"Proc. ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yang et al., 2016] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked Attention Net- works for Image Question Answering. In Proc. CVPR, 2016.","title":"Stacked Attention Networks for Image Question Answering","authors":"Yang","year":"2016","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yu et al., 2015] Licheng Yu, Eunbyung Park, Alexander C Berg, and Tamara L Berg. Visual madlibs: Fill in the blank image generation and question answering. In Proc. ICCV, 2015.","title":"Visual madlibs: Fill in the blank image generation and question answering","authors":"Licheng Yu, Eunbyung Park, Alexander C Berg, Tamara L Berg","year":"2015","venue":"Proc. ICCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhu et al., 2015] Yuke Zhu, Ce Zhang, Christopher Ré, and Li Fei-Fei. Building a Large-scale Multimodal Knowledge Base System for Answering Visual Queries. arXiv preprint arXiv:1507.05670, 2015.","title":"Building a Large-scale Multimodal Knowledge Base System for Answering Visual Queries","authors":"Yuke Zhu, Ce Zhang, Christopher Ré, Li Fei-Fei","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}