{"type":"ScholarlyArticle","title":"Integrating Knowledge and Reasoning in Image Understanding","authors":[{"name":"Somak Aditya","slug":"somak-aditya","given":"Somak","family":"Aditya","affiliation":"Bigdata Experience Lab , Adobe Research , India"},{"name":"Yezhou Yang","slug":"yezhou-yang","given":"Yezhou","family":"Yang","affiliation":"CIDSE , Arizona State University , USA"},{"name":"Chitta Baral","slug":"chitta-baral","given":"Chitta","family":"Baral","affiliation":"CIDSE , Arizona State University , USA"}],"abstract":"Deep learning based data-driven approaches have been successfully applied in various image understanding\r\napplications ranging from object recognition, semantic segmentation to visual question answering.\r\nHowever, the lack of knowledge integration as well as higher-level reasoning capabilities with the methods still pose a hindrance. In this work, we present a brief survey of a few representative\r\nreasoning mechanisms, knowledge integration methods and their corresponding image understanding applications developed by various groups of researchers, approaching the problem from a variety of angles. Furthermore, we discuss upon key efforts on integrating external knowledge with neural networks. Taking cues from these efforts, we conclude by discussing potential pathways to improve reasoning capabilities.","doi":"10.24963/ijcai.2019/873","url":"https://ijcai.org/proceedings/2019/873","pdf":"https://ijcai.org/proceedings/2019/0873.pdf","markdown":"https://ijcai.org/proceedings/2019/873.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/873","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Survey track","track":"Survey track","number":873,"pages":{"first":6252,"last":6259},"keywords":["Machine Learning: Interpretability","Computer Vision: Language and Vision","Knowledge Representation and Reasoning: Common-Sense Reasoning"],"videos":[],"references":[{"position":1,"raw":"[Aditya et al., 2017] Somak Aditya, Yezhou Yang, Chitta Baral, Yiannis Aloimonos, and Cornelia Fermüller. Image understanding using vision and reasoning through scene description graph. Computer Vision and Image Under- standing, pages 33-45, 2017.","title":"Image understanding using vision and reasoning through scene description graph","authors":"Somak Aditya, Yezhou Yang, Chitta Baral, Yiannis Aloimonos, Cornelia Fermüller","year":"2017","venue":"Computer Vision and Image Understanding","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Aditya et al., 2018a] Somak Aditya, Yezhou Yang, and Chitta Baral. Explicit Reasoning over End-to-End Neural Architectures for Visual Question Answering. In AAAI, pages 629-637, 2018.","title":"Explicit Reasoning over End-to-End Neural Architectures for Visual Question Answering","authors":"Aditya","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Aditya et al., 2018b] Somak Aditya, Yezhou Yang, Chitta Baral, and Yiannis Aloimonos. Combining knowledge and reasoning through probabilistic soft logic for image puz- zle solving. In UAI 2018, pages 238-248. Association For Uncertainty in Artificial Intelligence (AUAI), 2018.","title":"Combining knowledge and reasoning through probabilistic soft logic for image puzzle solving","authors":"Aditya","year":"2018","venue":"UAI 2018","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Aditya et al., 2019] Somak Aditya, Rudra Saha, Yezhou Yang, and Chitta Baral. Spatial knowledge distillation to aid visual reasoning. IEEE Winter Conference on Applica- tions of Computer Vision (WACV), pages 227-235, 2019.","title":"Spatial knowledge distillation to aid visual reasoning","authors":"Somak Aditya, Rudra Saha, Yezhou Yang, Chitta Baral","year":"2019","venue":"IEEE Winter Conference on Applications of Computer Vision (WACV)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Baader et al., 2003] Franz Baader, Diego Calvanese, Deb- orah McGuinness, Peter Patel-Schneider, and Daniele Nardi. The description logic handbook: Theory, imple- mentation and applications. Cambridge university press, 2003.","title":"The description logic handbook: Theory, implementation and applications","authors":"Baader","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Bach et al., 2017] Stephen H Bach, Matthias Broecheler, Bert Huang, and Lise Getoor. Hinge-loss markov ran- dom fields and probabilistic soft logic. Journal of Machine Learning Research, 18:1-67, 2017.","title":"Hinge-loss markov random fields and probabilistic soft logic","authors":"Matthias Stephen H Bach, Bert Broecheler, Lise Huang, Getoor","year":"2017","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cadene et al., 2019] Remi Cadene, Hedi Ben-Younes, Nicolas Thome, and Matthieu Cord. Murel: Multimodal Relational Reasoning for Visual Question Answering. In IEEE Conference on Computer Vision and Pattern Recognition CVPR, 2019.","title":"Murel: Multimodal Relational Reasoning for Visual Question Answering","authors":"Cadene","year":"2019","venue":"IEEE Conference on Computer Vision and Pattern Recognition CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dasiopoulou et al., 2009] Stamatia Dasiopoulou, Ioannis Kompatsiaris, and Michael G Strintzis. Applying fuzzy dls in the extraction of image semantics. In Journal on Data Semantics XIV, pages 105-132. Springer, 2009.","title":"Applying fuzzy dls in the extraction of image semantics","authors":"Stamatia Dasiopoulou, Ioannis Kompatsiaris, Michael G Strintzis","year":"2009","venue":"Journal on Data Semantics XIV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Davis and Marcus, 2015] Ernest Davis and Gary Marcus. Commonsense reasoning and commonsense knowledge in artificial intelligence. Commun. ACM, 58(9):92-103, Au- gust 2015.","title":"Commonsense reasoning and commonsense knowledge in artificial intelligence","authors":"Marcus ; Davis, Ernest Davis, Gary Marcus","year":"2015","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[de Boer et al., 2015] Maaike de Boer, Laura Daniele, Paul Brandt, and Maya Sappelli. Applying semantic reasoning in image retrieval. Proc. ALLDATA, 2015.","title":"Applying semantic reasoning in image retrieval","authors":"Laura Maaike De Boer, Paul Daniele, Maya Brandt, Sappelli","year":"2015","venue":"Proc. ALLDATA","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[De Raedt et al., 2007] Luc De Raedt, Angelika Kimmig, and Hannu Toivonen. Problog: A probabilistic prolog and its application in link discovery. In Proceedings of the 20th International Joint Conference on Artifical Intel- ligence, IJCAI'07, pages 2468-2473, San Francisco, CA, USA, 2007.","title":"Problog: A probabilistic prolog and its application in link discovery","authors":"De Raedt","year":"2007","venue":"Proceedings of the 20th International Joint Conference on Artifical Intelligence, IJCAI'07","doi":null,"url":null,"internal_url":"/proceedings/2007/396"},{"position":12,"raw":"[Gupta et al., 2009] Abhinav Gupta, Aniruddha Kembhavi, and Larry S Davis. Observing human-object interactions: Using spatial and functional compatibility for recognition. IEEE Transactions on Pattern Analysis and Machine In- telligence, 31(10):1775-1789, 2009.","title":"Observing human-object interactions: Using spatial and functional compatibility for recognition","authors":"Abhinav Gupta, Aniruddha Kembhavi, Larry S Davis","year":"2009","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Havasi et al., 2007] Catherine Havasi, Robert Speer, and Ja- son Alonso. Conceptnet 3: a flexible, multilingual seman- tic network for common sense knowledge. In Recent ad- vances in natural language processing, pages 27-29. Cite- seer, 2007.","title":"Conceptnet 3: a flexible, multilingual semantic network for common sense knowledge","authors":"Catherine Havasi, Robert Speer, Jason Alonso","year":"2007","venue":"Recent advances in natural language processing","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. stat, 1050:9, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":"stat","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hudson and Manning, 2019] Drew A Hudson and Christo- pher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR), 2019.","title":"Gqa: A new dataset for real-world visual reasoning and compositional question answering","authors":"A Drew, Christopher D Hudson, Manning","year":"2019","venue":"Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Johnson et al., 2015] Justin Johnson, Ranjay Krishna, Michael Stark, Jia Li, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3668-3678, June 2015.","title":"Image retrieval using scene graphs","authors":"Justin Johnson, Ranjay Krishna, Michael Stark, Jia Li, Michael Bernstein, Li Fei-Fei","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Johnson et al., 2017] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for com- positional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2901-2910, 2017.","title":"Clevr: A diagnostic dataset for compositional language and elementary visual reasoning","authors":"Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, Ross Girshick","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Krishna et al., 2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowd- sourced dense image annotations. Int. J. Comput. Vision, 123(1):32-73, May 2017.","title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","authors":"Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael S Bernstein, Li Fei-Fei","year":"2017","venue":"Int. J. Comput. Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Le et al., 2013] Dieu-Thu Le, Jasper Uijlings, and Raffaella Bernardi. Exploiting language models for visual recogni- tion. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 769-779, 2013.","title":"Dieu-Thu Le, Jasper Uijlings, and Raffaella Bernardi. Exploiting language models for visual recognition","authors":"Le","year":"2013","venue":"Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lee et al., 2017] Joohyung Lee, Samidh Talsania, and Yi Wang. Computing lp mln using asp and mln solvers. Theory and Practice of Logic Programming, 17(5-6):942- 960, 2017.","title":"Computing lp mln using asp and mln solvers","authors":"Lee","year":"2017","venue":"Theory and Practice of Logic Programming","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lenat, 1995] Douglas B. Lenat. Cyc: A large-scale in- vestment in knowledge infrastructure. Commun. ACM, 38(11):33-38, November 1995.","title":"Cyc: A large-scale investment in knowledge infrastructure","authors":"B Douglas, Lenat","year":"1995","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[London et al., 2013] Ben London, Sameh Khamis, Stephen Bach, Bert Huang, Lise Getoor, and Larry Davis. Col- lective activity detection using hinge-loss markov random fields. In Proceedings of the IEEE CVPR Workshops, pages 566-571, 2013.","title":"Collective activity detection using hinge-loss markov random fields","authors":"London","year":"2013","venue":"Proceedings of the IEEE CVPR Workshops","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Manhaeve et al., 2018] Robin Manhaeve, Sebastijan Du- mancic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. Deepproblog: Neural probabilistic logic pro- gramming. In Advances in Neural Information Processing Systems, pages 3753-3763, 2018.","title":"Deepproblog: Neural probabilistic logic programming","authors":"Manhaeve","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Marino et al., 2017] Kenneth Marino, Ruslan Salakhutdi- nov, and Abhinav Gupta. The more you know: Using knowledge graphs for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2673-2681, 2017.","title":"The more you know: Using knowledge graphs for image classification","authors":"Kenneth Marino, Ruslan Salakhutdinov, Abhinav Gupta","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Miller, 1995] George A. Miller. Wordnet: A lexical database for english. Commun. ACM, 38(11):39-41, November 1995.","title":"Wordnet: A lexical database for english","authors":"George A Miller","year":"1995","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Randell et al., 1992] David A. Randell, Zhan Cui, and An- thony G. Cohn. A spatial logic based on regions and con- nection. In Proceedings 3rd International Conference ON Knowledge Representation And Reasoning, 1992.","title":"A spatial logic based on regions and connection","authors":"David A Randell, Zhan Cui, Anthony G Cohn","year":"1992","venue":"Proceedings 3rd International Conference ON Knowledge Representation And Reasoning","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Richardson and Domingos, 2006] Matthew Richardson and Pedro Domingos. Markov logic networks. Machine learn- ing, 62(1-2):107-136, 2006.","title":"Matthew Richardson and Pedro Domingos. Markov logic networks","authors":"Domingos Richardson","year":"2006","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Rocktäschel and Riedel, 2017] Tim Rocktäschel and Sebas- tian Riedel. End-to-end differentiable proving. In Ad- vances in Neural Information Processing Systems, pages 3788-3800, 2017.","title":"Tim Rocktäschel and Sebastian Riedel. End-to-end differentiable proving","authors":"Riedel Rocktäschel","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sanket Shah and Talukdar, 2019] Naganand Yadati San- ket Shah, Anand Mishra and Partha Pratim Talukdar. Kvqa: Knowledge-aware visual question answering. In AAAI, 2019.","title":"Kvqa: Knowledge-aware visual question answering","authors":"Sanket Shah, Talukdar","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Santoro et al., 2017] Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In NIPS, pages 4967-4976, 2017.","title":"A simple neural network module for relational reasoning","authors":"Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, Timothy Lillicrap","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Serafini and Garcez, 2016] Luciano Serafini and Ar- tur d'Avila Garcez. Logic tensor networks: Deep learning and logical reasoning from data and knowledge. arXiv preprint arXiv:1606.04422, 2016.","title":"Luciano Serafini and Artur d","authors":"Garcez ; 'avila Serafini, Garcez","year":"2016","venue":"Logic tensor networks: Deep learning and logical reasoning from data and knowledge","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Suchan and Bhatt, 2016] Jakob Suchan and Mehul Bhatt. The geometry of a scene: On deep semantics for visual perception driven cognitive film, studies. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1-9. IEEE, 2016.","title":"The geometry of a scene: On deep semantics for visual perception driven cognitive film, studies","authors":"Bhatt Suchan, Jakob Suchan, Mehul Bhatt","year":"2016","venue":"IEEE Winter Conference on Applications of Computer Vision (WACV)","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Suchanek et al., 2007] Fabian M. Suchanek, Gjergji Kas- neci, and Gerhard Weikum. Yago: A core of semantic knowledge. In Proceedings of the 16th International Con- ference on World Wide Web, WWW '07, pages 697-706, New York, NY, USA, 2007. ACM.","title":"Yago: A core of semantic knowledge","authors":"Fabian M Suchanek, Gjergji Kasneci, Gerhard Weikum","year":"2007","venue":"Proceedings of the 16th International Conference on World Wide Web, WWW '07","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Summers-Stay et al., 2012] Douglas Summers-Stay, Ching L Teo, Yezhou Yang, Cornelia Fermüller, and Yiannis Aloimonos. Using a minimal action grammar for activity understanding in the real world. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 4104-4111. IEEE, 2012.","title":"Using a minimal action grammar for activity understanding in the real world","authors":"Douglas Summers-Stay, Ching L Teo, Yezhou Yang, Cornelia Fermüller, Yiannis Aloimonos","year":"2012","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Wang et al., 2017] Peng Wang, Qi Wu, Chunhua Shen, An- thony Dick, and Anton van den Hengel. Fvqa: fact-based visual question answering. IEEE TPAMI, 2017.","title":"Fvqa: fact-based visual question answering","authors":"Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, Anton Van Den, Hengel","year":"2017","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wu et al., 2016] Qi Wu, Peng Wang, Chunhua Shen, An- thony Dick, and Anton van den Hengel. Ask me anything: Free-form visual question answering based on knowledge from external sources. In IEEE Conference on Computer Vision and Pattern Recognition CVPR, pages 4622-4630, 2016.","title":"Ask me anything: Free-form visual question answering based on knowledge from external sources","authors":"Qi Wu, Peng Wang, Chunhua Shen, Anthony Dick, Anton Van Den, Hengel","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition CVPR","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Wu et al., 2017] Qi Wu, Damien Teney, Peng Wang, Chun- hua Shen, Anthony Dick, and Anton van den Hengel. Visual question answering: A survey of methods and datasets. Computer Vision and Image Understanding, 163:21-40, 2017.","title":"Visual question answering: A survey of methods and datasets","authors":"Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, Anton Van Den, Hengel","year":"2017","venue":"Computer Vision and Image Understanding","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Wulfmeier et al., 2016] Markus Wulfmeier, Dushyant Rao, and Ingmar Posner. Incorporating human domain knowl- edge into large scale cost function learning. arXiv preprint arXiv:1612.04318, 2016.","title":"Incorporating human domain knowledge into large scale cost function learning","authors":"Markus Wulfmeier, Dushyant Rao, Ingmar Posner","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Yu et al., 2017] Ruichi Yu, Ang Li, Vlad I. Morariu, and Larry S. Davis. Visual Relationship Detection with Inter- nal and External Linguistic Knowledge Distillation. ICCV, 2017.","title":"Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation","authors":"Ruichi Yu, Ang Li, Vlad I Morariu, Larry S Davis","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zheng et al., 2015] Bo Zheng, Yibiao Zhao, Joey Yu, Kat- sushi Ikeuchi, and Song-Chun Zhu. Scene understanding by reasoning stability and safety. International Journal of Computer Vision, 112(2):221-238, 2015.","title":"Scene understanding by reasoning stability and safety","authors":"Bo Zheng, Yibiao Zhao, Joey Yu, Katsushi Ikeuchi, Song-Chun Zhu","year":"2015","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Zhou et al., 2018] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In ECCV, September 2018.","title":"Temporal relational reasoning in videos","authors":"Zhou","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Zhu et al., 2014] Yuke Zhu, Alireza Fathi, and Li Fei-Fei. Reasoning about object affordances in a knowledge base representation. In ECCV (2), pages 408-424. Springer, 2014.","title":"Reasoning about object affordances in a knowledge base representation","authors":"Yuke Zhu, Alireza Fathi, Li Fei-Fei","year":"2014","venue":"ECCV (2)","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}