{"type":"ScholarlyArticle","title":"Logic Tensor Networks for Semantic Image Interpretation","authors":[{"name":"Ivan Donadello","slug":"ivan-donadello","given":"Ivan","family":"Donadello","affiliation":"Fondazione Bruno Kessler and University of Trento Trento , Italy"},{"name":"Luciano Serafini","slug":"luciano-serafini","given":"Luciano","family":"Serafini","affiliation":"Fondazione Bruno Kessler Via Sommarive 18 , I-38123 Trento , Italy"},{"name":"Artur d'Avila Garcez","slug":"artur-d-avila-garcez","given":"Artur d'Avila","family":"Garcez","affiliation":"City , University of London Northampton Square London EC1V 0HB , UK"}],"abstract":"Semantic Image Interpretation (SII) is the task of extracting structured semantic descriptions from images. It is widely agreed that the combined use of visual data and background knowledge is of great importance for SII. Recently, Statistical Relational Learning (SRL) approaches have been developed for reasoning under uncertainty and learning in the presence of data and rich knowledge. Logic Tensor Networks (LTNs) are a SRL framework which integrates neural networks with first-order fuzzy logic to allow (i) efficient learning from noisy data in the presence of logical constraints, and (ii) reasoning with logical formulas describing general properties of the data. In this paper, we develop and apply LTNs to two of the main tasks of SII, namely, the classification of an image's bounding boxes and the detection of the relevant part-of relations between objects. To the best of our knowledge, this is the first successful application of SRL to such SII tasks. The proposed approach is evaluated on a standard image processing benchmark. Experiments show that background knowledge in the form of logical constraints can improve the performance of purely data-driven approaches, including the state-of-the-art Fast Region-based Convolutional Neural Networks (Fast R-CNN). Moreover, we show that the use of logical background knowledge adds robustness to the learning system when errors are present in the labels of the training data.","doi":"10.24963/ijcai.2017/221","url":"https://ijcai.org/proceedings/2017/221","pdf":"https://ijcai.org/proceedings/2017/0221.pdf","markdown":"https://ijcai.org/proceedings/2017/221.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/221","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":221,"pages":{"first":1596,"last":1602},"keywords":["Robotics and Vision: Vision and Perception","Machine Learning: Knowledge-based Learning","Uncertainty in AI: Uncertainty in AI"],"videos":[],"references":[{"position":1,"raw":"[Atif et al., 2014] J. Atif, C. Hudelot, and I Bloch. Explana- tory reasoning for image understanding using formal con- cept analysis and description logics. Systems, Man, and Cybernetics: Systems, IEEE Transactions on, 44(5):552- 570, May 2014.","title":"Explanatory reasoning for image understanding using formal concept analysis and description logics","authors":"J Atif, C Hudelot, Bloch","year":"2014","venue":"Systems, Man, and Cybernetics: Systems, IEEE Transactions on","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bach et al., 2015] Stephen H. Bach, Matthias Broecheler, Bert Huang, and Lise Getoor. Hinge-loss markov random fields and probabilistic soft logic. CoRR, abs/1505.04406, 2015.","title":"Hinge-loss markov random fields and probabilistic soft logic","authors":"H Stephen, Matthias Bach, Bert Broecheler, Lise Huang, Getoor","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bergmann, 2008] M. Bergmann. An Introduction to Many- Valued and Fuzzy Logic: Semantics, Algebras, and Derivation Systems. Cambridge University Press, 2008.","title":"An Introduction to Many-Valued and Fuzzy Logic: Semantics, Algebras, and Derivation Systems","authors":"M Bergmann","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2012] Na Chen, Qian-Yi Zhou, and Viktor Prasanna. Understanding web images by object relation network. In Proceedings of the 21st International Con- ference on World Wide Web, WWW '12, pages 291-300, New York, NY, USA, 2012. ACM.","title":"Understanding web images by object relation network","authors":"Na Chen, Qian-Yi Zhou, Viktor Prasanna","year":"2012","venue":"Proceedings of the 21st International Conference on World Wide Web, WWW '12","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2014] X. Chen, R. Mottaghi, X. Liu, S. Fidler, R. Urtasun, and A. Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In CVPR, 2014.","title":"Detect what you can: Detecting and representing objects using holistic models and body parts","authors":"Chen","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dasiopoulou et al., 2009] S. Dasiopoulou, Y. Kompatsiaris, and M.l G. Strintzis. Applying fuzzy dls in the extraction of image semantics. J. Data Semantics, 14:105-132, 2009.","title":"Applying fuzzy dls in the extraction of image semantics","authors":"S Dasiopoulou, Y Kompatsiaris, M, G Strintzis","year":"2009","venue":"J. Data Semantics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Diligenti et al., 2015] Michelangelo Diligenti, Marco Gori, and Claudio Saccà. Semantic-based regularization for learning and inference. Artificial Intelligence, 2015.","title":"Michelangelo Diligenti, Marco Gori, and Claudio Saccà. Semantic-based regularization for learning and inference","authors":"Diligenti","year":"2015","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Donadello and Serafini, 2016] I. Donadello and L. Serafini. Integration of numeric and symbolic information for se- mantic image interpretation. Intelligenza Artificiale, 10(1):33-47, 2016.","title":"Integration of numeric and symbolic information for semantic image interpretation","authors":"Serafini Donadello, I Donadello, L Serafini","year":"2016","venue":"Intelligenza Artificiale","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fellbaum, 1998] Christiane Fellbaum, editor. WordNet: an electronic lexical database. MIT Press, 1998.","title":"WordNet: an electronic lexical database","authors":null,"year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Forestier et al., 2013] G. Forestier, C. Wemmert, and A. Puissant. Coastal image interpretation using back- ground knowledge and semantics. Computers & Geo- sciences, 54:88-96, 2013.","title":"Coastal image interpretation using background knowledge and semantics","authors":"G Forestier, C Wemmert, A Puissant","year":"2013","venue":"Computers & Geosciences","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Girshick, 2015] Ross Girshick. Fast r-cnn. In International Conference on Computer Vision (ICCV), 2015.","title":"Fast r-cnn","authors":"Ross Girshick","year":"2015","venue":"International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Guarino and Guizzardi, 2016] N. Guarino and G. Guiz- zardi. On the reification of relationships. In 24th Italian Symp. on Advanced Database Sys., pages 350-357, 2016.","title":"On the reification of relationships","authors":"N Guarino, G Guizzardi","year":"2016","venue":"24th Italian Symp. on Advanced Database Sys","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gutmann et al., 2010] B. Gutmann, M. Jaeger, and L. De Raedt. Extending problog with continuous distributions. In Proc. ILP, pages 76-91. Springer, 2010.","title":"Extending problog with continuous distributions","authors":"B Gutmann, M Jaeger, L De Raedt","year":"2010","venue":"Proc. ILP","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hudelot et al., 2008] C. Hudelot, J. Atif, and I. Bloch. Fuzzy spatial relation ontology for image interpretation. Fuzzy Sets and Systems, 159(15):1929-1951, 2008.","title":"Fuzzy spatial relation ontology for image interpretation","authors":"C Hudelot, J Atif, I Bloch","year":"2008","venue":"Fuzzy Sets and Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Krishna et al., 2016] R. Krishna, Y. Zhu, O. Groth, J. John- son, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.- L. Li, D. Shamma, M. Bernstein, and Li F.-F. Visual genome: Connecting language and vision using crowd- sourced dense image annotations, 2016.","title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","authors":"R Krishna, Y Zhu, O Groth, J Johnson, K Hata, J Kravitz, S Chen, Y Kalantidis, L.-L Li, D Shamma, M Bernstein, Li F.-F","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kulkarni et al., 2011] G.h Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg. Baby talk: Un- derstanding and generating image descriptions. In CVPR, 2011.","title":"Baby talk: Understanding and generating image descriptions","authors":"G Kulkarni, V Premraj, S Dhar, S Li, Y Choi, A C Berg, T L Berg","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lu et al., 2016] C. Lu, R. Krishna, M. Bernstein, and L Fei- Fei. Visual relationship detection with language priors. In ECCV, pages 852-869, 2016.","title":"Visual relationship detection with language priors","authors":"C Lu, R Krishna, M Bernstein, L Fei-Fei","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Marszalek and Schmid, 2007] Marcin Marszalek and Cordelia Schmid. Semantic Hierarchies for Visual Object Recognition. In CVPR, 2007.","title":"Semantic Hierarchies for Visual Object Recognition","authors":"Marcin Marszalek, Cordelia Schmid","year":"2007","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Neumann and Möller, 2008] Bernd Neumann and Ralf Möller. On scene interpretation with description logics. Image and Vision Computing, 26(1):82 -101, 2008. Cognitive Vision-Special Issue.","title":"On scene interpretation with description logics","authors":"Möller Neumann, Bernd Neumann, Ralf Möller","year":"2008","venue":"Image and Vision Computing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Nyga et al., 2014] D. Nyga, F. Balint-Benczedi, and M. Beetz. Pr2 looking at things-ensemble learning for unstructured information processing with markov logic networks. In IEEE Intl. Conf.on Robotics and Automation, pages 3916-3923, 2014.","title":"Pr2 looking at things-ensemble learning for unstructured information processing with markov logic networks","authors":"D Nyga, F Balint-Benczedi, M Beetz","year":"2014","venue":"IEEE Intl. Conf.on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Peraldi et al., 2009] I. S. Espinosa Peraldi, A. Kaya, and R. Möller. Formalizing multimedia interpretation based on abduction over description logic aboxes. In Proc. of the 22nd Intl. Workshop on Description Logics, volume 477 of CEUR Workshop Proceedings. CEUR-WS.org, 2009.","title":"Formalizing multimedia interpretation based on abduction over description logic aboxes","authors":"I S Espinosa Peraldi, A Kaya, R Möller","year":"2009","venue":"Proc. of the 22nd Intl. Workshop on Description Logics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ramanathan et al., 2015] V. Ramanathan, C. Li, J. Deng, W. Han, Z. Li, K. Gu, Y. Song, S. Bengio, C. Rosenberg, and L. Fei-Fei. Learning semantic relationships for better action retrieval in images. In CVPR, 2015.","title":"Learning semantic relationships for better action retrieval in images","authors":"V Ramanathan, C Li, J Deng, W Han, Z Li, K Gu, Y Song, S Bengio, C Rosenberg, L Fei-Fei","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ravkic et al., 2015] Irma Ravkic, Jan Ramon, and Jesse Davis. Learning relational dependency networks in hybrid domains. Machine Learning, 100(2-3):217-254, 2015.","title":"Learning relational dependency networks in hybrid domains","authors":"Ravkic","year":"2015","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Reed et al., 2014] S. Reed, H. Lee, D. Anguelov, C. Szegedy, D. Erhan, and A. Rabinovich. Training deep neural networks on noisy labels with bootstrapping. CoRR, abs/1412.6596, 2014.","title":"Training deep neural networks on noisy labels with bootstrapping","authors":"S Reed, H Lee, D Anguelov, C Szegedy, D Erhan, A Rabinovich","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Richardson and Domingos, 2006] Matthew Richardson and Pedro Domingos. Markov logic networks. Machine Learn- ing, 62(1-2):107-136, 2006.","title":"Matthew Richardson and Pedro Domingos. Markov logic networks","authors":"Domingos Richardson","year":"2006","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Rocktaschel et al., 2015] T. Rocktaschel, S. Singh, and S. Riedel. Injecting logical background knowledge into embeddings for relation extraction. In NAACL, 2015.","title":"Injecting logical background knowledge into embeddings for relation extraction","authors":"T Rocktaschel, S Singh, S Riedel","year":"2015","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Serafini and d'Avila Garcez, 2016] L. Serafini and A. S. d'Avila Garcez. Learning and reasoning with logic tensor networks. In Proc. AI*IA, pages 334-348, 2016.","title":"Learning and reasoning with logic tensor networks","authors":"L Serafini, A S Garcez","year":"2016","venue":"Proc. AI*IA","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Socher et al., 2013] R. Socher, D. Chen, C. D. Manning, and A. Ng. Reasoning with neural tensor networks for knowledge base completion. In Advances in Neural Infor- mation Processing Systems, pages 926-934, 2013.","title":"Reasoning with neural tensor networks for knowledge base completion","authors":"R Socher, D Chen, C D Manning, A Ng","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang and Domingos, 2008] Jue Wang and Pedro M Domingos. Hybrid markov logic networks. In AAAI, volume 8, pages 1106-1111, 2008.","title":"Hybrid markov logic networks","authors":"Jue Wang, Pedro M Domingos","year":"2008","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhu et al., 2014] Y. Zhu, A. Fathi, and L. Fei-Fei. Reason- ing about object affordances in a knowledge base repre- sentation. In ECCV, pages 408-424. 2014.","title":"Reasoning about object affordances in a knowledge base representation","authors":"Y Zhu, A Fathi, L Fei-Fei","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}