{"type":"ScholarlyArticle","title":"How a General-Purpose Commonsense Ontology can Improve Performance of Learning-Based Image Retrieval","authors":[{"name":"Rodrigo Toro Icarte","slug":"rodrigo-toro-icarte","given":"Rodrigo Toro","family":"Icarte","affiliation":"† University of Toronto"},{"name":"Jorge A. Baier","slug":"jorge-a-baier","given":"Jorge A.","family":"Baier","affiliation":"‡ Chilean Center for Semantic Web Research , Pontificia Universidad Católica de Chile ,"},{"name":"Cristian Ruz","slug":"cristian-ruz","given":"Cristian","family":"Ruz","affiliation":"‡ Chilean Center for Semantic Web Research , Pontificia Universidad Católica de Chile ,"},{"name":"Alvaro Soto","slug":"alvaro-soto","given":"Alvaro","family":"Soto","affiliation":"‡ Chilean Center for Semantic Web Research , Pontificia Universidad Católica de Chile ,"}],"abstract":"The knowledge representation community has built general-purpose ontologies which contain large amounts of commonsense knowledge over relevant aspects of the world, including useful visual information, e.g.: \"a ball is used by a football player\", \"a tennis player is located at a tennis court\". Current state-of-the-art approaches for visual recognition do not exploit these rule-based knowledge sources. Instead, they learn recognition models directly from training examples. In this paper, we study how general-purpose ontologies—specifically, MIT's ConceptNet ontology—can improve the performance of state-of-the-art vision systems. As a testbed, we tackle the problem of sentence-based image retrieval. Our retrieval approach incorporates knowledge from ConceptNet on top of a large pool of object detectors derived from a deep learning technique. In our experiments, we show that ConceptNet can improve performance on a common benchmark dataset. Key to our performance is the use of the ESPGAME dataset to select visually relevant relations from ConceptNet. Consequently, a main conclusion of this work is that general-purpose commonsense ontologies improve performance on visual reasoning tasks when properly filtered to select meaningful visual relations.","doi":"10.24963/ijcai.2017/178","url":"https://ijcai.org/proceedings/2017/178","pdf":"https://ijcai.org/proceedings/2017/0178.pdf","markdown":"https://ijcai.org/proceedings/2017/178.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/178","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Knowledge Representation, Reasoning, and Logic","track":"Main track","number":178,"pages":{"first":1283,"last":1289},"keywords":["Knowledge Representation, Reasoning, and Logic: Common-Sense Reasoning","Machine Learning: Knowledge-based Learning","Robotics and Vision: Vision and Perception"],"videos":[],"references":[{"position":1,"raw":"[Biederman, 1972] Irving Biederman. Perceiving real-world scenes. Science, 177(4043):77-80, 1972.","title":"Perceiving real-world scenes","authors":"Irving Biederman","year":"1972","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2013] Xinlei Chen, Abhinav Shrivastava, and Abhinav Gupta. Neil: Extracting visual knowledge from web data. In ICCV, pages 1409-1416, 2013.","title":"Neil: Extracting visual knowledge from web data","authors":"Chen","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Divvala et al., 2014] Santosh K Divvala, Ali Farhadi, and Carlos Guestrin. Learning everything about anything: Webly-supervised visual concept learning. In CVPR, pages 3270-3277, 2014.","title":"Learning everything about anything: Webly-supervised visual concept learning","authors":"Ali Santosh K Divvala, Carlos Farhadi, Guestrin","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Espinace et al., 2013] Pablo Espinace, Thomas Kollar, Nicholas Roy, and Alvaro Soto. Indoor scene recogni- tion by a mobile robot through adaptive object detection. Robotics and Autonomous Systems, 61(9):932-947, 2013.","title":"Indoor scene recognition by a mobile robot through adaptive object detection","authors":"Espinace","year":"2013","venue":"Robotics and Autonomous Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fang et al., 2015] Hao Fang, Saurabh Gupta, Forrest Ian- dola, Rupesh K Srivastava, Li Deng, Piotr Dollár, Jian- feng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. From captions to visual concepts and back. In CVPR, pages 1473-1482, 2015.","title":"From captions to visual concepts and back","authors":"Saurabh Hao Fang, Forrest Gupta, Iandola, K Rupesh, Li Srivastava, Piotr Deng, Jianfeng Dollár, Xiaodong Gao, Margaret He, John C Mitchell, Platt","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Havasi et al., 2007] Catherine Havasi, Robert Speer, and Ja- son Alonso. Conceptnet 3: a flexible, multilingual se- mantic network for common sense knowledge. In RANLP, pages 27-29, 2007.","title":"Conceptnet 3: a flexible, multilingual semantic network for common sense knowledge","authors":"Catherine Havasi, Robert Speer, Jason Alonso","year":"2007","venue":"RANLP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Karpathy and Fei-Fei, 2015] Andrej Karpathy and Li Fei- Fei. Deep visual-semantic alignments for generating im- age descriptions. In CVPR, pages 3128-3137, 2015.","title":"Deep visual-semantic alignments for generating image descriptions","authors":"Fei-Fei ; Karpathy, Andrej Karpathy, Li Fei-Fei","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Klein et al., 2015] Benjamin Klein, Guy Lev, Gil Sadeh, and Lior Wolf. Associating neural word embeddings with deep image representations using fisher vectors. In CVPR, pages 4437-4446, 2015.","title":"Associating neural word embeddings with deep image representations using fisher vectors","authors":"Benjamin Klein, Guy Lev, Gil Sadeh, Lior Wolf","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097- 1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Le et al., 2013] Dieu-Thu Le, Jasper RR Uijlings, and Raf- faella Bernardi. Exploiting language models for visual recognition. In EMNLP, pages 769-779, 2013.","title":"Dieu-Thu Le, Jasper RR Uijlings, and Raffaella Bernardi. Exploiting language models for visual recognition","authors":"Le","year":"2013","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lenat, 1995] Douglas B Lenat. Cyc: A large-scale invest- ment in knowledge infrastructure. Communications of the ACM, 38(11):33-38, 1995.","title":"Cyc: A large-scale investment in knowledge infrastructure","authors":"Douglas B Lenat","year":"1995","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In ECCV, pages 740-755, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2011] Jingen Liu, Benjamin Kuipers, and Silvio Savarese. Recognizing human actions by attributes. In CVPR, pages 3337-3344, 2011.","title":"Recognizing human actions by attributes","authors":"Liu","year":"2011","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lobel et al., 2013] Hans Lobel, René Vidal, and Alvaro Soto. Hierarchical joint max-margin learning of mid and top level representations for visual recognition. In ICCV, pages 1697-1704, 2013.","title":"Hierarchical joint max-margin learning of mid and top level representations for visual recognition","authors":"Lobel","year":"2013","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Maillot and Thonnat, 2008] Nicolas Eric Maillot and Monique Thonnat. Ontology based complex object recog- nition. Image and Vision Computing, 26(1):102-113, 2008.","title":"Ontology based complex object recognition","authors":"Thonnat ; Maillot, Eric Nicolas, Monique Maillot, Thonnat","year":"2008","venue":"Image and Vision Computing","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Marques et al., 2011] Oge Marques, Elan Barenholtz, and Vincent Charvillat. Context modeling in computer vision: techniques, implications, and applications. Multimedia Tools and Applications, 51(1):303-339, 2011.","title":"Context modeling in computer vision: techniques, implications, and applications","authors":"Marques","year":"2011","venue":"Multimedia Tools and Applications","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mitchell, 1997] Tom M Mitchell. Machine learning. 1997. Burr Ridge, IL: McGraw Hill, 45(37):870-877, 1997.","title":"Machine learning","authors":"M Tom, Mitchell","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ohlsson et al., 2013] Stellan Ohlsson, Robert H Sloan, György Turán, and Aaron Urasky. Verbal iq of a four- year old achieved by an ai system. In AAAI, pages 89-91, 2013.","title":"Verbal iq of a fouryear old achieved by an ai system","authors":"Ohlsson","year":"2013","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ordonez et al., 2015] Vicente Ordonez, Wei Liu, Jia Deng, Yejin Choi, Alexander C Berg, and Tamara L Berg. Pre- dicting entry-level categories. International Journal of Computer Vision, 115(1):29-43, 2015.","title":"Predicting entry-level categories","authors":"Vicente Ordonez, Wei Liu, Jia Deng, Yejin Choi, Alexander C Berg, Tamara L Berg","year":"2015","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rabinovich et al., 2007] Andrew Rabinovich, Andrea Vedaldi, Carolina Galleguillos, Eric Wiewiora, and Serge Belongie. Objects in context. In ICCV, pages 1-8, 2007.","title":"Objects in context","authors":"Andrew Rabinovich, Andrea Vedaldi, Carolina Galleguillos, Eric Wiewiora, Serge Belongie","year":"2007","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ranzato et al., 2008] Marc'aurelio Ranzato, Y lan Boureau, and Yann L. Cun. Sparse feature learning for deep belief networks. In NIPS, pages 1185-1192, 2008.","title":"Sparse feature learning for deep belief networks","authors":"Ranzato","year":"2008","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Singh et al., 2002] Push Singh, Thomas Lin, Erik Mueller, Grace Lim, Travell Perkins, and Wan Li Zhu. Open mind common sense: Knowledge acquisition from the general public. On the move to meaningful internet systems 2002: CoopIS, DOA, and ODBASE, pages 1223-1237, 2002.","title":"Open mind common sense: Knowledge acquisition from the general public","authors":"Singh","year":"2002","venue":"On the move to meaningful internet systems 2002: CoopIS, DOA, and ODBASE","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Singh et al., 2012] Saurabh Singh, Abhinav Gupta, and Alexei Efros. Unsupervised discovery of mid-level dis- criminative patches. ECCV 2012, pages 73-86, 2012.","title":"Unsupervised discovery of mid-level discriminative patches","authors":"Saurabh Singh, Abhinav Gupta, Alexei Efros","year":"2012","venue":"ECCV 2012","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Snoek et al., 2007] Cees GM Snoek, Bouke Huurnink, Laura Hollink, Maarten De Rijke, Guus Schreiber, and Marcel Worring. Adding semantics to detectors for video retrieval. IEEE Transactions on multimedia, 9(5):975- 986, 2007.","title":"Adding semantics to detectors for video retrieval","authors":"G M Cees, Bouke Snoek, Laura Huurnink, Maarten Hollink, Guus De Rijke, Marcel Schreiber, Worring","year":"2007","venue":"IEEE Transactions on multimedia","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neu- ral image caption generator. In CVPR, pages 3156-3164, 2015.","title":"Show and tell: A neural image caption generator","authors":"Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Von Ahn and Dabbish, 2004] Luis Von Ahn and Laura Dab- bish. Labeling images with a computer game. In SIGCHI, pages 319-326, 2004.","title":"Luis Von Ahn and Laura Dabbish. Labeling images with a computer game","authors":"Von Ahn, Dabbish","year":"2004","venue":"SIGCHI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Weston et al., 2011] Jason Weston, Samy Bengio, and Nicolas Usunier. Wsabie: Scaling up to large vocabulary image annotation. In IJCAI, pages 2764-2770, 2011.","title":"Wsabie: Scaling up to large vocabulary image annotation","authors":"Jason Weston, Samy Bengio, Nicolas Usunier","year":"2011","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2011/460"},{"position":30,"raw":"[Xie and He, 2013] Lexing Xie and Xuming He. Picture tags and world knowledge: learning tag relations from visual semantic sources. In ACM-MM, pages 967-976, 2013.","title":"Picture tags and world knowledge: learning tag relations from visual semantic sources","authors":"Lexing Xie, Xuming He","year":"2013","venue":"ACM-MM","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhu et al., 2014] Yuke Zhu, Alireza Fathi, and Li Fei-Fei. Reasoning about object affordances in a knowledge base representation. In ECCV, pages 408-424, 2014.","title":"Reasoning about object affordances in a knowledge base representation","authors":"Yuke Zhu, Alireza Fathi, Li Fei-Fei","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}