{"type":"ScholarlyArticle","title":"Object Detection Meets Knowledge Graphs","authors":[{"name":"Yuan Fang","slug":"yuan-fang","given":"Yuan","family":"Fang","affiliation":"* Institute for Infocomm Research , A"},{"name":"Kingsley Kuan","slug":"kingsley-kuan","given":"Kingsley","family":"Kuan","affiliation":"* Institute for Infocomm Research , A"},{"name":"Jie Lin","slug":"jie-lin","given":"Jie","family":"Lin","affiliation":"* Institute for Infocomm Research , A"},{"name":"Cheston Tan","slug":"cheston-tan","given":"Cheston","family":"Tan","affiliation":"* Institute for Infocomm Research , A"},{"name":"Vijay Chandrasekhar","slug":"vijay-chandrasekhar","given":"Vijay","family":"Chandrasekhar","affiliation":"STAR , Singapore"}],"abstract":"Object detection in images is a crucial task in computer vision, with important applications ranging from security surveillance to autonomous vehicles. Existing state-of-the-art algorithms, including deep neural networks, only focus on utilizing features within an image itself, largely neglecting the vast amount of background knowledge about the real world. In this paper, we propose a novel framework of knowledge-aware object detection, which enables the integration of external knowledge such as knowledge graphs into any object detection algorithm. The framework employs the notion of semantic consistency to quantify and generalize knowledge, which improves object detection through a re-optimization process to achieve better consistency with background knowledge. Finally, empirical evaluation on two benchmark datasets show that our approach can significantly increase recall by up to 6.3 points without compromising mean average precision, when compared to the state-of-the-art baseline.","doi":"10.24963/ijcai.2017/230","url":"https://ijcai.org/proceedings/2017/230","pdf":"https://ijcai.org/proceedings/2017/0230.pdf","markdown":"https://ijcai.org/proceedings/2017/230.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/230","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":230,"pages":{"first":1661,"last":1667},"keywords":["Machine Learning: Knowledge-based Learning","Robotics and Vision: Vision and Perception"],"videos":[],"references":[{"position":1,"raw":"[Bell et al., 2016] Sean Bell, C. Lawrence Zitnick, Kavita Bala, and Ross B. Girshick. Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks. In CVPR, pages 2874-2883, 2016.","title":"Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks","authors":"Sean Bell, C Lawrence Zitnick, Kavita Bala, Ross B Girshick","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Deng et al., 2014] Jia Deng, Nan Ding, Yangqing Jia, An- drea Frome, Kevin Murphy, Samy Bengio, Yuan Li, Hart- mut Neven, and Hartwig Adam. Large-scale object classi- fication using label relation graphs. In ECCV, Part I, pages 48-64, 2014.","title":"Large-scale object classification using label relation graphs","authors":"Jia Deng, Nan Ding, Yangqing Jia, Andrea Frome, Kevin Murphy, Samy Bengio, Yuan Li, Hartmut Neven, Hartwig Adam","year":"2014","venue":"ECCV, Part I","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dong et al., 2014] Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. Knowledge vault: a Web-scale approach to probabilistic knowledge fusion. In KDD, pages 601-610, 2014.","title":"Knowledge vault: a Web-scale approach to probabilistic knowledge fusion","authors":"Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, Wei Zhang","year":"2014","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Everingham et al., 2010] Mark Everingham, Luc J. Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The PASCAL visual object classes (VOC) challenge. IJCV, 88(2):303-338, 2010.","title":"The PASCAL visual object classes (VOC) challenge","authors":"Mark Everingham, J Luc, Van Gool, K I Christopher, John M Williams, Andrew Winn, Zisserman","year":"2010","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fang and Chang, 2011] Yuan Fang and Kevin Chen-Chuan Chang. Searching patterns for relation extraction over the web: rediscovering the pattern-relation duality. In WSDM, pages 825-834, 2011.","title":"Searching patterns for relation extraction over the web: rediscovering the pattern-relation duality","authors":"Yuan Fang, Kevin Chen, -Chuan Chang","year":"2011","venue":"WSDM","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fang et al., 2013] Yuan Fang, Kevin Chen-Chuan Chang, and Hady Wirawan Lauw. RoundTripRank: Graph-based proximity with importance and specificity. In ICDE, pages 613-624, 2013.","title":"RoundTripRank: Graph-based proximity with importance and specificity","authors":"Yuan Fang, Kevin Chen-Chuan, Chang, Hady Wirawan Lauw","year":"2013","venue":"ICDE","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Girshick et al., 2014] Ross B. Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierar- chies for accurate object detection and semantic segmen- tation. In CVPR, pages 580-587, 2014.","title":"Rich feature hierarchies for accurate object detection and semantic segmentation","authors":"Ross B Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Girshick, 2015] Ross B. Girshick. Fast R-CNN. In ICCV, pages 1440-1448, 2015.","title":"Fast R-CNN","authors":"Ross B Girshick","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hong et al., 2017] Jongkwang Hong, Yongwon Hong, Youngjung Uh, and Hyeran Byun. Discovering over- looked objects: Context-based boosting of object detection in indoor scenes. Pattern Recogn. Lett., 86:56-61, 2017.","title":"Discovering overlooked objects: Context-based boosting of object detection in indoor scenes","authors":"Hong","year":"2017","venue":"Pattern Recogn. Lett","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krishna et al., 2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowd- sourced dense image annotations. IJCV, 123:32-73, 2017.","title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","authors":"Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael S Bernstein, Li Fei-Fei","year":"2017","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS, pages 1106- 1114, 2012.","title":"ImageNet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lenat, 1995] Douglas B. Lenat. CYC: A large-scale in- vestment in knowledge infrastructure. Commun. ACM, 38(11):32-38, 1995.","title":"CYC: A large-scale investment in knowledge infrastructure","authors":"B Douglas, Lenat","year":"1995","venue":"Commun. ACM","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge J. Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: com- mon objects in context. In ECCV, Part V, pages 740-755, 2014.","title":"Microsoft COCO: common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge J Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence Zitnick","year":"2014","venue":"ECCV, Part V","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu and Singh, 2004] Hugo Liu and Push Singh. ConceptNet-a practical commonsense reasoning tool-kit. BT Technology Journal, 22(4):211-226, 2004.","title":"Hugo Liu and Push Singh. ConceptNet-a practical commonsense reasoning tool-kit","authors":"Singh Liu","year":"2004","venue":"BT Technology Journal","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lu et al., 2016] Cewu Lu, Ranjay Krishna, Michael S. Bernstein, and Fei-Fei Li. Visual relationship detection with language priors. In ECCV, Part I, pages 852-869, 2016.","title":"Visual relationship detection with language priors","authors":"Lu","year":"2016","venue":"ECCV, Part I","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Paulheim, 2017] Heiko Paulheim. Knowledge graph refine- ment: A survey of approaches and evaluation methods. Se- mantic Web, 8(3):489-508, 2017.","title":"Knowledge graph refinement: A survey of approaches and evaluation methods","authors":"Heiko Paulheim","year":"2017","venue":"Semantic Web","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rabinovich et al., 2007] Andrew Rabinovich, Andrea Vedaldi, Carolina Galleguillos, Eric Wiewiora, and Serge J. Belongie. Objects in context. In ICCV, pages 1-8, 2007.","title":"Objects in context","authors":"Andrew Rabinovich, Andrea Vedaldi, Carolina Galleguillos, Eric Wiewiora, Serge J Belongie","year":"2007","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross B. Gir- shick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In NIPS, pages 91-99, 2015.","title":"Faster R-CNN: towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sermanet et al., 2013] Pierre Sermanet, David Eigen, Xi- ang Zhang, Michaël Mathieu, Rob Fergus, and Yann LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. CoRR, arXiv:1312.6229, 2013.","title":"Overfeat: Integrated recognition, localization and detection using convolutional networks","authors":"Pierre Sermanet, David Eigen, Xiang Zhang, Michaël Mathieu, Rob Fergus, Yann Lecun","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Suchanek et al., 2007] Fabian M. Suchanek, Gjergji Kas- neci, and Gerhard Weikum. YAGO: a core of semantic knowledge. In WWW, pages 697-706, 2007.","title":"YAGO: a core of semantic knowledge","authors":"Fabian M Suchanek, Gjergji Kasneci, Gerhard Weikum","year":"2007","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabi- novich. Going deeper with convolutions. In CVPR, pages 1-9, 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tong et al., 2006] Hanghang Tong, Christos Faloutsos, and Jia-Yu Pan. Fast random walk with restart and its applica- tions. In ICDM, pages 613-622, 2006.","title":"Fast random walk with restart and its applications","authors":"Tong","year":"2006","venue":"ICDM","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Vondrick et al., 2016] Carl Vondrick, Deniz Oktay, Hamed Pirsiavash, and Antonio Torralba. Predicting motivations of actions by leveraging text. In CVPR, pages 2997-3005, 2016.","title":"Predicting motivations of actions by leveraging text","authors":"Carl Vondrick, Deniz Oktay, Hamed Pirsiavash, Antonio Torralba","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wu et al., 2016] Qi Wu, Peng Wang, Chunhua Shen, An- thony R. Dick, and Anton van den Hengel. Ask me anything: Free-form visual question answering based on knowledge from external sources. In CVPR, pages 4622- 4630, 2016.","title":"Ask me anything: Free-form visual question answering based on knowledge from external sources","authors":"Qi Wu, Peng Wang, Chunhua Shen, Anthony R Dick, Anton Van Den, Hengel","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhu et al., 2013] Fanwei Zhu, Yuan Fang, Kevin Chen- Chuan Chang, and Jing Ying. Incremental and accuracy- aware personalized pagerank through scheduled approxi- mation. PVLDB, 6(6):481-492, 2013.","title":"Incremental and accuracyaware personalized pagerank through scheduled approximation","authors":"Fanwei Zhu, Yuan Fang, Kevin Chen-Chuan, Chang, Jing Ying","year":"2013","venue":"PVLDB","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}