{"type":"ScholarlyArticle","title":"Zero-Shot Object Detection via Learning an Embedding from Semantic Space to Visual Space","authors":[{"name":"Licheng Zhang","slug":"licheng-zhang","given":"Licheng","family":"Zhang","affiliation":"Depatment of Computer Science and Technology , Southern University of Science and Technology; Research Institute of Trustworthy Autonomous Systems , Shenzhen , 518055 , China"},{"name":"Xianzhi Wang","slug":"xianzhi-wang","given":"Xianzhi","family":"Wang","affiliation":"University of Technology Sydney"},{"name":"Lina Yao","slug":"lina-yao","given":"Lina","family":"Yao","affiliation":"University of New South Wales"},{"name":"Lin Wu","slug":"lin-wu","given":"Lin","family":"Wu","affiliation":"Hefei University of Technology"},{"name":"Feng Zheng","slug":"feng-zheng","given":"Feng","family":"Zheng","affiliation":"Depatment of Computer Science and Technology , Southern University of Science and Technology; Research Institute of Trustworthy Autonomous Systems , Shenzhen , 518055 , China"}],"abstract":"Zero-shot object detection (ZSD) has received considerable attention from the community of computer vision in recent years. It aims to simultaneously locate and categorize previously unseen objects during inference. One crucial problem of ZSD is how to accurately predict the label of each object proposal, i.e. categorizing object proposals, when conducting ZSD for unseen categories.\r\nPrevious ZSD models generally relied on learning an embedding from visual space to semantic space or learning a joint embedding between semantic description and visual representation. As the features in the learned semantic space or the joint projected space tend to suffer from the hubness problem, namely the feature vectors are likely embedded to an area of incorrect labels, and thus it will lead to lower detection precision. In this paper, instead, we propose to learn a deep embedding from the semantic space to the visual space, which enables to well alleviate the hubness problem, because, compared with semantic space or joint embedding space, the distribution in visual space has smaller variance. After learning a deep embedding model, we perform $k$ nearest neighbor search in the visual space of unseen categories to determine the category of each semantic description. Extensive experiments on two public datasets show that our approach significantly outperforms the existing methods.","doi":"10.24963/ijcai.2020/126","url":"https://ijcai.org/proceedings/2020/126","pdf":"https://ijcai.org/proceedings/2020/0126.pdf","markdown":"https://ijcai.org/proceedings/2020/126.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/126","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":126,"pages":{"first":906,"last":912},"keywords":["Machine Learning: Deep Learning","Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Computer Vision: Language and Vision"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25494/file_25494.m3u8","length_sec":330,"provider":"hls"}],"references":[{"position":1,"raw":"[Ba et al., 2015] Jimmy Lei Ba, Kevin Swersky, Sanja Fi- dler, and Ruslan Salakhutdinov. Predicting deep zero-shot convolutional neural networks using textual descriptions. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), ICCV '15, pages 4247-4255. IEEE Computer Society, 2015.","title":"Predicting deep zero-shot convolutional neural networks using textual descriptions","authors":"Jimmy Lei Ba, Kevin Swersky, Sanja Fidler, Ruslan Salakhutdinov","year":"2015","venue":"Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), ICCV '15","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bansal et al., 2018] Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran. Zero-shot object detection. CoRR, abs/1804.04340, 2018.","title":"Zero-shot object detection","authors":"Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, Ajay Divakaran","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Conneau et al., 2016] Alexis Conneau, Holger Schwenk, Loïc Barrault, and Yann LeCun. Very deep convolu- tional networks for natural language processing. CoRR, abs/1606.01781, 2016.","title":"Very deep convolutional networks for natural language processing","authors":"Alexis Conneau, Holger Schwenk, Loïc Barrault, Yann Lecun","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Demirel et al., 2018] Berkan Demirel, Ramazan Gokberk Cinbis, and Nazli Ikizler-Cinbis. Zero-shot object detec- tion by hybrid region embedding. CoRR, abs/1805.06157, 2018.","title":"Zero-shot object detection by hybrid region embedding","authors":"Berkan Demirel, Ramazan Gokberk Cinbis, Nazli Ikizler-Cinbis","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Frome et al., 2013] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic em- bedding model. In Advances in Neural Information Pro- cessing Systems 26, pages 2121-2129.","title":"Devise: A deep visual-semantic embedding model","authors":"Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc' Aurelio Ranzato, Tomas Mikolov","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"Curran Associates, Inc., 2013.","title":null,"authors":"Curran Associates, Inc","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Girshick et al., 2013] Ross B. Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierar- chies for accurate object detection and semantic segmen- tation. CoRR, abs/1311.2524, 2013.","title":"Rich feature hierarchies for accurate object detection and semantic segmentation","authors":"Ross B Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Girshick, 2015] Ross B. Girshick. Fast R-CNN. CoRR, abs/1504.08083, 2015.","title":"Fast R-CNN","authors":"Ross B Girshick","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CoRR, abs/1512.03385, 2015.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. CoRR, abs/1703.06870, 2017.","title":"Mask R-CNN","authors":"He","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2014] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. [Krishna et al., 2016]","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual genome: Connecting language and vision using crowd- sourced dense image annotations. CoRR, abs/1602.07332, 2016.","title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","authors":"Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael S Bernstein, Fei-Fei Li","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2019] Zhihui Li, Lina Yao, Xiaoqin Zhang, Xi- anzhi Wang, Salil Kanhere, and Huaxiang Zhang. Zero- shot object detection with textual descriptions. Proceed- ings of the AAAI Conference on Artificial Intelligence, 33(01):8690-8697, Jul. 2019.","title":"Zeroshot object detection with textual descriptions","authors":"Zhihui Li, Lina Yao, Xiaoqin Zhang, Xianzhi Wang, Salil Kanhere, Huaxiang Zhang","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge J. Be- longie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs/1405.0312, 2014.","title":"Microsoft COCO: common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge J Belongie, Lubomir D Bourdev, Ross B Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence Zitnick","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2015] Wei Liu, Dragomir Anguelov, Dumitru Er- han, Christian Szegedy, Scott E. Reed, Cheng-Yang Fu, and Alexander C. Berg. SSD: single shot multibox detec- tor. CoRR, abs/1512.02325, 2015.","title":"SSD: single shot multibox detector","authors":"Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E Reed, Cheng-Yang Fu, Alexander C Berg","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Empirical Meth- ods in Natural Language Processing (EMNLP), pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rahman et al., 2018] Shafin Rahman, Salman Hameed Khan, and Fatih Porikli. Zero-shot object detection: Learning to simultaneously recognize and localize novel concepts. CoRR, abs/1803.06049, 2018.","title":"Zero-shot object detection: Learning to simultaneously recognize and localize novel concepts","authors":"Shafin Rahman, Salman Hameed Khan, Fatih Porikli","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Reed et al., 2016] Scott E. Reed, Zeynep Akata, Bernt Schiele, and Honglak Lee. Learning deep repre- sentations of fine-grained visual descriptions. CoRR, abs/1605.05395, 2016.","title":"Learning deep representations of fine-grained visual descriptions","authors":"Scott E Reed, Zeynep Akata, Bernt Schiele, Honglak Lee","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross B. Gir- shick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. CoRR, abs/1506.01497, 2015.","title":"Faster R-CNN: towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Shigeto et al., 2015] Yutaro Shigeto, Ikumi Suzuki, Kazuo Hara, Masashi Shimbo, and Yuji Matsumoto. Ridge regression, hubness, and zero-shot learning. In ECML/PKDD, 2015.","title":"Ridge regression, hubness, and zero-shot learning","authors":"Yutaro Shigeto, Ikumi Suzuki, Kazuo Hara, Masashi Shimbo, Yuji Matsumoto","year":"2015","venue":"ECML/PKDD","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shrivastava et al., 2016] Abhinav Shrivastava, Abhinav Gupta, and Ross B. Girshick. Training region-based object detectors with online hard example mining. CoRR, abs/1604.03540, 2016.","title":"Training region-based object detectors with online hard example mining","authors":"Abhinav Shrivastava, Abhinav Gupta, Ross B Girshick","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Socher et al., 2013] Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. In Advances in Neural Information Processing Systems 26, pages 935-943.","title":"Zero-shot learning through cross-modal transfer","authors":"Richard Socher, Milind Ganjoo, Christopher D Manning, Andrew Ng","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"Curran Associates, Inc., 2013.","title":null,"authors":"Curran Associates, Inc","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Szegedy et al., 2016] Christian Szegedy, Sergey Ioffe, and Vincent Vanhoucke. Inception-v4, inception-resnet and the impact of residual connections on learning. CoRR, abs/1602.07261, 2016.","title":"Christian Szegedy, Sergey Ioffe, and Vincent Vanhoucke. Inception-v4, inception-resnet and the impact of residual connections on learning","authors":"Szegedy","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2016] Li Zhang, Tao Xiang, and Shaogang Gong. Learning a deep embedding model for zero-shot learning. CoRR, abs/1611.05088, 2016.","title":"Learning a deep embedding model for zero-shot learning","authors":"Li Zhang, Tao Xiang, Shaogang Gong","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}