{"type":"ScholarlyArticle","title":"Fine-grained Image Classification by Visual-Semantic Embedding","authors":[{"name":"Huapeng Xu","slug":"huapeng-xu","given":"Huapeng","family":"Xu","affiliation":"Southeast University , Nanjing , China"},{"name":"Guilin Qi","slug":"guilin-qi","given":"Guilin","family":"Qi","affiliation":"Southeast University , Nanjing , China"},{"name":"Jingjing Li","slug":"jingjing-li","given":"Jingjing","family":"Li","affiliation":"University of Electronic Science and Technology of China , Chendu , China"},{"name":"Meng Wang","slug":"meng-wang","given":"Meng","family":"Wang","affiliation":"Xi'an Jiaotong University , Xi'an , China"},{"name":"Kang Xu","slug":"kang-xu","given":"Kang","family":"Xu","affiliation":"Nanjing University of Posts and Telecommunications , Nanjing , China"},{"name":"Huan Gao","slug":"huan-gao","given":"Huan","family":"Gao","affiliation":"Southeast University , Nanjing , China"}],"abstract":"This paper investigates a challenging problem,which is known as fine-grained image classification(FGIC). Different from conventional computer visionproblems, FGIC suffers from the large intraclassdiversities and subtle inter-class differences.Existing FGIC approaches are limited to exploreonly the visual information embedded in the images.In this paper, we present a novel approachwhich can use handy prior knowledge from eitherstructured knowledge bases or unstructured text tofacilitate FGIC. Specifically, we propose a visual-semanticembedding model which explores semanticembedding from knowledge bases and text, andfurther trains a novel end-to-end CNN frameworkto linearly map image features to a rich semanticembedding space. Experimental results on a challenginglarge-scale UCSD Bird-200-2011 datasetverify that our approach outperforms several state-of-the-art methods with significant advances.","doi":"10.24963/ijcai.2018/145","url":"https://ijcai.org/proceedings/2018/145","pdf":"https://ijcai.org/proceedings/2018/0145.pdf","markdown":"https://ijcai.org/proceedings/2018/145.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/145","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":145,"pages":{"first":1043,"last":1049},"keywords":["Machine Learning: Clustering","Machine Learning: Deep Learning","Machine Learning: Knowledge-based Learning","Natural Language Processing: Embeddings","Machine Learning: Feature Selection ; Learning Sparse Models","Computer Vision: Language and Vision"],"videos":[],"references":[{"position":1,"raw":"[Akata et al., 2015] Zeynep Akata, Scott E. Reed, Daniel Walter, Honglak Lee, and Bernt Schiele. Evaluation of output embeddings for fine-grained image classification. In CVPR, pages 2927-2936, 2015.","title":"Evaluation of output embeddings for fine-grained image classification","authors":"Zeynep Akata, Scott E Reed, Daniel Walter, Honglak Lee, Bernt Schiele","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Farhadi et al., 2009] Ali Farhadi, Ian Endres, Derek Hoiem, and David A. Forsyth. Describing objects by their at- tributes. In CVPR, pages 1778-1785, 2009.","title":"Describing objects by their attributes","authors":"Ali Farhadi, Ian Endres, Derek Hoiem, David A Forsyth","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Frome et al., 2013] Andrea Frome, Gregory S. Cor- rado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc'Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In NIPS, pages 2121-2129, 2013.","title":"Devise: A deep visual-semantic embedding model","authors":"Andrea Frome, Gregory S Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc'aurelio Ranzato, Tomas Mikolov","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Girshick et al., 2014] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, pages 580-587, 2014.","title":"Rich feature hierarchies for accurate object detection and semantic segmentation","authors":"Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He and Peng, 2017] Xiangteng He and Yuxin Peng. Fine- grained image classification via combining vision and lan- guage. In CVPR, pages 7332-7340, 2017.","title":"Xiangteng He and Yuxin Peng. Finegrained image classification via combining vision and language","authors":"Peng He","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Huang et al., 2016] Shaoli Huang, Zhe Xu, Dacheng Tao, and Ya Zhang. Part-stacked CNN for fine-grained visual categorization. In CVPR, pages 1173-1182, 2016.","title":"Part-stacked CNN for fine-grained visual categorization","authors":"Huang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In ICML, pages 448-456, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jaderberg et al., 2015] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. In NIPS, pages 2017-2025, 2015.","title":"Spatial transformer networks","authors":"Jaderberg","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1106- 1114, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lehmann et al., 2015] Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Sören Auer, and Christian Bizer. Dbpedia - A large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web Journal, pages 167-195, 2015.","title":"Dbpedia -A large-scale, multilingual knowledge base extracted from wikipedia","authors":"Lehmann","year":"2015","venue":"Semantic Web Journal","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lin et al., 2015a] Di Lin, Xiaoyong Shen, Cewu Lu, and Ji- aya Jia. Deep LAC: deep localization, alignment and clas- sification for fine-grained recognition. In CVPR, pages 1666-1674, 2015.","title":"Deep LAC: deep localization, alignment and classification for fine-grained recognition","authors":"Lin","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin et al., 2015b] Tsung-Yu Lin, Aruni.R Chowdhury, and Subhransu Maji. Bilinear CNN models for fine-grained visual recognition. In ICCV, pages 1449-1457, 2015.","title":"Bilinear CNN models for fine-grained visual recognition","authors":"Lin","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lin et al., 2015c] Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In AAAI, pages 2181-2187, 2015.","title":"Learning entity and relation embeddings for knowledge graph completion","authors":"Lin","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Marino et al., 2017] Kenneth Marino, Ruslan Salakhutdi- nov, and Abhinav Gupta. The more you know: Using knowledge graphs for image classification. In CVPR, pages 20-28, 2017.","title":"The more you know: Using knowledge graphs for image classification","authors":"Marino","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their composi- tionality. In NIPS, pages 3111-3119, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ouyang et al., 2017] Wanli Ouyang, Xingyu Zeng, Xiao- gang Wang, Shi Qiu, Ping Luo, Yonglong Tian, Hong- sheng Li, Shuo Yang, Zhe Wang, Hongyang Li, et al. Deepid-net: Object detection with deformable part based convolutional neural networks. IEEE TPAMI, pages 1320- 1334, 2017.","title":"Deepid-net: Object detection with deformable part based convolutional neural networks","authors":"Wanli Ouyang, Xingyu Zeng, Xiaogang Wang, Shi Qiu, Ping Luo, Yonglong Tian, Hongsheng Li, Shuo Yang, Zhe Wang, Hongyang Li","year":"2017","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Peng et al., 2018] Yuxin Peng, Xiangteng He, and Junjie Zhao. Object-part attention model for fine-grained im- age classification. IEEE Trans. Image Processing, pages 1487-1500, 2018.","title":"Object-part attention model for fine-grained image classification","authors":"Yuxin Peng, Xiangteng He, Junjie Zhao","year":"2018","venue":"IEEE Trans. Image Processing","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sidorov et al., 2014] Grigori Sidorov, Alexander Gelbukh, Helena G.Adorno, and David Pinto. Soft similarity and soft cosine measure: Similarity of features in vector space model. Computación y Sistemas, pages 491-504, 2014.","title":"Soft similarity and soft cosine measure: Similarity of features in vector space model","authors":"Grigori Sidorov, Alexander Gelbukh, Helena G Adorno, David Pinto","year":"2014","venue":"Computación y Sistemas","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., pages 1929-1958, 2014.","title":"Dropout: A simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ruslan Salakhutdinov","year":"2014","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Szegedy et al., 2016] Christian Szegedy, Vincent Van- houcke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, pages 2818-2826, 2016.","title":"Rethinking the inception architecture for computer vision","authors":"Szegedy","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wah et al., 2011] C. Wah, S. Branson, P. Welinder, P. Per- ona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.","title":"The Caltech-UCSD Birds-200-2011 Dataset","authors":"C Wah, S Branson, P Welinder, P Perona, S Belongie","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2016] Sen Wang, Pingbo Pan, Guodong Long, Weitong Chen, Xue Li, and Quan Z. Sheng. Compact representation for large-scale unconstrained video analy- sis. World Wide Web, pages 231-246, 2016.","title":"Compact representation for large-scale unconstrained video analysis","authors":"Sen Wang, Pingbo Pan, Guodong Long, Weitong Chen, Xue Li, Quan Z Sheng","year":"2016","venue":"World Wide Web","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2014] Ning Zhang, Jeff Donahue, Ross B. Gir- shick, and Trevor Darrell. Part-based r-cnns for fine- grained category detection. In ECCV, pages 834-849, 2014.","title":"Part-based r-cnns for finegrained category detection","authors":"Ning Zhang, Jeff Donahue, Ross B Girshick, Trevor Darrell","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2016a] Han Zhang, Tao Xu, Mohamed Elho- seiny, Xiaolei Huang, Shaoting Zhang, Ahmed Elgammal, and Dimitris Metaxas. Spda-cnn: Unifying semantic part detection and abstraction for fine-grained recognition. In CVPR, pages 1143-1152, 2016.","title":"Spda-cnn: Unifying semantic part detection and abstraction for fine-grained recognition","authors":"Zhang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2016b] Xiaopeng Zhang, Hongkai Xiong, Wengang Zhou, Weiyao Lin, and Qi Tian. Picking deep fil- ter responses for fine-grained image recognition. In CVPR, pages 1134-1142, 2016.","title":"Picking deep filter responses for fine-grained image recognition","authors":"Zhang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}