{"type":"ScholarlyArticle","title":"Knowledge-Embedded Representation Learning for Fine-Grained Image Recognition","authors":[{"name":"Tianshui Chen","slug":"tianshui-chen","given":"Tianshui","family":"Chen","affiliation":"Sun Yat-sen University , China"},{"name":"Liang Lin","slug":"liang-lin","given":"Liang","family":"Lin","affiliation":"Sun Yat-sen University , China; SenseTime Research , China"},{"name":"Riquan Chen","slug":"riquan-chen","given":"Riquan","family":"Chen","affiliation":"Sun Yat-sen University , China"},{"name":"Yang Wu","slug":"yang-wu","given":"Yang","family":"Wu","affiliation":"Sun Yat-sen University , China"},{"name":"Xiaonan Luo","slug":"xiaonan-luo","given":"Xiaonan","family":"Luo","affiliation":"Guilin University of Electronic Technology , China"}],"abstract":"Humans can naturally understand an image in depth with the aid of rich knowledge accumulated from daily lives or professions. For example, to achieve fine-grained image recognition (e.g., categorizing hundreds of subordinate categories of birds) usually requires a comprehensive visual concept organization including category labels and part-level attributes. In this work, we investigate how to unify rich professional knowledge with deep neural network architectures and propose a Knowledge-Embedded Representation Learning (KERL) framework for handling the problem of fine-grained image recognition. Specifically, we organize the rich visual concepts in the form of knowledge graph and employ a Gated Graph Neural Network to propagate node message through the graph for generating the knowledge representation. By introducing a novel gated mechanism, our KERL framework incorporates this knowledge representation into the discriminative image feature learning, i.e., implicitly associating the specific attributes with the feature maps. Compared with existing methods of fine-grained image classification, our KERL framework has several appealing properties: i) The embedded high-level knowledge enhances the feature representation, thus facilitating distinguishing the subtle differences among subordinate categories. ii) Our framework can learn feature maps with a meaningful configuration that the highlighted regions finely accord with the nodes (specific attributes) of the knowledge graph. Extensive experiments on the widely used Caltech-UCSD bird dataset demonstrate the superiority of our KERL framework over existing state-of-the-art methods.","doi":"10.24963/ijcai.2018/87","url":"https://ijcai.org/proceedings/2018/87","pdf":"https://ijcai.org/proceedings/2018/0087.pdf","markdown":"https://ijcai.org/proceedings/2018/87.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/87","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":87,"pages":{"first":627,"last":634},"keywords":["Computer Vision: Computer Vision","Computer Vision: Structural and Model-Based Approaches, Knowledge Representation and Reasoning"],"videos":[],"references":[{"position":1,"raw":"[Branson et al., 2014] Steve Branson, Grant Van Horn, Serge Belongie, and Pietro Perona. Bird species categorization using pose normalized deep convolutional nets. arXiv preprint arXiv:1406.2952, 2014.","title":"Bird species categorization using pose normalized deep convolutional nets","authors":"Steve Branson, Grant Van Horn, Serge Belongie, Pietro Perona","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2016] Tianshui Chen, Liang Lin, Lingbo Liu, Xiaonan Luo, and Xuelong Li. Disc: Deep image saliency computing via progressive representation learn- ing. TNNLS, 27(6):1135-1149, 2016.","title":"Disc: Deep image saliency computing via progressive representation learning","authors":"Chen","year":"2016","venue":"TNNLS","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2018] Tianshui Chen, Zhouxia Wang, Guanbin Li, and Liang Lin. Recurrent attentional reinforcement learning for multi-label image recognition. In AAAI, pages 6730-6737, 2018.","title":"Recurrent attentional reinforcement learning for multi-label image recognition","authors":"Chen","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Duvenaud et al., 2015] David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Tim- othy Hirzel, Alán Aspuru-Guzik, and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. In NIPS, pages 2224-2232, 2015.","title":"Convolutional networks on graphs for learning molecular fingerprints","authors":"Dougal David K Duvenaud, Jorge Maclaurin, Rafael Iparraguirre, Timothy Bombarell, Alán Hirzel, Ryan P Aspuru-Guzik, Adams","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fu et al., 2017] Jianlong Fu, Heliang Zheng, and Tao Mei. Look closer to see better: recurrent attention convolu- tional neural network for fine-grained image recognition. In CVPR, pages 4438-4446, 2017.","title":"Look closer to see better: recurrent attention convolutional neural network for fine-grained image recognition","authors":"Jianlong Fu, Heliang Zheng, Tao Mei","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gao et al., 2016] Yang Gao, Oscar Beijbom, Ning Zhang, and Trevor Darrell. Compact bilinear pooling. In CVPR, pages 317-326, 2016.","title":"Compact bilinear pooling","authors":"Yang Gao, Oscar Beijbom, Ning Zhang, Trevor Darrell","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He and Peng, 2017a] Xiangteng He and Yuxin Peng. Fine- graind image classification via combining vision and lan- guage. CVPR, pages 5994-6002, 2017.","title":"Xiangteng He and Yuxin Peng. Finegraind image classification via combining vision and language","authors":"Peng He","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He and Peng, 2017b] Xiangteng He and Yuxin Peng. Weakly supervised learning of part selection model with spatial constraints for fine-grained image classification. In AAAI, pages 4075-4081, 2017.","title":"Xiangteng He and Yuxin Peng. Weakly supervised learning of part selection model with spatial constraints for fine-grained image classification","authors":"Peng He","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2016] Shaoli Huang, Zhe Xu, Dacheng Tao, and Ya Zhang. Part-stacked cnn for fine-grained visual categorization. In CVPR, pages 1173-1182, 2016.","title":"Part-stacked cnn for fine-grained visual categorization","authors":"Huang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jaderberg et al., 2015] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In NIPS, pages 2017-2025, 2015.","title":"Spatial transformer networks","authors":"Jaderberg","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kong and Fowlkes, 2017] Shu Kong and Charless Fowlkes. Low-rank bilinear pooling for fine-grained classification. In CVPR, pages 7025-7034, 2017.","title":"Low-rank bilinear pooling for fine-grained classification","authors":"Shu Kong, Charless Fowlkes","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krause et al., 2015] Jonathan Krause, Hailin Jin, Jianchao Yang, and Li Fei-Fei. Fine-grained recognition without part annotations. In CVPR, pages 5546-5555, 2015.","title":"Fine-grained recognition without part annotations","authors":"Jonathan Krause, Hailin Jin, Jianchao Yang, Li Fei-Fei","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lao et al., 2011] Ni Lao, Tom Mitchell, and William W Co- hen. Random walk inference and learning in a large scale knowledge base. In EMNLP, pages 529-539, 2011.","title":"Random walk inference and learning in a large scale knowledge base","authors":"Ni Lao, Tom Mitchell, William W Cohen","year":"2011","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2015] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493, 2015.","title":"Gated graph sequence neural networks","authors":"Yujia Li, Daniel Tarlow, Marc Brockschmidt, Richard Zemel","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2017] Ruiyu Li, Makarand Tapaswi, Renjie Liao, Jiaya Jia, Raquel Urtasun, and Sanja Fidler. Situation recognition with graph neural networks. In CVPR, pages 4173-4182, 2017.","title":"Situation recognition with graph neural networks","authors":"Li","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liang et al., 2016] Xiaodan Liang, Xiaohui Shen, Jiashi Feng, Liang Lin, and Shuicheng Yan. Semantic object parsing with graph lstm. In ECCV, pages 125-143, 2016.","title":"Semantic object parsing with graph lstm","authors":"Xiaodan Liang, Xiaohui Shen, Jiashi Feng, Liang Lin, Shuicheng Yan","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lin et al., 2015a] Di Lin, Xiaoyong Shen, Cewu Lu, and Ji- aya Jia. Deep lac: Deep localization, alignment and clas- sification for fine-grained recognition. In CVPR, pages 1666-1674, 2015.","title":"Deep lac: Deep localization, alignment and classification for fine-grained recognition","authors":"Lin","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lin et al., 2015b] Tsung-Yu Lin, Aruni RoyChowdhury, and Subhransu Maji. Bilinear cnn models for fine-grained visual recognition. In ICCV, pages 1449-1457, 2015.","title":"Bilinear cnn models for fine-grained visual recognition","authors":"Lin","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lin et al., 2017] Liang Lin, Lili Huang, Tianshui Chen, Yukang Gan, and Hui Cheng. Knowledge-guided recur- rent neural network learning for task-oriented action pre- diction. In ICME, pages 625-630, 2017.","title":"Knowledge-guided recurrent neural network learning for task-oriented action prediction","authors":"Liang Lin, Lili Huang, Tianshui Chen, Yukang Gan, Hui Cheng","year":"2017","venue":"ICME","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2016] Xiao Liu, Tian Xia, Jiang Wang, and Yuan- qing Lin. Fully convolutional attention localization net- works: Efficient attention localization for fine-grained recognition. arXiv preprint arXiv:1603.06765, 2016.","title":"Fully convolutional attention localization networks: Efficient attention localization for fine-grained recognition","authors":"Xiao Liu, Tian Xia, Jiang Wang, Yuanqing Lin","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2017] Xiao Liu, Jiang Wang, Shilei Wen, Er- rui Ding, and Yuanqing Lin. Localizing by describing: Attribute-guided attention localization for fine-grained recognition. In AAAI, pages 4190-4196, 2017.","title":"Localizing by describing: Attribute-guided attention localization for fine-grained recognition","authors":"Xiao Liu, Jiang Wang, Shilei Wen, Errui Ding, Yuanqing Lin","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Malisiewicz and Efros, 2009] Tomasz Malisiewicz and Alyosha Efros. Beyond categories: The visual memex model for reasoning about object relationships. In NIPS, pages 1222-1230, 2009.","title":"Beyond categories: The visual memex model for reasoning about object relationships","authors":"Efros Malisiewicz, Tomasz Malisiewicz, Alyosha Efros","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Marino et al., 2017] Kenneth Marino, Ruslan Salakhutdi- nov, and Abhinav Gupta. The more you know: Using knowledge graphs for image classification. In CVPR, pages 2673-2681, 2017.","title":"The more you know: Using knowledge graphs for image classification","authors":"Kenneth Marino, Ruslan Salakhutdinov, Abhinav Gupta","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Mnih et al., 2014] Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In NIPS, pages 2204-2212, 2014.","title":"Recurrent models of visual attention","authors":"Volodymyr Mnih, Nicolas Heess, Alex Graves","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Niepert et al., 2016] Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. Learning convolutional neural networks for graphs. In ICML, pages 2014-2023, 2016.","title":"Learning convolutional neural networks for graphs","authors":"Mathias Niepert, Mohamed Ahmed, Konstantin Kutzkov","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Peng et al., 2018] Yuxin Peng, Xiangteng He, and Junjie Zhao. Object-part attention model for fine-grained image classification. TIP, 27(3):1487-1500, 2018.","title":"Object-part attention model for fine-grained image classification","authors":"Peng","year":"2018","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Qi et al., 2017] Xiaojuan Qi, Renjie Liao, Jiaya Jia, Sanja Fidler, and Raquel Urtasun. 3d graph neural networks for rgbd semantic segmentation. In CVPR, pages 5199-5208, 2017.","title":"3d graph neural networks for rgbd semantic segmentation","authors":"Xiaojuan Qi, Renjie Liao, Jiaya Jia, Sanja Fidler, Raquel Urtasun","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wah et al., 2011] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech- ucsd birds-200-2011 dataset. 2011.","title":"The caltechucsd birds-200-2011 dataset","authors":"Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, Serge Belongie","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Wang et al., 2015] Dequan Wang, Zhiqiang Shen, Jie Shao, Wei Zhang, Xiangyang Xue, and Zheng Zhang. Multiple granularity descriptors for fine-grained categorization. In ICCV, pages 2399-2406, 2015.","title":"Multiple granularity descriptors for fine-grained categorization","authors":"Dequan Wang, Zhiqiang Shen, Jie Shao, Wei Zhang, Xiangyang Xue, Zheng Zhang","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wang et al., 2017] Zhouxia Wang, Tianshui Chen, Guanbin Li, Ruijia Xu, and Liang Lin. Multi-label image recog- nition by recurrently discovering attentional regions. In ICCV, pages 464-472, 2017.","title":"Multi-label image recognition by recurrently discovering attentional regions","authors":"Wang","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Xiao et al., 2015] Tianjun Xiao, Yichong Xu, Kuiyuan Yang, Jiaxing Zhang, Yuxin Peng, and Zheng Zhang. The application of two-level attention models in deep convolu- tional neural network for fine-grained image classification. In CVPR, pages 842-850, 2015.","title":"The application of two-level attention models in deep convolutional neural network for fine-grained image classification","authors":"Tianjun Xiao, Yichong Xu, Kuiyuan Yang, Jiaxing Zhang, Yuxin Peng, Zheng Zhang","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2014] Ning Zhang, Jeff Donahue, Ross Gir- shick, and Trevor Darrell. Part-based r-cnns for fine- grained category detection. In ECCV, pages 834-849, 2014.","title":"Part-based r-cnns for finegrained category detection","authors":"Ning Zhang, Jeff Donahue, Ross Girshick, Trevor Darrell","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhang et al., 2016a] Han Zhang, Tao Xu, Mohamed Elho- seiny, Xiaolei Huang, Shaoting Zhang, Ahmed Elgammal, and Dimitris Metaxas. Spda-cnn: Unifying semantic part detection and abstraction for fine-grained recognition. In CVPR, pages 1143-1152, 2016.","title":"Spda-cnn: Unifying semantic part detection and abstraction for fine-grained recognition","authors":"Zhang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2016b] Xiaopeng Zhang, Hongkai Xiong, Wengang Zhou, Weiyao Lin, and Qi Tian. Picking deep fil- ter responses for fine-grained image recognition. In CVPR, pages 1134-1142, 2016.","title":"Picking deep filter responses for fine-grained image recognition","authors":"Zhang","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhao et al., 2017] Bo Zhao, Xiao Wu, Jiashi Feng, Qiang Peng, and Shuicheng Yan. Diversified visual attention net- works for fine-grained object classification. TMM, pages 1245-1256, 2017.","title":"Diversified visual attention networks for fine-grained object classification","authors":"Bo Zhao, Xiao Wu, Jiashi Feng, Qiang Peng, Shuicheng Yan","year":"2017","venue":"TMM","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zheng et al., 2017] Heliang Zheng, Jianlong Fu, Tao Mei, and Jiebo Luo. Learning multi-attention convolutional neural network for fine-grained image recognition. In CVPR, pages 396-404, 2017.","title":"Learning multi-attention convolutional neural network for fine-grained image recognition","authors":"Zheng","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Zhou et al., 2016] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, pages 2921-2929, 2016.","title":"Learning deep features for discriminative localization","authors":"Zhou","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Zhu et al., 2014] Yuke Zhu, Alireza Fathi, and Li Fei-Fei. Reasoning about object affordances in a knowledge base representation. In ECCV, pages 408-424, 2014.","title":"Reasoning about object affordances in a knowledge base representation","authors":"Yuke Zhu, Alireza Fathi, Li Fei-Fei","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}