{"type":"ScholarlyArticle","title":"Multi-scale and Discriminative Part Detectors Based Features for Multi-label Image Classification","authors":[{"name":"Gong Cheng","slug":"gong-cheng","given":"Gong","family":"Cheng","affiliation":""},{"name":"Decheng Gao","slug":"decheng-gao","given":"Decheng","family":"Gao","affiliation":""},{"name":"Yang Liu","slug":"yang-liu","given":"Yang","family":"Liu","affiliation":""},{"name":"Junwei Han","slug":"junwei-han","given":"Junwei","family":"Han","affiliation":""}],"abstract":"Convolutional neural networks (CNNs) have shown their promise for image classification task. However, global CNN features still lack geometric invariance for addressing the problem of intra-class variations and so are not optimal for multi-label image classification. This paper proposes a new and effective framework built upon CNNs to learn Multi-scale and Discriminative Part Detectors (MsDPD)-based feature representations for multi-label image classification. Specifically, at each scale level, we (i) first present an entropy-rank based scheme to generate and select a set of discriminative part detectors (DPD), and then (ii) obtain a number of DPD-based convolutional feature maps with each feature map representing the occurrence probability of a particular part detector and learn DPD-based features by using a task-driven pooling scheme. The two steps are formulated into a unified framework by developing a new objective function, which jointly trains part detectors incrementally and integrates the learning of feature representations into the classification task. Finally, the multi-scale features are fused to produce the predictions. Experimental results on PASCAL VOC 2007 and VOC 2012 datasets demonstrate that the proposed method achieves better accuracy when compared with the existing state-of-the-art multi-label classification methods.","doi":"10.24963/ijcai.2018/90","url":"https://ijcai.org/proceedings/2018/90","pdf":"https://ijcai.org/proceedings/2018/0090.pdf","markdown":"https://ijcai.org/proceedings/2018/90.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/90","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Computer Vision","track":"Main track","number":90,"pages":{"first":649,"last":655},"keywords":["Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Machine Learning: Classification","Machine Learning: Multi-instance;Multi-label;Multi-view learning"],"videos":[],"references":[{"position":1,"raw":"[Everingham et al., 2015] M. Everingham, S. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. IJCV, 111(1): 98-136, 2015.","title":"The pascal visual object classes challenge: A retrospective","authors":"M Everingham, S A Eslami, L Van Gool, C K Williams, J Winn, A Zisserman","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Geng and Luo, 2014] X. Geng and L. Luo. Multilabel ranking with inconsistent rankers. In CVPR, 2014.","title":"Multilabel ranking with inconsistent rankers","authors":"Luo Geng, X Geng, L Luo","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[George and Floerkemeier, 2014] M. George and C. Floerkemeier. Recognizing products: A per-exemplar multi-label image classification approach. In ECCV, 2014.","title":"Recognizing products: A per-exemplar multi-label image classification approach","authors":"M George, C Floerkemeier","year":"2014","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gong et al., 2013] Y. Gong, Y. Jia, T. Leung, A. Toshev, and S. Ioffe. Deep convolutional ranking for multilabel image annotation. arXiv preprint arXiv:1312.4894, 2013.","title":"Deep convolutional ranking for multilabel image annotation","authors":"Y Gong, Y Jia, T Leung, A Toshev, S Ioffe","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2016] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"K He, X Zhang, S Ren, J Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Herranz et al., 2016] L. Herranz, S. Jiang, and X. Li. Scene recognition with CNNs: objects, scales and dataset bias. In CVPR, 2016.","title":"Scene recognition with CNNs: objects, scales and dataset bias","authors":"L Herranz, S Jiang, X Li","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jing et al., 2015] L. Jing, L. Yang, J. Yu, and M. K. Ng. Semi-supervised low-rank mapping learning for multi-label classification. In CVPR, 2015.","title":"Semi-supervised low-rank mapping learning for multi-label classification","authors":"L Jing, L Yang, J Yu, M K Ng","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Krizhevsky et al., 2012] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"A Krizhevsky, I Sutskever, G E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li et al., 2016a] C. Li, B. Wang, V. Pavlu, and J. Aslam. Conditional bernoulli mixtures for multi-label classification. In ICML, 2016a.","title":"Conditional bernoulli mixtures for multi-label classification","authors":"Li","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2016b] Q. Li, M. Qiao, W. Bian, and D. Tao. Conditional graphical lasso for multi-label image classification. In CVPR, 2016b.","title":"Conditional graphical lasso for multi-label image classification","authors":"Li","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2017] Y. Li, Y. Song, and J. Luo. Improving Pairwise Ranking for Multi-label Image Classification. In CVPR, 2017.","title":"Improving Pairwise Ranking for Multi-label Image Classification","authors":"Y Li, Y Song, J Luo","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2016] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg. Ssd: Single shot multibox detector. In ECCV, 2016.","title":"Ssd: Single shot multibox detector","authors":"W Liu, D Anguelov, D Erhan, C Szegedy, S Reed, C.-Y Fu, A C Berg","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Murray and Perronnin, 2014] N. Murray and F. Perronnin. Generalized max pooling. In CVPR, 2014.","title":"Generalized max pooling","authors":"N Murray, F Perronnin","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Murthy et al., 2016] V. N. Murthy, V. Singh, T. Chen, R. Manmatha, and D. Comaniciu. Deep decision network for multi-class image classification. In CVPR, 2016.","title":"Deep decision network for multi-class image classification","authors":"V N Murthy, V Singh, T Chen, R Manmatha, D Comaniciu","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Oquab et al., 2014] M. Oquab, L. Bottou, I. Laptev, and J. Sivic. Learning and transferring mid-level image representations using convolutional neural networks. In CVPR, 2014.","title":"Learning and transferring mid-level image representations using convolutional neural networks","authors":"M Oquab, L Bottou, I Laptev, J Sivic","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sharif Razavian et al., 2014] A. Sharif Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. CNN features off-the-shelf: an astounding baseline for recognition. In CVPRW, 2014.","title":"CNN features off-the-shelf: an astounding baseline for recognition","authors":"Sharif Razavian","year":"2014","venue":"CVPRW","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Simon et al., 2014] M. Simon, E. Rodner, and J. Denzler. Part detector discovery in deep convolutional neural networks. In ACCV, 2014.","title":"Part detector discovery in deep convolutional neural networks","authors":"M Simon, E Rodner, J Denzler","year":"2014","venue":"ACCV","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Simonyan and Zisserman, 2015] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"K Simonyan, A Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Szegedy et al., 2015] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR, 2015.","title":"Going deeper with convolutions","authors":"C Szegedy, W Liu, Y Jia, P Sermanet, S Reed, D Anguelov, D Erhan, V Vanhoucke, A Rabinovich","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tan et al., 2015] M. Tan, Q. Shi, A. van den Hengel, C. Shen, J. Gao, F. Hu, and Z. Zhang. Learning graph structure for multi-label image classification via clique generation. In CVPR, 2015.","title":"Learning graph structure for multi-label image classification via clique generation","authors":"M Tan, Q Shi, A Van Den Hengel, C Shen, J Gao, F Hu, Z Zhang","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2016] J. Wang, Y. Yang, J. Mao, Z. Huang, C. Huang, and W. Xu. CNN-RNN: A unified framework for multi-label image classification. In CVPR, 2016.","title":"CNN-RNN: A unified framework for multi-label image classification","authors":"J Wang, Y Yang, J Mao, Z Huang, C Huang, W Xu","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wei et al., 2014] Y. Wei, W. Xia, J. Huang, B. Ni, J. Dong, Y. Zhao, and S. Yan. CNN: Single-label to multi-label. arXiv preprint arXiv:1406.5726, 2014.","title":"CNN: Single-label to multi-label","authors":"Y Wei, W Xia, J Huang, B Ni, J Dong, Y Zhao, S Yan","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wei et al., 2016] Y. Wei, W. Xia, M. Lin, J. Huang, B. Ni, J. Dong, Y. Zhao, and S. Yan. HCP: A flexible CNN framework for multi-label image classification. IEEE TPAMI, 38(9): 1901-1907, 2016.","title":"HCP: A flexible CNN framework for multi-label image classification","authors":"Y Wei, W Xia, M Lin, J Huang, B Ni, J Dong, Y Zhao, S Yan","year":"2016","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Williams and Hinton, 1986] D. Williams and G. Hinton. Learning representations by back-propagating errors. Nature, 323(6088): 533-538, 1986.","title":"Learning representations by back-propagating errors","authors":"D Williams, G Hinton","year":"1986","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wu et al., 2015] R. Wu, B. Wang, W. Wang, and Y. Yu. Harvesting discriminative meta objects with deep CNN features for scene classification. In ICCV, 2015.","title":"Harvesting discriminative meta objects with deep CNN features for scene classification","authors":"R Wu, B Wang, W Wang, Y Yu","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xie et al., 2015] G.-S. Xie, X.-Y. Zhang, X. Shu, S. Yan, and C.-L. Liu. Task-driven feature pooling for image classification. In ICCV, 2015.","title":"Task-driven feature pooling for image classification","authors":"G.-S Xie, X.-Y Zhang, X Shu, S Yan, C.-L Liu","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xie et al., 2017a] G.-S. Xie, X.-Y. Zhang, S. Yan, and C.-L. Liu. SDE: A Novel Selective, Discriminative and Equalizing Feature Representation for Visual Recognition. IJCV, 1-24, 2017a.","title":"SDE: A Novel Selective, Discriminative and Equalizing Feature Representation for Visual Recognition","authors":"Xie","year":"2017","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xie et al., 2017b] P. Xie, R. Salakhutdinov, L. Mou, and E. P. Xing. Deep Determinantal Point Process for Large-Scale Multi-Label Classification. In ICCV, 2017b.","title":"Deep Determinantal Point Process for Large-Scale Multi-Label Classification","authors":"Xie","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2016] H. Yang, J. Tianyi Zhou, Y. Zhang, B.-B. Gao, J. Wu, and J. Cai. Exploit bounding box annotations for multi-label object recognition. In CVPR, 2016.","title":"Exploit bounding box annotations for multi-label object recognition","authors":"H Yang, J Tianyi Zhou, Y Zhang, B.-B Gao, J Wu, J Cai","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yeh et al., 2017] C.-K. Yeh, W.-C. Wu, W.-J. Ko, and Y.-C. F. Wang. Learning Deep Latent Space for Multi-Label Classification. In AAAI, 2017.","title":"Learning Deep Latent Space for Multi-Label Classification","authors":"C.-K Yeh, W.-C Wu, W.-J Ko, Y.-C F Wang","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhu et al., 2017] F. Zhu, H. Li, W. Ouyang, N. Yu, and X. Wang. Learning Spatial Regularization with Image-level Supervisions for Multi-label Image Classification. In CVPR, 2017.","title":"Learning Spatial Regularization with Image-level Supervisions for Multi-label Image Classification","authors":"F Zhu, H Li, W Ouyang, N Yu, X Wang","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}