{"type":"ScholarlyArticle","title":"CLE-ViT: Contrastive Learning Encoded Transformer for Ultra-Fine-Grained Visual Categorization","authors":[{"name":"Xiaohan Yu","slug":"xiaohan-yu","given":"Xiaohan","family":"Yu","affiliation":"School of Engineering and Built Environment , Griffith University , Australia"},{"name":"Jun Wang","slug":"jun-wang","given":"Jun","family":"Wang","affiliation":"Department of Computer Science , University of Warwick , UK"},{"name":"Yongsheng Gao","slug":"yongsheng-gao","given":"Yongsheng","family":"Gao","affiliation":"School of Engineering and Built Environment , Griffith University , Australia"}],"abstract":"Ultra-fine-grained visual classification (ultra-FGVC) targets at classifying sub-grained categories of fine-grained objects. This inevitably requires discriminative representation learning within a limited training set. Exploring intrinsic features from the object itself, e.g., predicting the rotation of a given image, has demonstrated great progress towards learning discriminative representation. Yet none of these works consider explicit supervision for learning mutual information at instance level. To this end, this paper introduces CLE-ViT, a novel contrastive learning encoded transformer, to address the fundamental problem in ultra-FGVC. The core design is a self-supervised module that performs self-shuffling and masking and then distinguishes these altered images from other images. This drives the model to learn an optimized feature space that has a large inter-class distance while remaining tolerant to intra-class variations. By incorporating this self-supervised module, the network acquires more knowledge from the intrinsic structure of the input data, which improves the generalization ability without requiring extra manual annotations. CLE-ViT demonstrates strong performance on 7 publicly available datasets, demonstrating its effectiveness in the ultra-FGVC task. The code is available at https://github.com/Markin-Wang/CLEViT.","doi":"10.24963/ijcai.2023/504","url":"https://ijcai.org/proceedings/2023/504","pdf":"https://ijcai.org/proceedings/2023/0504.pdf","markdown":"https://ijcai.org/proceedings/2023/504.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/504","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":504,"pages":{"first":4531,"last":4539},"keywords":["Machine Learning: ML: Applications","Machine Learning: ML: Classification"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2019] Yue Chen, Yalong Bai, Wei Zhang, and Tao Mei. Destruction and construction learning for fine- grained image recognition. In CVPR, pages 5157-5166, 2019.","title":"Destruction and construction learning for finegrained image recognition","authors":"Yue Chen, Yalong Bai, Wei Zhang, Tao Mei","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2020a] Ting Chen, Simon Kornblith, Moham- mad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, pages 1597-1607, 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Chen","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2020b] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020.","title":"Improved baselines with momentum contrastive learning","authors":"Chen","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Choe and Shim, 2019] Junsuk Choe and Hyunjung Shim. Attention-based dropout layer for weakly supervised ob- ject localization. In CVPR, pages 2219-2228, 2019.","title":"Attention-based dropout layer for weakly supervised object localization","authors":"Shim ; ] Junsuk Choe, Hyunjung Choe, Shim","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255, 2009.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[DeVries and Taylor, 2017] Terrance DeVries and Gra- ham W Taylor. Improved regularization of convolu- tional neural networks with cutout. arXiv preprint arXiv:1708.04552, 2017.","title":"Improved regularization of convolutional neural networks with cutout","authors":"Terrance Devries, Graham W Taylor","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Doersch et al., 2015] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learn- ing by context prediction. In ICCV, pages 1422-1430, 2015.","title":"Unsupervised visual representation learning by context prediction","authors":"Carl Doersch, Abhinav Gupta, Alexei A Efros","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Du et al., 2020] Ruoyi Du, Dongliang Chang, Ayan Kumar Bhunia, Jiyang Xie, Zhanyu Ma, Yi-Zhe Song, and Jun Guo. Fine-grained visual classification via progressive multi-granularity training of jigsaw patches. In ECCV, pages 153-168. Springer, 2020.","title":"Fine-grained visual classification via progressive multi-granularity training of jigsaw patches","authors":"Du","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ge et al., 2019] Weifeng Ge, Xiangru Lin, and Yizhou Yu. Weakly supervised complementary parts models for fine- grained image classification from the bottom up. In CVPR, pages 3034-3043, 2019.","title":"Weakly supervised complementary parts models for finegrained image classification from the bottom up","authors":"Weifeng Ge, Xiangru Lin, Yizhou Yu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gidaris et al., 2018] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.","title":"Unsupervised representation learning by predicting image rotations","authors":"Spyros Gidaris, Praveer Singh, Nikos Komodakis","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Grill et al., 2020] Jean-Bastien Grill, Florian Strub, Flo- rent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhao- han Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learn- ing. In NeurIPS, volume 33, pages 21271-21284, 2020.","title":"Bootstrap your own latent-a new approach to self-supervised learning","authors":"Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[He et al., 2022] Ju He, Jie-Neng Chen, Shuai Liu, Adam Kortylewski, Cheng Yang, Yutong Bai, and Changhu Wang. Transfg: A transformer architecture for fine- grained recognition. In AAAI, volume 36, pages 852-860, 2022.","title":"Transfg: A transformer architecture for finegrained recognition","authors":"He","year":"2022","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Huang and Li, 2020] Zixuan Huang and Yin Li. Inter- pretable and accurate fine-grained recognition via region grouping. In CVPR, pages 8662-8672, 2020.","title":"Interpretable and accurate fine-grained recognition via region grouping","authors":"Li ; Huang, Zixuan Huang, Yin Li","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, pages 1097- 1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Larese et al., 2014] Mónica G Larese, Rafael Namías, Roque M Craviotto, Miriam R Arango, Carina Gallo, and Pablo M Granitto. Automatic classification of legumes using leaf vein image features. Pattern Recognition, 47(1):158-168, 2014.","title":"Automatic classification of legumes using leaf vein image features","authors":"Rafael Mónica G Larese, Namías, M Roque, Miriam R Craviotto, Carina Arango, Pablo M Gallo, Granitto","year":"2014","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV), 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Oord et al., 2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03748, 2018.","title":"Representation learning with contrastive predictive coding","authors":"Aaron Van Den Oord, Yazhe Li, Oriol Vinyals","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Singh and Lee, 2017] Krishna Kumar Singh and Yong Jae Lee. Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization. In ICCV, pages 3544-3553, 2017.","title":"Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization","authors":"Lee ; Singh, Krishna Kumar Singh, Yong Jae, Lee","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Thapa et al., 2020] Ranjita Thapa, Noah Snavely, Serge Be- longie, and Awais Khan. The plant pathology 2020 chal- lenge dataset to classify foliar disease of apples. arXiv preprint arXiv:2004.11958, 2020.","title":"The plant pathology 2020 challenge dataset to classify foliar disease of apples","authors":"Ranjita Thapa, Noah Snavely, Serge Belongie, Awais Khan","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Touvron et al., 2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablay- rolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICLR, pages 10347-10357, 2021.","title":"Training data-efficient image transformers & distillation through attention","authors":"Touvron","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, volume 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wah et al., 2011] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech- ucsd birds-200-2011 dataset. Technical Report CNS-TR- 2011-001, 2011.","title":"The caltechucsd birds-200-2011 dataset","authors":"Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, Serge Belongie","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2021a] Jun Wang, Xiaohan Yu, and Yongsheng Gao. Feature fusion vision transformer for fine-grained visual categorization. In BMVC, 2021.","title":"Feature fusion vision transformer for fine-grained visual categorization","authors":"Wang","year":"2021","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2021b] Peng Wang, Kai Han, Xiu-Shen Wei, Lei Zhang, and Lei Wang. Contrastive learning based hy- brid networks for long-tailed image classification. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 943-952, 2021.","title":"Contrastive learning based hy-brid networks for long-tailed image classification","authors":"Wang","year":"2021","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yu et al., 2020] Xiaohan Yu, Yang Zhao, Yongsheng Gao, Shengwu Xiong, and Xiaohui Yuan. Patchy image struc- ture classification using multi-orientation region trans- form. In AAAI, volume 34, pages 12741-12748, 2020.","title":"Patchy image structure classification using multi-orientation region transform","authors":"Xiaohan Yu, Yang Zhao, Yongsheng Gao, Shengwu Xiong, Xiaohui Yuan","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yu et al., 2021a] Xiaohan Yu, Yang Zhao, Yongsheng Gao, and Shengwu Xiong. Maskcov: A random mask covari- ance network for ultra-fine-grained visual categorization. Pattern Recognition, 119:108067, 2021.","title":"Maskcov: A random mask covariance network for ultra-fine-grained visual categorization","authors":"Yu","year":"2021","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yu et al., 2021b] Xiaohan Yu, Yang Zhao, Yongsheng Gao, Xiaohui Yuan, and Shengwu Xiong. Benchmark platform for ultra-fine-grained visual categorization beyond human performance. In ICCV, pages 10285-10295, 2021.","title":"Benchmark platform for ultra-fine-grained visual categorization beyond human performance","authors":"Yu","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yu et al., 2022] Xiaohan Yu, Yang Zhao, and Yongsheng Gao. Spare: Self-supervised part erasing for ultra- fine-grained visual categorization. Pattern Recognition, 128:108691, 2022.","title":"Spare: Self-supervised part erasing for ultrafine-grained visual categorization","authors":"Xiaohan Yu, Yang Zhao, Yongsheng Gao","year":"2022","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yu et al., 2023] Xiaohan Yu, Jun Wang, Yang Zhao, and Yongsheng Gao. Mix-vit: Mixing attentive vision trans- former for ultra-fine-grained visual categorization. Pattern Recognition, 135:109131, 2023.","title":"Mix-vit: Mixing attentive vision transformer for ultra-fine-grained visual categorization","authors":"Xiaohan Yu, Jun Wang, Yang Zhao, Yongsheng Gao","year":"2023","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yun et al., 2019] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, pages 6023-6032, 2019.","title":"Cutmix: Regularization strategy to train strong classifiers with localizable features","authors":"Yun","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhuang et al., 2020] Peiqin Zhuang, Yali Wang, and Yu Qiao. Learning attentive pairwise interaction for fine-grained classification. In AAAI, volume 34, pages 13130-13137, 2020.","title":"Learning attentive pairwise interaction for fine-grained classification","authors":"Zhuang","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}