{"type":"ScholarlyArticle","title":"Deciphering the Projection Head: Representation Evaluation Self-supervised Learning","authors":[{"name":"Jiajun Ma","slug":"jiajun-ma","given":"Jiajun","family":"Ma","affiliation":"Hong Kong University of Science and Technology; Hong Kong University of Science and Technology ( Guangzhou )"},{"name":"Tianyang Hu","slug":"tianyang-hu","given":"Tianyang","family":"Hu","affiliation":"Huawei Noah's Ark Lab"},{"name":"Wenjia Wang","slug":"wenjia-wang","given":"Wenjia","family":"Wang","affiliation":"Hong Kong University of Science and Technology; Hong Kong University of Science and Technology ( Guangzhou )"}],"abstract":"Self-supervised learning (SSL) aims to learn the intrinsic features of data without labels. Despite the diverse SSL architectures, the projection head always plays an important role in improving downstream task performance. In this study, we systematically investigate the role of the projection head in SSL. We find that the projection head targets the uniformity aspect, which maps samples into uniform distribution and enables the encoder to focus on extracting semantic features. Drawing on this insight, we propose a Representation Evaluation Design (RED) in SSL models in which a shortcut connection between the representation and the projection vectors is built. Our extensive experiments with different architectures (including SimCLR, MoCo-V2, and SimSiam) on various datasets demonstrate that the RED-SSL consistently outperforms their baseline counterparts in downstream tasks. Furthermore, the RED-SSL learned representations exhibit superior robustness to previously unseen augmentations and out-of-distribution data.","doi":"10.24963/ijcai.2024/522","url":"https://ijcai.org/proceedings/2024/522","pdf":"https://ijcai.org/proceedings/2024/0522.pdf","markdown":"https://ijcai.org/proceedings/2024/522.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/522","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Machine Learning","track":"Main Track","number":522,"pages":{"first":4724,"last":4732},"keywords":["Machine Learning: ML: Representation learning","Machine Learning: ML: Explainable/Interpretable machine learning","Machine Learning: ML: Self-supervised Learning"],"videos":[],"references":[{"position":1,"raw":"[Arora et al., 2019] Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representa- tion learning. arXiv preprint arXiv:1902.09229, 2019.","title":"A theoretical analysis of contrastive unsupervised representation learning","authors":"Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, Nikunj Saunshi","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Beirlant et al., 1997] Jan Beirlant, Edward J Dudewicz, Lás- zló Györfi, Edward C Van der Meulen, et al. Nonparametric entropy estimation: An overview. International Journal of Mathematical and Statistical Sciences, 6(1):17-39, 1997.","title":"Nonparametric entropy estimation: An overview","authors":"Beirlant","year":"1997","venue":"International Journal of Mathematical and Statistical Sciences","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Caron et al., 2020] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Pro- cessing Systems, 33:9912-9924, 2020.","title":"Unsupervised learning of visual features by contrasting cluster assignments","authors":"Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, Armand Joulin","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Caron et al., 2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF inter- national conference on computer vision, pages 9650-9660, 2021.","title":"Emerging properties in self-supervised vision transformers","authors":"Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin","year":"2021","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cazenavette et al., 2021] George Cazenavette, Calvin Mur- dock, and Simon Lucey. Architectural adversarial robust- ness: The case for deep pursuit. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7150-7158, 2021.","title":"Architectural adversarial robustness: The case for deep pursuit","authors":"Cazenavette","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen and He, 2021] Xinlei Chen and Kaiming He. Explor- ing simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 15750-15758, 2021.","title":"Exploring simple siamese representation learning","authors":"Chen, He ;, Xinlei Chen, Kaiming He","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"Chen et al., ] X Chen, S Xie, and K He. An empirical study of training self-supervised vision transformers. in 2021 ieee. In CVF International Conference on Computer Vision (ICCV), pages 9620-9629.","title":"An empirical study of training self-supervised vision transformers","authors":"Chen","year":"2021","venue":"CVF International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2020a] Ting Chen, Simon Kornblith, Moham- mad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Interna- tional conference on machine learning, pages 1597-1607.","title":"A simple framework for contrastive learning of visual representations","authors":"Chen","year":"2020","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"PMLR, 2020.","title":null,"authors":"Pmlr","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chen et al., 2020b] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020.","title":"Improved baselines with momentum contrastive learning","authors":"Chen","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255.","title":"Imagenet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Gao et al., 2021] Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821, 2021.","title":"Simcse: Simple contrastive learning of sentence embeddings","authors":"Gao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Grill et al., 2020] Jean-Bastien Grill, Florian Strub, Flo- rent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhao- han Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learn- ing. Advances in neural information processing systems, 33:21271-21284, 2020.","title":"Bootstrap your own latent-a new approach to self-supervised learning","authors":"Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Gupta et al., 2022] Kartik Gupta, Thalaiyasingam Ajanthan, Anton van den Hengel, and Stephen Gould. Understanding and improving the role of projection head in self-supervised learning. arXiv preprint arXiv:2212.11491, 2022.","title":"Understanding and improving the role of projection head in self-supervised learning","authors":"Kartik Gupta, Thalaiyasingam Ajanthan, Anton Van Den, Stephen Hengel, Gould","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[HaoChen et al., 2021] Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. Ad- vances in Neural Information Processing Systems, 34:5000- 5011, 2021.","title":"Provable guarantees for selfsupervised deep learning with spectral contrastive loss","authors":"Colin Jeff Z Haochen, Adrien Wei, Tengyu Gaidon, Ma","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsuper- vised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.","title":"Momentum contrast for unsupervised visual representation learning","authors":"Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick","year":"2020","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[He et al., 2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.","title":"Masked autoencoders are scalable vision learners","authors":"Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Hu et al., 2022] Tianyang Hu, Zhili Liu, Fengwei Zhou, Wenjia Wang, and Weiran Huang. Your contrastive learn- ing is secretly doing stochastic neighbor embedding. arXiv preprint arXiv:2205.14814, 2022.","title":"Your contrastive learning is secretly doing stochastic neighbor embedding","authors":"Tianyang Hu, Zhili Liu, Fengwei Zhou, Wenjia Wang, Weiran Huang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ji et al., 2021] Wenlong Ji, Zhun Deng, Ryumei Nakada, James Zou, and Linjun Zhang. The power of contrast for feature learning: A theoretical analysis. arXiv preprint arXiv:2110.02473, 2021.","title":"The power of contrast for feature learning: A theoretical analysis","authors":"Wenlong Ji, Zhun Deng, Ryumei Nakada, James Zou, Linjun Zhang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Jing et al., 2021] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.09348, 2021.","title":"Understanding dimensional collapse in contrastive self-supervised learning","authors":"Li Jing, Pascal Vincent, Yann Lecun, Yuandong Tian","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Khosla et al., 2020] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised con- trastive learning. Advances in neural information process- ing systems, 33:18661-18673, 2020.","title":"Supervised contrastive learning","authors":"Khosla","year":"2020","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Krizhevsky, 2009] Alex Krizhevsky. Learning multiple lay- ers of features from tiny images. Technical report, 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Li et al., 2020] Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsu- pervised representations. arXiv preprint arXiv:2005.04966, 2020.","title":"Prototypical contrastive learning of unsu-pervised representations","authors":"Li","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Li et al., 2022] Alexander C Li, Alexei A Efros, and Deepak Pathak. Understanding collapse in non-contrastive siamese representation learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXI, pages 490-505. Springer, 2022.","title":"Understanding collapse in non-contrastive siamese representation learning","authors":"Alexei A Alexander C Li, Deepak Efros, Pathak","year":"2022","venue":"Computer Vision-ECCV 2022: 17th European Conference","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Lombardi and Pant, 2016] Damiano Lombardi and Sanjay Pant. Nonparametric k-nearest-neighbor entropy estimator. Physical Review E, 93(1):013310, 2016.","title":"Nonparametric k-nearest-neighbor entropy estimator","authors":"Damiano Lombardi, Sanjay Pant","year":"2016","venue":"Physical Review E","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Oord et al., 2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.","title":"Representation learning with contrastive predictive coding","authors":"Aaron Van Den Oord, Yazhe Li, Oriol Vinyals","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Purushwalkam and Gupta, 2020] Senthil Purushwalkam and Abhinav Gupta. Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases. Ad- vances in Neural Information Processing Systems, 33:3407- 3418, 2020.","title":"Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases","authors":"Gupta Purushwalkam, Senthil Purushwalkam, Abhinav Gupta","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural lan- guage supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.","title":"Learning transferable visual models from natural language supervision","authors":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark","year":"2021","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Reshniak and Webster, 2020] Viktor Reshniak and Clay- ton G Webster. Robust learning with implicit residual networks. Machine Learning and Knowledge Extraction, 3(1):34-55, 2020.","title":"Viktor Reshniak and Clayton G Webster. Robust learning with implicit residual networks","authors":"Webster Reshniak","year":"2020","venue":"Machine Learning and Knowledge Extraction","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Saunshi et al., 2022] Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy. Understanding con- trastive learning requires incorporating inductive biases. arXiv preprint arXiv:2202.14037, 2022.","title":"Understanding contrastive learning requires incorporating inductive biases","authors":"Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, Akshay Krishnamurthy","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Sohn, 2016] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016.","title":"Improved deep metric learning with multi-class n-pair loss objective","authors":"Kihyuk Sohn, Sohn","year":"2016","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Tian et al., 2021] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning, pages 10268-10278. PMLR, 2021.","title":"Understanding self-supervised learning dynamics without contrastive pairs","authors":"Yuandong Tian, Xinlei Chen, Surya Ganguli","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Tian, 2022] Yuandong Tian. Understanding the role of non- linearity in training dynamics of contrastive learning. arXiv preprint arXiv:2206.01342, 2022.","title":"Understanding the role of nonlinearity in training dynamics of contrastive learning","authors":"Yuandong Tian, Tian","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Tosh et al., 2021] Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redun- dancy, and linear models. In Algorithmic Learning Theory, pages 1179-1206. PMLR, 2021.","title":"Contrastive learning, multi-view redundancy, and linear models","authors":"Christopher Tosh, Akshay Krishnamurthy, Daniel Hsu","year":"2021","venue":"Algorithmic Learning Theory","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Von Kügelgen et al., 2021] Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self- supervised learning with data augmentations provably iso- lates content from style. Advances in neural information processing systems, 34:16451-16467, 2021.","title":"Selfsupervised learning with data augmentations provably isolates content from style","authors":"Von Kügelgen","year":"2021","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Wang and Isola, 2020] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Interna- tional Conference on Machine Learning, pages 9929-9939. PMLR, 2020.","title":"Understanding contrastive representation learning through alignment and uniformity on the hypersphere","authors":"Wang, Isola ;, Tongzhou Wang, Phillip Isola","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Wang and Liu, 2021] Feng Wang and Huaping Liu. Under- standing the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition, pages 2495-2504, 2021.","title":"Understanding the behaviour of contrastive loss","authors":"Wang, Liu ; ] Feng Wang, Huaping Liu","year":"2021","venue":"Proceedings of the IEEE/CVF conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Wang et al., 2021] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3024-3033, 2021.","title":"Dense contrastive learning for self-supervised visual pre-training","authors":"Wang","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Wang et al., 2022a] Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. arXiv preprint arXiv:2203.13457, 2022.","title":"Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap","authors":"Wang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Wang et al., 2022b] Yizhou Wang, Shixiang Tang, Feng Zhu, Lei Bai, Rui Zhao, Donglian Qi, and Wanli Ouyang. Re- visiting the transferability of supervised pretraining: an mlp perspective. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 9183-9193, 2022.","title":"Revisiting the transferability of supervised pretraining: an mlp perspective","authors":"Wang","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Wang et al., 2022c] Zhaoqing Wang, Qiang Li, Guoxin Zhang, Pengfei Wan, Wen Zheng, Nannan Wang, Ming- ming Gong, and Tongliang Liu. Exploring set similarity for dense self-supervised representation learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16590-16599, 2022.","title":"Exploring set similarity for dense self-supervised representation learning","authors":"Wang","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Wen and Li, 2021] Zixin Wen and Yuanzhi Li. Toward un- derstanding the feature learning process of self-supervised contrastive learning. In International Conference on Ma- chine Learning, pages 11112-11122. PMLR, 2021.","title":"Toward understanding the feature learning process of self-supervised contrastive learning","authors":"Wen, Li ; ] Zixin Wen, Yuanzhi Li","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Wen and Li, 2022] Zixin Wen and Yuanzhi Li. The mecha- nism of prediction head in non-contrastive self-supervised learning. arXiv preprint arXiv:2205.06226, 2022.","title":"The mechanism of prediction head in non-contrastive self-supervised learning","authors":"Wen, Li ; ] Zixin Wen, Yuanzhi Li","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Wu et al., 2018] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 3733-3742, 2018.","title":"Unsupervised feature learning via nonparametric instance discrimination","authors":"Zhirong Wu, Yuanjun Xiong, Stella X Yu, Dahua Lin","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Yeh et al., 2022] Chun-Hsiao Yeh, Cheng-Yao Hong, Yen- Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann LeCun. De- coupled contrastive learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, Octo- ber 23-27, 2022, Proceedings, Part XXVI, pages 668-684. Springer, 2022.","title":"Decoupled contrastive learning","authors":"Chun-Hsiao Yeh, Cheng-Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, Yann Lecun","year":"2022","venue":"Computer Vision-ECCV 2022: 17th European Conference","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Zbontar et al., 2021] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In Inter- national Conference on Machine Learning, pages 12310- 12320. PMLR, 2021.","title":"Barlow twins: Selfsupervised learning via redundancy reduction","authors":"Jure Zbontar, Li Jing, Ishan Misra, Yann Lecun, Stéphane Deny","year":"2021","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}