{"type":"ScholarlyArticle","title":"A Survey on Masked Autoencoder for Visual Self-supervised Learning","authors":[{"name":"Chaoning Zhang","slug":"chaoning-zhang","given":"Chaoning","family":"Zhang","affiliation":"Kyung Hee University , South Korea"},{"name":"Chenshuang Zhang","slug":"chenshuang-zhang","given":"Chenshuang","family":"Zhang","affiliation":"Kyung Hee University , South Korea"},{"name":"Junha Song","slug":"junha-song","given":"Junha","family":"Song","affiliation":"Korea Advanced Institute of Science and Technology , South Korea"},{"name":"John Seon Keun Yi","slug":"john-seon-keun-yi","given":"John Seon Keun","family":"Yi","affiliation":"Georgia Institute of Technology , United States"},{"name":"In So Kweon","slug":"in-so-kweon","given":"In So","family":"Kweon","affiliation":"Korea Advanced Institute of Science and Technology , South Korea"}],"abstract":"With the increasing popularity of masked autoencoders, self-supervised learning (SSL) in vision undertakes a similar trajectory as in NLP. Specifically, generative pretext tasks with the masked prediction have become a de facto standard SSL practice in NLP (e.g., BERT). By contrast, early attempts at generative methods in vision have been outperformed by their discriminative counterparts (like contrastive learning). However, the success of masked image modeling has revived the autoencoder-based visual pretraining method. As a milestone to bridge the gap with BERT in NLP, masked autoencoder in vision has attracted unprecedented attention. This work conducts a survey on masked autoencoders for visual SSL.","doi":"10.24963/ijcai.2023/762","url":"https://ijcai.org/proceedings/2023/762","pdf":"https://ijcai.org/proceedings/2023/0762.pdf","markdown":"https://ijcai.org/proceedings/2023/762.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/762","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Survey Track","track":"Survey Track","number":762,"pages":{"first":6805,"last":6813},"keywords":["Survey: Computer Vision","Survey: Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Assran et al., 2022] Mahmoud Assran, Mathilde Caron, Is- han Misra, Piotr Bojanowski, Florian Bordes, Pascal Vin- cent, Armand Joulin, Michael Rabbat, and Nicolas Bal- las. Masked siamese networks for label-efficient learning. arXiv preprint arXiv:2204.07141, 2022.","title":"Masked siamese networks for label-efficient learning","authors":"Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, Nicolas Ballas","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Baevski et al., 2022] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint arXiv:2202.03555, 2022.","title":"Data2vec: A general framework for self-supervised learning in speech, vision and language","authors":"Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, Michael Auli","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bao et al., 2022] Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. ICLR, 2022.","title":"Beit: Bert pre-training of image transformers","authors":"Bao","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cao et al., 2022] Shuhao Cao, Peng Xu, and David A Clifton. How to understand masked autoencoders. arXiv preprint arXiv:2202.03670, 2022.","title":"How to understand masked autoencoders","authors":"Cao","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Caron et al., 2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104.14294, 2021.","title":"Emerging properties in self-supervised vision transformers","authors":"Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen and He, 2021] Xinlei Chen and Kaiming He. Explor- ing simple siamese representation learning. In CVPR, 2021.","title":"Exploring simple siamese representation learning","authors":"Chen, He ;, Xinlei Chen, Kaiming He","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2020a] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020.","title":"Generative pretraining from pixels","authors":"Chen","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2020b] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In OpenAI blog, 2020.","title":"Generative pretraining from pixels","authors":"Chen","year":"2020","venue":"OpenAI blog","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Chen et al., 2021] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. ICCV, 2021.","title":"An empirical study of training self-supervised vision transformers","authors":"Chen","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chen et al., 2022a] Jun Chen, Ming Hu, Boyang Li, and Mohamed Elhoseiny. Efficient self-supervised vision pre- training with local masked reconstruction. arXiv preprint arXiv:2206.00790, 2022.","title":"Efficient self-supervised vision pretraining with local masked reconstruction","authors":"Chen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Chen et al., 2022b] Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Con- text autoencoder for self-supervised representation learn- ing. arXiv preprint arXiv:2202.03026, 2022.","title":"Context autoencoder for self-supervised representation learning","authors":"Chen","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. NAACL, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Dong et al., 2021] Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pre-training of vision transformers. arXiv preprint arXiv:2111.12710, 2021.","title":"Peco: Perceptual codebook for bert pre-training of vision transformers","authors":"Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, Nenghai Yu","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Dosovitskiy et al., 2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Trans- formers for image recognition at scale. In ICLR, 2021.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[El-Nouby et al., 2021] Alaaeldin El-Nouby, Gautier Izac- ard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre-training? arXiv preprint arXiv:2112.10740, 2021.","title":"Alaaeldin El-Nouby","authors":"El-Nouby","year":"2021","venue":"Are large-scale datasets necessary for self-supervised pre-training","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Fang et al., 2022a] Yuxin Fang, Li Dong, Hangbo Bao, Xinggang Wang, and Furu Wei. Corrupted image model- ing for self-supervised visual pre-training. arXiv preprint arXiv:2202.03382, 2022.","title":"Corrupted image modeling for self-supervised visual pre-training","authors":"Fang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Fang et al., 2022b] Yuxin Fang, Shusheng Yang, Shijie Wang, Yixiao Ge, Ying Shan, and Xinggang Wang. Unleashing vanilla vision transformer with masked im- age modeling for object detection. arXiv preprint arXiv:2204.02964, 2022.","title":"Unleashing vanilla vision transformer with masked image modeling for object detection","authors":"Fang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Geng et al., 2022] Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurams, Sergey Levine, and Pieter Abbeel. Multi- modal masked autoencoders learn transferable represen- tations. arXiv preprint arXiv:2205.14204, 2022.","title":"Multimodal masked autoencoders learn transferable representations","authors":"Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurams, Sergey Levine, Pieter Abbeel","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Girdhar et al., 2022] Rohit Girdhar, Alaaeldin El-Nouby, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Omnimae: Single model masked pretraining on images and videos. arXiv preprint arXiv:2206.08356, 2022.","title":"Omnimae: Single model masked pretraining on images and videos","authors":"Rohit Girdhar, Alaaeldin El-Nouby, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, Ishan Misra","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Gupta et al., 2022] Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Martín-Martín, and Li Fei-Fei. Maskvit: Masked visual pre-training for video prediction. arXiv preprint arXiv:2206.11894, 2022.","title":"Maskvit: Masked visual pre-training for video prediction","authors":"Gupta","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Sain- ing Xie, and Ross Girshick. Momentum contrast for unsu- pervised visual representation learning. In CVPR, 2020.","title":"Momentum contrast for unsupervised visual representation learning","authors":"Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[He et al., 2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In CVPR, 2022.","title":"Masked autoencoders are scalable vision learners","authors":"Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Huang et al., 2022] Lang Huang, Shan You, Mingkai Zheng, Fei Wang, Chen Qian, and Toshihiko Yamasaki. Green hierarchical vision transformer for masked image modeling. arXiv preprint arXiv:2205.13515, 2022.","title":"Green hierarchical vision transformer for masked image modeling","authors":"Lang Huang, Shan You, Mingkai Zheng, Fei Wang, Chen Qian, Toshihiko Yamasaki","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Jing et al., 2022] Li Jing, Jiachen Zhu, and Yann Le- Cun. Masked siamese convnets. arXiv preprint arXiv:2206.07700, 2022.","title":"Masked siamese convnets","authors":"Li Jing, Jiachen Zhu, Yann Le-Cun","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Khan et al., 2022] Salman Khan, Muzammal Naseer, et al. Transformers in vision: A survey. ACM computing surveys (CSUR), 2022.","title":"Transformers in vision: A survey","authors":"Salman Khan, Muzammal Naseer","year":"2022","venue":"ACM computing surveys (CSUR)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Larsson et al., 2016] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for au- tomatic colorization. In ECCV, 2016.","title":"Learning representations for automatic colorization","authors":"Larsson","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Li et al., 2021] Zhaowen Li, Zhiyang Chen, Fan Yang, Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, et al. Mst: Masked self-supervised transformer for visual representation. NeurIPS, 2021.","title":"Mst: Masked self-supervised transformer for visual representation","authors":"Zhaowen Li, Zhiyang Chen, Fan Yang, Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang","year":"2021","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Li et al., 2022a] Siyuan Li, Di Wu, Fang Wu, Zelin Zang, Kai Wang, Lei Shang, Baigui Sun, Hao Li, Stan Li, et al. Architecture-agnostic masked image modeling-from vit back to cnn. arXiv preprint arXiv:2205.13943, 2022.","title":"Architecture-agnostic masked image modeling-from vit back to cnn","authors":"Li","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Li et al., 2022b] Xiang Li, Wenhai Wang, Lingfeng Yang, and Jian Yang. Uniform masking: Enabling mae pre- training for pyramid-based vision transformers with local- ity. arXiv preprint arXiv:2205.10063, 2022.","title":"Uniform masking: Enabling mae pretraining for pyramid-based vision transformers with locality","authors":"Li","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Li et al., 2022c] Xiaotong Li, Yixiao Ge, Kun Yi, Zixuan Hu, Ying Shan, and Ling-Yu Duan. mc-beit: Multi-choice discretization for image bert pre-training. arXiv preprint arXiv:2203.15371, 2022.","title":"mc-beit: Multi-choice discretization for image bert pre-training","authors":"Li","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Liu et al., 2022] Jihao Liu, Xin Huang, Yu Liu, and Hong- sheng Li. Mixmim: Mixed and masked image modeling for efficient visual representation learning. arXiv preprint arXiv:2205.13137, 2022.","title":"Mixmim: Mixed and masked image modeling for efficient visual representation learning","authors":"Jihao Liu, Xin Huang, Yu Liu, Hongsheng Li","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Lu et al., 2022] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08916, 2022.","title":"Unified-io: A unified model for vision, language, and multi-modal tasks","authors":"Lu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Misra and Maaten, 2020] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant rep- resentations. In CVPR, 2020.","title":"Self-supervised learning of pretext-invariant representations","authors":"Maaten Misra, Ishan Misra, Laurens Van Der Maaten","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Pan et al., 2022] Jiachun Pan, Pan Zhou, and Shuicheng Yan. Towards understanding why mask-reconstruction pretraining helps in downstream tasks. arXiv preprint arXiv:2206.03826, 2022.","title":"Towards understanding why mask-reconstruction pretraining helps in downstream tasks","authors":"Pan","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Pathak et al., 2016] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Con- text encoders: Feature learning by inpainting. In CVPR, 2016.","title":"Context encoders: Feature learning by inpainting","authors":"Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, Alexei A Efros","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Tao et al., 2022] Chenxin Tao, Xizhou Zhu, Gao Huang, Yu Qiao, Xiaogang Wang, and Jifeng Dai. Siamese image modeling for self-supervised vision representation learn- ing. arXiv preprint arXiv:2206.01204, 2022.","title":"Siamese image modeling for self-supervised vision representation learning","authors":"Chenxin Tao, Xizhou Zhu, Gao Huang, Yu Qiao, Xiaogang Wang, Jifeng Dai","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Tian et al., 2022] Yunjie Tian, Lingxi Xie, Jiemin Fang, Mengnan Shi, Junran Peng, Xiaopeng Zhang, Jianbin Jiao, Qi Tian, and Qixiang Ye. Beyond masking: Demystify- ing token-based pre-training for vision transformers. arXiv preprint arXiv:2203.14313, 2022.","title":"Beyond masking: Demystifying token-based pre-training for vision transformers","authors":"Yunjie Tian, Lingxi Xie, Jiemin Fang, Mengnan Shi, Junran Peng, Xiaopeng Zhang, Jianbin Jiao, Qi Tian, Qixiang Ye","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Tong et al., 2022] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training. arXiv preprint arXiv:2203.12602, 2022.","title":"Videomae: Masked autoencoders are dataefficient learners for self-supervised video pre-training","authors":"Zhan Tong, Yibing Song, Jue Wang, Limin Wang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Vincent et al., 2008] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extract- ing and composing robust features with denoising autoen- coders. In ICML, 2008.","title":"Extracting and composing robust features with denoising autoencoders","authors":"Pascal Vincent, Hugo Larochelle, Yoshua Bengio, Pierre-Antoine Manzagol","year":"2008","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Wang et al., 2022a] Luya Wang, Feng Liang, Yang- guang Li, Wanli Ouyang, Honggang Zhang, and Jing Shao. Repre: Improving self-supervised vision trans- former with reconstructive pre-training. arXiv preprint arXiv:2201.06857, 2022.","title":"Repre: Improving self-supervised vision transformer with reconstructive pre-training","authors":"Wang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2022/200"},{"position":41,"raw":"[Wang et al., 2022b] Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan. Bevt: Bert pretraining of video transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14733-14743, 2022.","title":"Bevt: Bert pretraining of video transformers","authors":"Wang","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Wei et al., 2022a] Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre- training. In CVPR, 2022.","title":"Masked feature prediction for self-supervised visual pretraining","authors":"Wei","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Wei et al., 2022b] Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Contrastive learning rivals masked image model- ing in fine-tuning via feature distillation. arXiv preprint arXiv:2205.14141, 2022.","title":"Contrastive learning rivals masked image modeling in fine-tuning via feature distillation","authors":"Wei","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":44,"raw":"[Wu and Mo, 2022] Jiantao Wu and Shentong Mo. Object- wise masked autoencoders for fast pre-training. arXiv preprint arXiv:2205.14338, 2022.","title":"Jiantao Wu and Shentong Mo. Objectwise masked autoencoders for fast pre-training","authors":"Mo Wu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":45,"raw":"[Xiao et al., 2021] Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early con- volutions help transformers see better. NeurIPS, 2021.","title":"Early convolutions help transformers see better","authors":"Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, Ross Girshick","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":46,"raw":"[Xie et al., 2022a] Jiahao Xie, Wei Li, Xiaohang Zhan, Ziwei Liu, Yew Soon Ong, and Chen Change Loy. Masked frequency modeling for self-supervised visual pre-training. arXiv preprint arXiv:2206.07706, 2022.","title":"Masked frequency modeling for self-supervised visual pre-training","authors":"Xie","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":47,"raw":"[Xie et al., 2022b] Zhenda Xie, Zheng Zhang, Yue Cao, Yu- tong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image model- ing. In CVPR, 2022.","title":"Simmim: A simple framework for masked image modeling","authors":"Xie","year":"2022","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":48,"raw":"[Xie et al., 2022c] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Yixuan Wei, Qi Dai, and Han Hu. On data scaling in masked image modeling. arXiv preprint arXiv:2206.04664, 2022.","title":"On data scaling in masked image modeling","authors":"Xie","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":49,"raw":"[Yi et al., 2022] Kun Yi, Yixiao Ge, Xiaotong Li, Shusheng Yang, Dian Li, Jianping Wu, Ying Shan, and Xiaohu Qie. Masked image modeling with denoising contrast. arXiv preprint arXiv:2205.09616, 2022.","title":"Masked image modeling with denoising contrast","authors":"Kun Yi, Yixiao Ge, Xiaotong Li, Shusheng Yang, Dian Li, Jianping Wu, Ying Shan, Xiaohu Qie","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":50,"raw":"[Zhang et al., 2016] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In ECCV, 2016.","title":"Colorful image colorization","authors":"Richard Zhang, Phillip Isola, Alexei A Efros","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":51,"raw":"[Zhang et al., 2022a] Chaoning Zhang, Chenshuang Zhang, Junha Song, John Seon Keun Yi, Kang Zhang, and In So Kweon. A survey on masked autoencoder for self- supervised learning in vision and beyond. arXiv preprint arXiv:2208.00173, 2022.","title":"A survey on masked autoencoder for selfsupervised learning in vision and beyond","authors":"Zhang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":52,"raw":"[Zhang et al., 2022b] Xiaosong Zhang, Yunjie Tian, Wei Huang, Qixiang Ye, Qi Dai, Lingxi Xie, and Qi Tian. Hivit: Hierarchical vision transformer meets masked im- age modeling. arXiv preprint arXiv:2205.14949, 2022.","title":"Hivit: Hierarchical vision transformer meets masked image modeling","authors":"Zhang","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":53,"raw":"[Zhou et al., 2022] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. ICLR, 2022.","title":"ibot: Image bert pre-training with online tokenizer","authors":"Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, Tao Kong","year":"2022","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}