{"type":"ScholarlyArticle","title":"RePre: Improving Self-Supervised Vision Transformer with Reconstructive Pre-training","authors":[{"name":"Luya Wang","slug":"luya-wang","given":"Luya","family":"Wang","affiliation":"Beijing University of Posts and Telecommunications"},{"name":"Feng Liang","slug":"feng-liang","given":"Feng","family":"Liang","affiliation":"University of Texas at Austin"},{"name":"Yangguang Li","slug":"yangguang-li","given":"Yangguang","family":"Li","affiliation":"SenseTime Group Limited"},{"name":"Honggang Zhang","slug":"honggang-zhang","given":"Honggang","family":"Zhang","affiliation":"Beijing University of Posts and Telecommunications"},{"name":"Wanli Ouyang","slug":"wanli-ouyang","given":"Wanli","family":"Ouyang","affiliation":"The University of Sydney"},{"name":"Jing Shao","slug":"jing-shao","given":"Jing","family":"Shao","affiliation":"SenseTime Group Limited"}],"abstract":"Recently, self-supervised vision transformers have attracted unprecedented attention for their impressive representation learning ability. \r\nHowever, the dominant method, contrastive learning, mainly relies on an instance discrimination pretext task, which learns a global understanding of the image. \r\nThis paper incorporates local feature learning into self-supervised vision transformers via Reconstructive Pre-training (RePre).\r\nOur RePre extends contrastive frameworks by adding a branch for reconstructing raw image pixels in parallel with the existing contrastive objective. \r\nRePre equips with a lightweight convolution-based decoder that fuses the multi-hierarchy features from the transformer encoder. \r\nThe multi-hierarchy features provide rich supervisions from low to high semantic information, crucial for our RePre.\r\nOur RePre brings decent improvements on various contrastive frameworks with different vision transformer architectures. \r\nTransfer performance in downstream tasks outperforms supervised pre-training and state-of-the-art (SOTA) self-supervised counterparts.","doi":"10.24963/ijcai.2022/200","url":"https://ijcai.org/proceedings/2022/200","pdf":"https://ijcai.org/proceedings/2022/0200.pdf","markdown":"https://ijcai.org/proceedings/2022/200.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/200","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":200,"pages":{"first":1437,"last":1443},"keywords":["Computer Vision: Representation Learning","Computer Vision: Transfer, low-shot, semi- and un- supervised learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986985","length_sec":82,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985128","length_sec":654,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Ahn and Kwak, 2018] Jiwoon Ahn and Suha Kwak. Learn- ing pixel-level semantic affinity with image-level super- vision for weakly supervised semantic segmentation. In Proc. of CVPR, pages 4981-4990, 2018.","title":"Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation","authors":"Ahn, Kwak, Jiwoon Ahn, Suha Kwak","year":"2018","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bao et al., 2021] Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021.","title":"Beit: Bert pre-training of image transformers","authors":"Bao","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Caron et al., 2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vi- sion transformers. arXiv preprint:2104.14294, 2021.","title":"Emerging properties in self-supervised vision transformers","authors":"Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen and He, 2021] Xinlei Chen and Kaiming He. Explor- ing simple siamese representation learning. In Proc. of CVPR, pages 15750-15758, 2021.","title":"Exploring simple siamese representation learning","authors":"Chen, He ;, Xinlei Chen, Kaiming He","year":"2021","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2020a] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Proc. of ICML, pages 1691-1703. PMLR, 2020.","title":"Generative pretraining from pixels","authors":"Chen","year":"2020","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2020b] Ting Chen, Simon Kornblith, Moham- mad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, pages 1597-1607. PMLR, 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Chen","year":"2020","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2020c] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint:2003.04297, 2020.","title":"Improved baselines with momentum contrastive learning","authors":"Chen","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chen et al., 2021] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.02057, 2021.","title":"An empirical study of training self-supervised vision transformers","authors":"Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Grill et al., 2020] Jean-Bastien Grill, Florian Strub, Flo- rent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Pires, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733, 2020.","title":"Bootstrap your own latent: A new approach to self-supervised learning","authors":"Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Elena Pierre H Richemond, Carl Buchatskaya, Doersch, Pires","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proc. of ICCV, pages 2961-2969, 2017.","title":"Kaiming He, Georgia Gkioxari","authors":"He","year":"2017","venue":"Proc. of ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[He et al., 2021] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377, 2021.","title":"Masked autoencoders are scalable vision learners","authors":"Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Hsu et al., 2021] Wei-Ning Hsu, Benjamin Bolte, Yao- Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhut- dinov, and Abdelrahman Mohamed. Hubert: Self- supervised speech representation learning by masked pre- diction of hidden units. arXiv preprint:2106.07447, 2021.","title":"Hubert: Self-supervised speech representation learning by masked prediction of hidden units","authors":"Wei-Ning Hsu, Benjamin Bolte, Hubert Yao-Hung, Kushal Tsai, Ruslan Lakhotia, Abdelrahman Salakhutdinov, Mohamed","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kendall et al., 2018] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proc. of CVPR, pages 7482-7491, 2018.","title":"Multi-task learning using uncertainty to weigh losses for scene geometry and semantics","authors":"Alex Kendall, Yarin Gal, Roberto Cipolla","year":"2018","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030, 2021.","title":"Swin transformer: Hierarchical vision transformer using shifted windows","authors":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Radford et al., 2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.","title":"Language models are unsupervised multitask learners","authors":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever","year":"2019","venue":"OpenAI blog","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ramesh et al., 2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092, 2021.","title":"Zero-shot text-to-image generation","authors":"Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, Ilya Sutskever","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234-241. Springer, 2015.","title":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"MICCAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sun et al., 2017] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effec- tiveness of data in deep learning era. In Proc. of ICCV, pages 843-852, 2017.","title":"Revisiting unreasonable effectiveness of data in deep learning era","authors":"Chen Sun, Abhinav Shrivastava, Saurabh Singh, Abhinav Gupta","year":"2017","venue":"Proc. of ICCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Van der Maaten and Hinton, 2008] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9(11), 2008.","title":"Visualizing data using t-sne","authors":"Van Der Maaten, Laurens Hinton, Geoffrey Van Der Maaten, Hinton","year":"2008","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xie et al., 2021a] Zhenda Xie, Yutong Lin, Zhuliang Yao, Zheng Zhang, Qi Dai, Yue Cao, and Han Hu. Self- supervised learning with swin transformers. arXiv preprint arXiv:2105.04553, 2021.","title":"Selfsupervised learning with swin transformers","authors":"Xie","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xie et al., 2021b] Zhenda Xie, Zheng Zhang, Yue Cao, Yu- tong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image model- ing. arXiv preprint arXiv:2111.09886, 2021.","title":"Simmim: A simple framework for masked image modeling","authors":"Xie","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zheng et al., 2021] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethink- ing semantic segmentation from a sequence-to-sequence perspective with transformers. In Proc. of CVPR, pages 6881-6890, 2021.","title":"Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers","authors":"Zheng","year":"2021","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhou et al., 2021] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832, 2021.","title":"ibot: Image bert pre-training with online tokenizer","authors":"Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, Tao Kong","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}