{"type":"ScholarlyArticle","title":"Decentralized Unsupervised Learning of Visual Representations","authors":[{"name":"Yawen Wu","slug":"yawen-wu","given":"Yawen","family":"Wu","affiliation":"University of Pittsburgh"},{"name":"Zhepeng Wang","slug":"zhepeng-wang","given":"Zhepeng","family":"Wang","affiliation":"George Mason University"},{"name":"Dewen Zeng","slug":"dewen-zeng","given":"Dewen","family":"Zeng","affiliation":"University of Notre Dame"},{"name":"Meng Li","slug":"meng-li","given":"Meng","family":"Li","affiliation":"Facebook"},{"name":"Yiyu Shi","slug":"yiyu-shi","given":"Yiyu","family":"Shi","affiliation":"University of Notre Dame"},{"name":"Jingtong Hu","slug":"jingtong-hu","given":"Jingtong","family":"Hu","affiliation":"University of Pittsburgh"}],"abstract":"Collaborative learning enables distributed clients to learn a shared model for prediction while keeping the training data local on each client. However, existing collaborative learning methods require fully-labeled data for training, which is inconvenient or sometimes infeasible to obtain due to the high labeling cost and the requirement of expertise. The lack of labels makes collaborative learning impractical in many realistic settings. Self-supervised learning can address this challenge by learning from unlabeled data. Contrastive learning (CL), a self-supervised learning approach, can effectively learn visual representations from unlabeled image data. However, the distributed data collected on clients are usually not independent and identically distributed (non-IID) among clients, and each client may only have few classes of data, which degrades the performance of CL and learned representations. To tackle this problem, we propose a collaborative contrastive learning framework consisting of two approaches: feature fusion and neighborhood matching, by which a unified feature space among clients is learned for better data representations. Feature fusion provides remote features as accurate contrastive information to each client for better local learning. Neighborhood matching further aligns each client’s local features to the remote features such that well-clustered features among clients can be learned. Extensive experiments show the effectiveness of the proposed framework. It outperforms other methods by 11% on IID data and matches the performance of centralized learning.","doi":"10.24963/ijcai.2022/323","url":"https://ijcai.org/proceedings/2022/323","pdf":"https://ijcai.org/proceedings/2022/0323.pdf","markdown":"https://ijcai.org/proceedings/2022/323.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/323","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Data Mining","track":"Main Track","number":323,"pages":{"first":2326,"last":2333},"keywords":["Machine Learning: Self-supervised Learning","Computer Vision: Representation Learning","Data Mining: Federated Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986931","length_sec":89,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985074","length_sec":358,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Caron et al., 2018] Mathilde Caron, Piotr Bojanowski, Ar- mand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pages 132-149, 2018.","title":"Deep clustering for unsupervised learning of visual features","authors":"Mathilde Caron, Piotr Bojanowski, Armand Joulin, Matthijs Douze","year":"2018","venue":"Proceedings of the European Conference on Computer Vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Caron et al., 2020] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020.","title":"Unsupervised learning of visual features by contrasting cluster assignments","authors":"Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, Armand Joulin","year":"2020","venue":"Proceedings of Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2020a] Ting Chen, Simon Kornblith, Moham- mad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Pro- ceedings of the 37th International Conference on Machine Learning. PMLR, 13-18 Jul 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Chen","year":"2020","venue":"Proceedings of the 37th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2020b] Yiqiang Chen, Xin Qin, Jindong Wang, Chaohui Yu, and Wen Gao. Fedhealth: A federated transfer learning framework for wearable healthcare. IEEE Intelli- gent Systems, 2020.","title":"Fedhealth: A federated transfer learning framework for wearable healthcare","authors":"Chen","year":"2020","venue":"IEEE Intelligent Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gidaris et al., 2018] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018.","title":"Unsupervised representation learning by predicting image rotations","authors":"Spyros Gidaris, Praveer Singh, Nikos Komodakis","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[He et al., 2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsuper- vised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020.","title":"Momentum contrast for unsupervised visual representation learning","authors":"Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jef- frey Dean. Distilling the knowledge in a neural network. NIPS Deep Learning and Representation Learning Work- shop, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeffrey Dean","year":"2015","venue":"NIPS Deep Learning and Representation Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hsu et al., 2020] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Federated visual classification with real- world data distribution. In ECCV 2020: 16th European Conference, pages 76-92. Springer, 2020.","title":"Federated visual classification with realworld data distribution","authors":"Hsu","year":"2020","venue":"ECCV 2020: 16th European Conference","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2020] Yangsibo Huang, Zhao Song, Kai Li, and Sanjeev Arora. Instahide: Instance-hiding schemes for private distributed learning. In International Conference on Machine Learning, pages 4507-4518. PMLR, 2020.","title":"Instahide: Instance-hiding schemes for private distributed learning","authors":"Yangsibo Huang, Zhao Song, Kai Li, Sanjeev Arora","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jeong et al., 2018] Eunjeong Jeong, Seungeun Oh, Hyesung Kim, Jihong Park, Mehdi Bennis, and Seong-Lyun Kim. Communication-efficient on-device machine learning: Fed- erated distillation and augmentation under non-iid private data. arXiv preprint arXiv:1811.11479, 2018.","title":"Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data","authors":"Jeong","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kairouz et al., 2019] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Gra- ham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977, 2019.","title":"Advances and open problems in federated learning","authors":"Peter Kairouz, Brendan Mcmahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Nitin Arjun, Keith Bhagoji, Zachary Bonawitz, Graham Charles, Rachel Cormode, Cummings","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kalantidis et al., 2020] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Lar- lus. Hard negative mixing for contrastive learning. In Neural Information Processing Systems (NeurIPS), 2020.","title":"Hard negative mixing for contrastive learning","authors":"Yannis Kalantidis, Bulent Mert, Noe Sariyildiz, Philippe Pion, Diane Weinzaepfel, Larlus","year":"2020","venue":"Neural Information Processing Systems (NeurIPS)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krizhevsky et al., 2009] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky, Geoffrey Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2020] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Fed- erated optimization in heterogeneous networks. Proceed- ings of Machine Learning and Systems, 2:429-450, 2020.","title":"Federated optimization in heterogeneous networks","authors":"Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, Virginia Smith","year":"2020","venue":"Proceedings of Machine Learning and Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2019] Boyi Liu, Lujia Wang, and Ming Liu. Life- long federated reinforcement learning: a learning architec- ture for navigation in cloud robotic systems. IEEE Robotics and Automation Letters, 4(4):4555-4562, 2019.","title":"Lifelong federated reinforcement learning: a learning architecture for navigation in cloud robotic systems","authors":"Boyi Liu, Lujia Wang, Ming Liu","year":"2019","venue":"IEEE Robotics and Automation Letters","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Luo et al., 2019] Jiahuan Luo, Xueyang Wu, Yun Luo, Anbu Huang, Yunfeng Huang, Yang Liu, and Qiang Yang. Real- world image datasets for federated learning. arXiv preprint arXiv:1910.11089, 2019.","title":"Realworld image datasets for federated learning","authors":"Luo","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273-1282. PMLR, 2017.","title":"Communication-efficient learning of deep networks from decentralized data","authors":"Brendan Mcmahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Aguera Y Arcas","year":"2017","venue":"Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nguyen et al., 2020] Hung T Nguyen, Vikash Sehwag, Seyyedali Hosseinalipour, Christopher G Brinton, Mung Chiang, and H Vincent Poor. Fast-convergent federated learning. IEEE Journal on Selected Areas in Communica- tions, 39(1):201-218, 2020.","title":"Fast-convergent federated learning","authors":"Vikash Hung T Nguyen, Seyyedali Sehwag, Hosseinalipour, Mung Christopher G Brinton, H Chiang, Vincent Poor","year":"2020","venue":"IEEE Journal on Selected Areas in Communications","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Reisizadeh et al., 2020] Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization. In International Conference on Artificial Intelligence and Statistics, pages 2021-2031, 2020.","title":"Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization","authors":"Reisizadeh","year":"2020","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wu et al., 2018] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 3733-3742, 2018.","title":"Unsupervised feature learning via nonparametric instance discrimination","authors":"Zhirong Wu, Yuanjun Xiong, Stella X Yu, Dahua Lin","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wu et al., 2021a] Yawen Wu, Zhepeng Wang, Dewen Zeng, Yiyu Shi, and Jingtong Hu. Enabling on-device self- supervised contrastive learning with selective data contrast. In 2021 58th ACM/IEEE Design Automation Conference (DAC), pages 655-660. IEEE, 2021.","title":"Enabling on-device selfsupervised contrastive learning with selective data contrast","authors":"Wu","year":"2021","venue":"58th ACM/IEEE Design Automation Conference (DAC)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wu et al., 2021b] Yawen Wu, Dewen Zeng, Zhepeng Wang, Yiyu Shi, and Jingtong Hu. Federated contrastive learn- ing for volumetric medical image segmentation. In Inter- national Conference on Medical Image Computing and Computer-Assisted Intervention, pages 367-377. Springer, 2021.","title":"Federated contrastive learning for volumetric medical image segmentation","authors":"Wu","year":"2021","venue":"International Conference on Medical Image Computing and Computer-Assisted Intervention","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xiao et al., 2017] Han Xiao, Kashif Rasul, and Roland Voll- graf. Fashion-mnist: a novel image dataset for bench- marking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.","title":"Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms","authors":"Han Xiao, Kashif Rasul, Roland Vollgraf","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2020] Fengda Zhang, Kun Kuang, Zhaoyang You, Tao Shen, Jun Xiao, Yin Zhang, Chao Wu, Yueting Zhuang, and Xiaolin Li. Federated unsupervised represen- tation learning. arXiv preprint arXiv:2010.08982, 2020.","title":"Federated unsupervised representation learning","authors":"Zhang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhao et al., 2018] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Fed- erated learning with non-iid data. arXiv preprint arXiv:1806.00582, 2018.","title":"Federated learning with non-iid data","authors":"Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, Vikas Chandra","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}