{"type":"ScholarlyArticle","title":"RAIN: RegulArization on Input and Network for Black-Box Domain Adaptation","authors":[{"name":"Qucheng Peng","slug":"qucheng-peng","given":"Qucheng","family":"Peng","affiliation":"Center for Research in Computer Vision , University of Central Florida"},{"name":"Zhengming Ding","slug":"zhengming-ding","given":"Zhengming","family":"Ding","affiliation":"Department of Computer Science , Tulane University"},{"name":"Lingjuan Lyu","slug":"lingjuan-lyu","given":"Lingjuan","family":"Lyu","affiliation":"Sony AI"},{"name":"Lichao Sun","slug":"lichao-sun","given":"Lichao","family":"Sun","affiliation":"Lehigh University"},{"name":"Chen Chen","slug":"chen-chen","given":"Chen","family":"Chen","affiliation":"Center for Research in Computer Vision , University of Central Florida"}],"abstract":"Source-Free domain adaptation transits the source-trained model towards target domain without exposing the source data, trying to dispel these concerns about data privacy and security. However, this paradigm is still at risk of data leakage due to adversarial attacks on the source model. Hence, the Black-Box setting only allows to use the outputs of source model, but still suffers from overfitting on the source domain more severely due to source model's unseen weights. In this paper, we propose a novel approach named RAIN (RegulArization on Input and Network) for Black-Box domain adaptation from both input-level and network-level regularization. For the input-level, we design a new data augmentation technique as Phase MixUp, which highlights task-relevant objects in the interpolations, thus enhancing input-level regularization and class consistency for target models. For network-level, we develop a Subnetwork Distillation mechanism to transfer knowledge from the target subnetwork to the full target network via knowledge distillation, which thus alleviates overfitting on the source domain by learning diverse target representations. Extensive experiments show that our method achieves state-of-the-art performance on several cross-domain benchmarks under both single- and multi-source black-box domain adaptation.","doi":"10.24963/ijcai.2023/458","url":"https://ijcai.org/proceedings/2023/458","pdf":"https://ijcai.org/proceedings/2023/0458.pdf","markdown":"https://ijcai.org/proceedings/2023/458.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/458","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":458,"pages":{"first":4118,"last":4126},"keywords":["Machine Learning: ML: Multi-task and transfer learning","Computer Vision: CV: Transfer, low-shot, semi- and un- supervised learning"],"videos":[],"references":[{"position":1,"raw":"[Ahmed et al., 2021] Sk Miraj Ahmed, Dripta S Raychaud- huri, Sujoy Paul, Samet Oymak, and Amit K Roy- Chowdhury. Unsupervised multi-source domain adapta- tion without access to source data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10103-10112, 2021.","title":"Unsupervised multi-source domain adaptation without access to source data","authors":"Ahmed","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2022] Dian Chen, Dequan Wang, Trevor Dar- rell, and Sayna Ebrahimi. Contrastive test-time adaptation. arXiv preprint arXiv:2204.10377, 2022.","title":"Contrastive test-time adaptation","authors":"Dian Chen, Dequan Wang, Trevor Darrell, Sayna Ebrahimi","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deng et al., 2021] Wanxia Deng, Qing Liao, Lingjun Zhao, Deke Guo, Gangyao Kuang, Dewen Hu, and Li Liu. Joint clustering and discriminative feature alignment for unsu- pervised domain adaptation. IEEE Transactions on Image Processing, pages 7842-7855, 2021.","title":"Joint clustering and discriminative feature alignment for unsupervised domain adaptation","authors":"Wanxia Deng, Qing Liao, Lingjun Zhao, Deke Guo, Gangyao Kuang, Dewen Hu, Li Liu","year":"2021","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Min- derer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.","title":"An image is worth 16x16 words: Transformers for image recognition at scale","authors":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ganin and Lempitsky, 2015] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backprop- agation. In International Conference on Machine Learn- ing, pages 1180-1189, 2015.","title":"Unsupervised domain adaptation by backpropagation","authors":"Yaroslav Ganin, Victor Lempitsky","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. Advances in Neural Information Processing Systems, 27, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. STAT, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":"STAT","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hou and Zheng, 2021] Yunzhong Hou and Liang Zheng. Visualizing adapted knowledge in domain transfer. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13824-13833, 2021.","title":"Visualizing adapted knowledge in domain transfer","authors":"Zheng ; ] Yunzhong Hou, Liang Hou, Zheng","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kurmi et al., 2021] Vinod K Kurmi, Venkatesh K Subrama- nian, Vinay P Namboodiri, , and . Domain impression: A source data free domain adaptation method. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 615-625, 2021.","title":"Domain impression: A source data free domain adaptation method","authors":"Vinod K Kurmi, Venkatesh K Subramanian, Vinay P Namboodiri","year":"2021","venue":"Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2020] Rui Li, Qianfen Jiao, Wenming Cao, Hau- San Wong, and Si Wu. Model adaptation: Unsuper- vised domain adaptation without source data. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9638-9647, 2020.","title":"Model adaptation: Unsupervised domain adaptation without source data","authors":"Rui Li, Qianfen Jiao, Wenming Cao, Hau-San, Si Wong, Wu","year":"2020","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liang et al., 2020] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pages 6028-6039. PMLR, 2020.","title":"Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation","authors":"Jian Liang, Dapeng Hu, Jiashi Feng","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liang et al., 2022] Jian Liang, Dapeng Hu, Ran He, and Ji- ashi Feng. Dine: Domain adaptation from single and mul- tiple black-box predictors. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8003-8013, June 2022.","title":"Dine: Domain adaptation from single and multiple black-box predictors","authors":"Jian Liang, Dapeng Hu, Ran He, Jiashi Feng","year":"2022","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2021] Quande Liu, Cheng Chen, Jing Qin, Qi Dou, and Pheng-Ann Heng. Feddg: Federated do- main generalization on medical image segmentation via episodic learning in continuous frequency space. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1013-1023, 2021.","title":"Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space","authors":"Quande Liu, Cheng Chen, Jing Qin, Qi Dou, Pheng-Ann Heng","year":"2021","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Long et al., 2015] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International Confer- ence on Machine Learning, pages 97-105, 2015.","title":"Learning transferable features with deep adaptation networks","authors":"Mingsheng Long, Yue Cao, Jianmin Wang, Michael Jordan","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Long et al., 2017] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Deep transfer learning with joint adaptation networks. In International conference on machine learning, pages 2208-2217. PMLR, 2017.","title":"Deep transfer learning with joint adaptation networks","authors":"Mingsheng Long, Han Zhu, Jianmin Wang, Michael I Jordan","year":"2017","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Muller et al., 2019] Rafael Muller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Ad- vances in Neural Information Processing Systems, 2019.","title":"When does label smoothing help? Advances in Neural Information Processing Systems","authors":"Muller","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Paszke et al., 2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in Neural Information Processing Systems, 2019.","title":"Pytorch: An imperative style, highperformance deep learning library","authors":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Peng et al., 2017] Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924, 2017.","title":"Visda: The visual domain adaptation challenge","authors":"Peng","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ruder, 2016] Sebastian Ruder. An overview of gradi- ent descent optimization algorithms. arXiv preprint arXiv:1609.04747, 2016.","title":"An overview of gradient descent optimization algorithms","authors":"Sebastian Ruder","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Saenko et al., 2010] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European Conference on Computer Vi- sion, pages 213-226, 2010.","title":"Adapting visual category models to new domains","authors":"Saenko","year":"2010","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Saito et al., 2018] Kuniaki Saito, Kohei Watanabe, Yoshi- taka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3723-3732, 2018.","title":"Maximum classifier discrepancy for unsupervised domain adaptation","authors":"Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, Tatsuya Harada","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shorten and Khoshgoftaar, 2019] Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmen- tation for deep learning. Journal of Big Data, 6(1):1-48, 2019.","title":"A survey on image data augmentation for deep learning","authors":"Connor Shorten, M Taghi, Khoshgoftaar","year":"2019","venue":"Journal of Big Data","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tang et al., 2020] Hui Tang, Ke Chen, and Kui Jia. Un- supervised domain adaptation via structurally regularized deep clustering. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 8725-8735, 2020.","title":"Unsupervised domain adaptation via structurally regularized deep clustering","authors":"Hui Tang, Ke Chen, Kui Jia","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tzeng et al., 2014] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confu- sion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474, 2014.","title":"Deep domain confusion: Maximizing for domain invariance","authors":"Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, Trevor Darrell","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Van der Maaten and Hinton, 2008] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.","title":"Visualizing data using t-sne","authors":"Van Der Maaten, Laurens Hinton, Geoffrey Van Der Maaten, Hinton","year":"2008","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Venkateswara et al., 2017] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Pan- chanathan. Deep hashing network for unsupervised do- main adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5018- 5027, 2017.","title":"Deep hashing network for unsupervised domain adaptation","authors":"Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, Sethuraman Panchanathan","year":"2017","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang and Deng, 2018] Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocomput- ing, 312:135-153, 2018.","title":"Deep visual domain adaptation: A survey","authors":"Mei Wang, Weihong Deng","year":"2018","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang et al., 2018] Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959, 2018.","title":"Dataset distillation","authors":"Wang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wu et al., 2020] Yuan Wu, Diana Inkpen, and Ahmed El- Roby. Dual mixup regularized learning for adversarial do- main adaptation. In European Conference on Computer Vision, pages 540-555. Springer, 2020.","title":"Dual mixup regularized learning for adversarial domain adaptation","authors":"Yuan Wu, Diana Inkpen, Ahmed El-Roby","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Xia et al., 2021] Haifeng Xia, Handong Zhao, and Zheng- ming Ding. Adaptive adversarial network for source-free domain adaptation. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 9010- 9019, 2021.","title":"Adaptive adversarial network for source-free domain adaptation","authors":"Haifeng Xia, Handong Zhao, Zhengming Ding","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yang and Soatto, 2020] Yanchao Yang and Stefano Soatto. Fda: Fourier domain adaptation for semantic segmenta- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4085- 4095, 2020.","title":"Fda: Fourier domain adaptation for semantic segmentation","authors":"Yang, Soatto ;, Yanchao Yang, Stefano Soatto","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yin et al., 2020] Hongxu Yin, Pavlo Molchanov, Jose M Al- varez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowl- edge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8715-8724, 2020.","title":"Dreaming to distill: Data-free knowledge transfer via deepinversion","authors":"Pavlo Hongxu Yin, Jose M Molchanov, Zhizhong Alvarez, Arun Li, Derek Mallya, Hoiem, K Niraj, Jan Jha, Kautz","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[YM. et al., 2020] Asano YM., Rupprecht C., and Vedaldi A. Self-labelling via simultaneous clustering and represen- tation learning. In International Conference on Learning Representations, 2020.","title":"Self-labelling via simultaneous clustering and representation learning","authors":"Y M Asano, C Rupprecht, A Vedaldi","year":"2020","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Yun et al., 2019] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023- 6032, 2019.","title":"Cutmix: Regularization strategy to train strong classifiers with localizable features","authors":"Yun","year":"2019","venue":"Proceedings of the IEEE/CVF international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang et al., 2018] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.","title":"mixup: Beyond empirical risk minimization","authors":"Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, David Lopez-Paz","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhang et al., 2021] Haojian Zhang, Yabin Zhang, Kui Jia, and Lei Zhang. Unsupervised domain adaptation of black- box source models. arXiv preprint arXiv:2101.02839, 2021.","title":"Unsupervised domain adaptation of blackbox source models","authors":"Zhang","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}