{"type":"ScholarlyArticle","title":"Domain Adaptation via Maximizing Surrogate Mutual Information","authors":[{"name":"Haiteng Zhao","slug":"haiteng-zhao","given":"Haiteng","family":"Zhao","affiliation":"Peking University"},{"name":"Chang Ma","slug":"chang-ma","given":"Chang","family":"Ma","affiliation":"Peking University"},{"name":"Qinyu Chen","slug":"qinyu-chen","given":"Qinyu","family":"Chen","affiliation":"Peking University"},{"name":"Zhi-Hong Deng","slug":"zhi-hong-deng","given":"Zhi-Hong","family":"Deng","affiliation":"Peking University"}],"abstract":"Unsupervised domain adaptation (UDA), which is an important topic in transfer learning, aims to predict unlabeled data from target domain with access to labeled data from the source domain. In this work, we propose a novel framework called SIDA (Surrogate Mutual Information Maximization Domain Adaptation) with strong theoretical guarantees. To be specific, SIDA implements adaptation by maximizing mutual information (MI) between features. In the framework, a surrogate joint distribution models the underlying joint distribution of the unlabeled target domain. Our theoretical analysis validates SIDA by bounding the expected risk on target domain with MI and surrogate distribution bias. Experiments show that our approach is comparable with state-of-the-art unsupervised adaptation methods on standard UDA tasks.","doi":"10.24963/ijcai.2022/237","url":"https://ijcai.org/proceedings/2022/237","pdf":"https://ijcai.org/proceedings/2022/0237.pdf","markdown":"https://ijcai.org/proceedings/2022/237.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/237","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":237,"pages":{"first":1700,"last":1706},"keywords":["Computer Vision: Transfer, low-shot, semi- and un- supervised learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987038","length_sec":77,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985181","length_sec":623,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Ben-David et al., 2007] Shai Ben-David, John Blitzer, Koby Crammer, Fernando Pereira, et al. Analysis of representations for domain adaptation. Advances in neural information processing systems, 2007.","title":"Analysis of representations for domain adaptation","authors":"Shai Ben-David, John Blitzer, Koby Crammer, Fernando Pereira","year":"2007","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen and Liu, 2020] Qingchao Chen and Yang Liu. Structure-aware feature fusion for unsupervised domain adaptation. In Proc. of AAAI, 2020.","title":"Structure-aware feature fusion for unsupervised domain adaptation","authors":"Chen, Liu, Qingchao Chen, Yang Liu","year":"2020","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2019] Chaoqi Chen, Weiping Xie, Wenbing Huang, Yu Rong, Xinghao Ding, Yue Huang, Tingyang Xu, and Junzhou Huang. Progressive feature alignment for unsupervised domain adaptation. In Proc. of CVPR, 2019.","title":"Progressive feature alignment for unsupervised domain adaptation","authors":"Chaoqi Chen, Weiping Xie, Wenbing Huang, Yu Rong, Xinghao Ding, Yue Huang, Tingyang Xu, Junzhou Huang","year":"2019","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2020a] Minghao Chen, Shuai Zhao, Haifeng Liu, and Deng Cai. Adversarial-learned loss for domain adaptation. In Proc. of AAAI, 2020.","title":"Adversarial-learned loss for domain adaptation","authors":"Chen","year":"2020","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2020b] Ting Chen, Simon Kornblith, Moham- mad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Inter- national conference on machine learning, 2020.","title":"A simple framework for contrastive learning of visual representations","authors":"Chen","year":"2020","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ganin et al., 2016] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Franc ¸ois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The jour- nal of machine learning research, 2016.","title":"Domain-adversarial training of neural networks","authors":"Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Mario Franc ¸ois Laviolette, Victor Marchand, Lempitsky","year":"2016","venue":"The journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hjelm et al., 2018] R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep rep- resentations by mutual information estimation and maxi- mization. In Proc. of ICLR, 2018.","title":"Learning deep representations by mutual information estimation and maximization","authors":"R, Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, Yoshua Bengio","year":"2018","venue":"Proc. of ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Khosla et al., 2020] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised con- trastive learning. In Proc. of NeurIPS, 2020.","title":"Supervised contrastive learning","authors":"Khosla","year":"2020","venue":"Proc. of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2020a] Mengxue Li, Yi-Ming Zhai, You-Wei Luo, Peng-Fei Ge, and Chuan-Xian Ren. Enhanced transport distance for unsupervised domain adaptation. In Proc. of CVPR, 2020.","title":"Enhanced transport distance for unsupervised domain adaptation","authors":"Li","year":"2020","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2020b] Rui Li, Qianfen Jiao, Wenming Cao, Hau- San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In Proc. of CVPR, 2020.","title":"Model adaptation: Unsupervised domain adaptation without source data","authors":"Li","year":"2020","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2020c] Shuang Li, Chi Liu, Qiuxia Lin, Binhui Xie, Zhengming Ding, Gao Huang, and Jian Tang. Do- main conditioned adaptation network. In Proc. of AAAI, 2020.","title":"Domain conditioned adaptation network","authors":"Li","year":"2020","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Long et al., 2015] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International confer- ence on machine learning, 2015.","title":"Learning transferable features with deep adaptation networks","authors":"Mingsheng Long, Yue Cao, Jianmin Wang, Michael Jordan","year":"2015","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Long et al., 2017a] Mingsheng Long, Zhangjie Cao, Jian- min Wang, and Michael I Jordan. Conditional adversar- ial domain adaptation. arXiv preprint arXiv:1705.10667, 2017.","title":"Conditional adversarial domain adaptation","authors":"Long","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Long et al., 2017b] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Deep transfer learning with joint adaptation networks. In International conference on machine learning, 2017.","title":"Deep transfer learning with joint adaptation networks","authors":"Long","year":"2017","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Long et al., 2018] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Conditional adversarial do- main adaptation. In NeurIPS, 2018.","title":"Conditional adversarial domain adaptation","authors":"Mingsheng Long, Zhangjie Cao, Jianmin Wang, Michael I Jordan","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Luo et al., 2020] You-Wei Luo, Chuan-Xian Ren, Pengfei Ge, Ke-Kun Huang, and Yu-Feng Yu. Unsupervised do- main adaptation via discriminative manifold embedding and alignment. In Proc. of AAAI, 2020.","title":"Unsupervised domain adaptation via discriminative manifold embedding and alignment","authors":"Luo","year":"2020","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nguyen et al., 2010] XuanLong Nguyen, Martin J Wain- wright, and Michael I Jordan. Estimating divergence func- tionals and the likelihood ratio by convex risk minimiza- tion. IEEE Transactions on Information Theory, 2010.","title":"Estimating divergence functionals and the likelihood ratio by convex risk minimization","authors":"Nguyen","year":"2010","venue":"IEEE Transactions on Information Theory","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Park et al., 2020] Changhwa Park, Jonghyun Lee, Jaeyoon Yoo, Minhoe Hur, and Sungroh Yoon. Joint contrastive learning for unsupervised domain adaptation. arXiv preprint arXiv:2006.10297, 2020.","title":"Joint contrastive learning for unsupervised domain adaptation","authors":"Park","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Peng et al., 2017] Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924, 2017.","title":"Visda: The visual domain adaptation challenge","authors":"Peng","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Poole et al., 2019] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In Proc. of ICML, 2019.","title":"On variational bounds of mutual information","authors":"Ben Poole, Sherjil Ozair, Aaron Van Den, Alex Oord, George Alemi, Tucker","year":"2019","venue":"Proc. of ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Redko et al., 2020] Ievgen Redko, Emilie Morvant, Amaury Habrard, Marc Sebban, and Younès Bennani. A survey on domain adaptation theory: learning bounds and theoretical guarantees. arXiv e-prints, 2020.","title":"A survey on domain adaptation theory: learning bounds and theoretical guarantees","authors":"Redko","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Saenko et al., 2010] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference on computer vision, 2010.","title":"Adapting visual category models to new domains","authors":"Saenko","year":"2010","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tang and Jia, 2020] Hui Tang and Kui Jia. Discriminative adversarial domain adaptation. In Proc. of AAAI, 2020.","title":"Discriminative adversarial domain adaptation","authors":"Hui Tang, Kui Jia","year":"2020","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Thota and Leontidis, 2021] Mamatha Thota and Georgios Leontidis. Contrastive domain adaptation. In Proc. of CVPR, 2021.","title":"Contrastive domain adaptation","authors":"Mamatha Thota, Georgios Leontidis","year":"2021","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Venkateswara et al., 2017] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Pan- chanathan. Deep hashing network for unsupervised do- main adaptation. In Proc. of CVPR, 2017.","title":"Deep hashing network for unsupervised domain adaptation","authors":"Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, Sethuraman Panchanathan","year":"2017","venue":"Proc. of CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xie et al., 2018] Shaoan Xie, Zibin Zheng, Liang Chen, and Chuan Chen. Learning semantic representations for unsu- pervised domain adaptation. In International conference on machine learning, 2018.","title":"Learning semantic representations for unsupervised domain adaptation","authors":"Shaoan Xie, Zibin Zheng, Liang Chen, Chuan Chen","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xu et al., 2019] Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin. Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adapta- tion. In Proc. of ICCV, 2019.","title":"Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation","authors":"Xu","year":"2019","venue":"Proc. of ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}