{"type":"ScholarlyArticle","title":"Towards Accurate and Robust Domain Adaptation under Noisy Environments","authors":[{"name":"Zhongyi Han","slug":"zhongyi-han","given":"Zhongyi","family":"Han","affiliation":"School of Software , Shandong University , Jinan 250101 . China"},{"name":"Xian-Jin Gui","slug":"xian-jin-gui","given":"Xian-Jin","family":"Gui","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , Nanjing 210023 , China"},{"name":"Chaoran Cui","slug":"chaoran-cui","given":"Chaoran","family":"Cui","affiliation":"School of Computer Science and Technology , Shandong University of Finance and Economics"},{"name":"Yilong Yin","slug":"yilong-yin","given":"Yilong","family":"Yin","affiliation":"School of Software , Shandong University , Jinan 250101 . China"}],"abstract":"In non-stationary environments, learning machines usually confront the domain adaptation scenario where the data distribution does change over time. Previous domain adaptation works have achieved great success in theory and practice. However, they always lose robustness in noisy environments where the labels and features of examples from the source domain become corrupted. In this paper, we report our attempt towards achieving accurate noise-robust domain adaptation. We first give a theoretical analysis that reveals how harmful noises influence unsupervised domain adaptation. To eliminate the effect of label noise, we propose an offline curriculum learning for minimizing a newly-defined empirical source risk. To reduce the impact of feature noise, we propose a proxy distribution based margin discrepancy. We seamlessly transform our methods into an adversarial network that performs efficient joint optimization for them, successfully mitigating the negative influence from both data corruption and distribution shift. A series of empirical studies show that our algorithm remarkably outperforms state of the art, over 10% accuracy improvements in some domain adaptation tasks under noisy environments.","doi":"10.24963/ijcai.2020/314","url":"https://ijcai.org/proceedings/2020/314","pdf":"https://ijcai.org/proceedings/2020/0314.pdf","markdown":"https://ijcai.org/proceedings/2020/314.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/314","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":314,"pages":{"first":2269,"last":2276},"keywords":["Machine Learning: Transfer, Adaptation, Multi-task Learning","Machine Learning: Learning Theory","Machine Learning: Adversarial Machine Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25071/file_25071.m3u8","length_sec":324,"provider":"hls"}],"references":[{"position":1,"raw":"[Ben-David et al., 2007] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representa- tions for domain adaptation. In Advances in neural infor- mation processing systems, pages 137-144, 2007.","title":"Analysis of representations for domain adaptation","authors":"Shai Ben-David, John Blitzer, Koby Crammer, Fernando Pereira","year":"2007","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ben-David et al., 2010a] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jen- nifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1-2):151-175, 2010.","title":"A theory of learning from different domains","authors":"Ben-David","year":"2010","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ben-David et al., 2010b] Shai Ben-David, Tyler Lu, Teresa Luu, and Dávid Pál. Impossibility theorems for domain adaptation. In International Conference on Artificial Intel- ligence and Statistics, pages 129-136, 2010.","title":"Impossibility theorems for domain adaptation","authors":"Ben-David","year":"2010","venue":"International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cortes et al., 2019] Corinna Cortes, Mehryar Mohri, and Andrés Munoz Medina. Adaptation based on generalized discrepancy. The Journal of Machine Learning Research, 20(1):1-30, 2019.","title":"Adaptation based on generalized discrepancy","authors":"Corinna Cortes, Mehryar Mohri, Andrés Munoz Medina","year":"2019","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Donahue et al., 2014] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In Proceedings of the 31th International Conference on Machine Learning, pages 647-655, 2014.","title":"Decaf: A deep convolutional activation feature for generic visual recognition","authors":"Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, Trevor Darrell","year":"2014","venue":"Proceedings of the 31th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ganin and Lempitsky, 2015] Yaroslav Ganin and Victor S. Lempitsky. Unsupervised domain adaptation by backprop- agation. In Proceedings of the 32nd International Confer- ence on Machine Learning, pages 1180-1189, 2015.","title":"Unsupervised domain adaptation by backpropagation","authors":"Yaroslav Ganin, Victor S Lempitsky","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Germain et al., 2013] Pascal Germain, Amaury Habrard, Franc ¸ois Laviolette, and Emilie Morvant. A pac-bayesian approach for domain adaptation with specialization to lin- ear classifiers. In Proceedings of the 30th International Conference on Machine Learning, pages 738-746, 2013.","title":"A pac-bayesian approach for domain adaptation with specialization to linear classifiers","authors":"Pascal Germain, Amaury Habrard, Emilie Franc ¸ois Laviolette, Morvant","year":"2013","venue":"Proceedings of the 30th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gopalan et al., 2011] Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. Domain adaptation for object recogni- tion: An unsupervised approach. In 2011 international conference on computer vision, pages 999-1006. IEEE, 2011.","title":"Domain adaptation for object recognition: An unsupervised approach","authors":"Raghuraman Gopalan, Ruonan Li, Rama Chellappa","year":"2011","venue":"2011 international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Han et al., 2018] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural net- works with extremely noisy labels. In Advances in neural information processing systems, pages 8527-8537, 2018.","title":"Co-teaching: Robust training of deep neural networks with extremely noisy labels","authors":"Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, Masashi Sugiyama","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ilyas et al., 2019] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems, pages 125-136, 2019.","title":"Adversarial examples are not bugs, they are features","authors":"Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, Aleksander Madry","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jiang et al., 2018] Lu Jiang, Zhengyuan Zhou, Thomas Le- ung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on cor- rupted labels. In Proceedings of the 35th International Conference on Machine Learning, pages 2309-2318, 2018.","title":"Mentornet: Learning datadriven curriculum for very deep neural networks on corrupted labels","authors":"Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, Li Fei-Fei","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kumar et al., 2010] M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable mod- els. In Advances in Neural Information Processing Systems, pages 1189-1197, 2010.","title":"Self-paced learning for latent variable models","authors":"Benjamin M Pawan Kumar, Daphne Packer, Koller","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Long et al., 2015] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable fea- tures with deep adaptation networks. In Proceedings of the 32nd International Conference on Machine Learning, pages 97-105, 2015.","title":"Learning transferable features with deep adaptation networks","authors":"Mingsheng Long, Yue Cao, Jianmin Wang, Michael I Jordan","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Long et al., 2016] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Unsupervised domain adapta- tion with residual transfer networks. In Advances in Neural Information Processing Systems, pages 136-144, 2016.","title":"Unsupervised domain adaptation with residual transfer networks","authors":"Mingsheng Long, Han Zhu, Jianmin Wang, Michael I Jordan","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Long et al., 2017] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Deep transfer learning with joint adaptation networks. In Proceedings of the 34th In- ternational Conference on Machine Learning, pages 2208- 2217, 2017.","title":"Deep transfer learning with joint adaptation networks","authors":"Mingsheng Long, Han Zhu, Jianmin Wang, Michael I Jordan","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Long et al., 2018] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Conditional adversarial do- main adaptation. In Advances in Neural Information Pro- cessing Systems, pages 1640-1650, 2018.","title":"Conditional adversarial domain adaptation","authors":"Mingsheng Long, Zhangjie Cao, Jianmin Wang, Michael I Jordan","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mansour et al., 2009] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. In 22nd Conference on Learning Theory, COLT, 2009.","title":"Domain adaptation: Learning bounds and algorithms","authors":"Yishay Mansour, Mehryar Mohri, Afshin Rostamizadeh","year":"2009","venue":"22nd Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pan et al., 2010] Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Net- works, 22(2):199-210, 2010.","title":"Domain adaptation via transfer component analysis","authors":"Ivor W Sinno Jialin Pan, James T Tsang, Qiang Kwok, Yang","year":"2010","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":"/proceedings/2009/200"},{"position":20,"raw":"[Patrini et al., 2017] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In 2017 IEEE Conference on Com- puter Vision and Pattern Recognition, pages 2233-2241, 2017.","title":"Making deep neural networks robust to label noise: A loss correction approach","authors":"Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, Lizhen Qu","year":"2017","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Saito et al., 2017] Kuniaki Saito, Yoshitaka Ushiku, and Tat- suya Harada. Asymmetric tri-training for unsupervised domain adaptation. In Proceedings of the 34th Interna- tional Conference on Machine Learning, pages 2988-2997, 2017.","title":"Asymmetric tri-training for unsupervised domain adaptation","authors":"Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sener et al., 2016] Ozan Sener, Hyun Oh Song, Ashutosh Saxena, and Silvio Savarese. Learning transferrable rep- resentations for unsupervised domain adaptation. In Ad- vances in Neural Information Processing Systems, pages 2110-2118, 2016.","title":"Learning transferrable representations for unsupervised domain adaptation","authors":"Ozan Sener, Hyun Oh Song, Ashutosh Saxena, Silvio Savarese","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shu et al., 2019] Yang Shu, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Transferable curriculum for weakly- supervised domain adaptation. In Thirty-Third AAAI Con- ference on Artificial Intelligence, pages 4951-4958, 2019.","title":"Transferable curriculum for weaklysupervised domain adaptation","authors":"Yang Shu, Zhangjie Cao, Mingsheng Long, Jianmin Wang","year":"2019","venue":"Thirty-Third AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tzeng et al., 2017] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, pages 7167-7176, 2017.","title":"Adversarial discriminative domain adaptation","authors":"Eric Tzeng, Judy Hoffman, Kate Saenko, Trevor Darrell","year":"2017","venue":"2017 IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Valiant, 1984] Leslie G Valiant. A theory of the learnable. In Proceedings of the sixteenth annual ACM symposium on Theory of computing, pages 436-445. ACM, 1984.","title":"A theory of the learnable","authors":"G Leslie, Valiant","year":"1984","venue":"Proceedings of the sixteenth annual ACM symposium on Theory of computing","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2019] Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael I. Jordan. Bridging theory and algorithm for domain adaptation. In Proceedings of the 36th Interna- tional Conference on Machine Learning, pages 7404-7413, 2019.","title":"Bridging theory and algorithm for domain adaptation","authors":"Yuchen Zhang, Tianle Liu, Mingsheng Long, Michael I Jordan","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}