{"type":"ScholarlyArticle","title":"Few-Shot Adaptation of Pre-Trained Networks for Domain Shift","authors":[{"name":"Wenyu Zhang","slug":"wenyu-zhang","given":"Wenyu","family":"Zhang","affiliation":"Institute for Infocomm Research , A"},{"name":"Li Shen","slug":"li-shen","given":"Li","family":"Shen","affiliation":"Institute for Infocomm Research , A"},{"name":"Wanyue Zhang","slug":"wanyue-zhang","given":"Wanyue","family":"Zhang","affiliation":"Max Planck Institute for Informatics; * STAR"},{"name":"Chuan-Sheng Foo","slug":"chuan-sheng-foo","given":"Chuan-Sheng","family":"Foo","affiliation":"Institute for Infocomm Research , A; Centre for Frontier AI Research , A"}],"abstract":"Deep networks are prone to performance degradation when there is a domain shift between the source (training) data and target (test) data. Recent test-time adaptation methods update batch normalization layers of pre-trained source models deployed in new target environments with streaming data. Although these methods can adapt on-the-fly without first collecting a large target domain dataset, their performance is dependent on streaming conditions such as mini-batch size and class-distribution which can be unpredictable in practice. In this work, we propose a framework for few-shot domain adaptation to address the practical challenges of data-efficient adaptation. Specifically, we propose a constrained optimization of feature normalization statistics in pre-trained source models supervised by a small target domain support set. Our method is easy to implement and improves source model performance with as little as one sample per class for classification tasks. Extensive experiments on 5 cross-domain classification and 4 semantic segmentation datasets show that our proposed method achieves more accurate and reliable performance than test-time adaptation, while not being constrained by streaming conditions.","doi":"10.24963/ijcai.2022/232","url":"https://ijcai.org/proceedings/2022/232","pdf":"https://ijcai.org/proceedings/2022/0232.pdf","markdown":"https://ijcai.org/proceedings/2022/232.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/232","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":232,"pages":{"first":1665,"last":1671},"keywords":["Machine Learning: Robustness","Computer Vision: Transfer, low-shot, semi- and un- supervised learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986683","length_sec":86,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984826","length_sec":348,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Cao et al., 2019] K. Cao, C. Wei, A. Gaidon, N. Arechiga, and T. Ma. Learning imbalanced datasets with label- distribution-aware margin loss. NeurIPS, 2019.","title":"Learning imbalanced datasets with labeldistribution-aware margin loss","authors":"K Cao, C Wei, A Gaidon, N Arechiga, T Ma","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chang et al., 2019] W.-G. Chang, T. You, S. Seo, S. Kwak, and B. Han. Domain-specific batch normalization for un- supervised domain adaptation. CVPR, 2019.","title":"Domain-specific batch normalization for unsupervised domain adaptation","authors":"W.-G Chang, T You, S Seo, S Kwak, B Han","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Choi et al., 2021] S. Choi, S. Jung, H. Yun, J. Kim, S. Kim, and J. Choo. Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whiten- ing. CVPR, 2021.","title":"Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening","authors":"S Choi, S Jung, H Yun, J Kim, S Kim, J Choo","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gulrajani and Lopez-Paz, 2021] I. Gulrajani and D. Lopez- Paz. In search of lost domain generalization. ICLR, 2021.","title":"search of lost domain generalization","authors":"Lopez-Paz ; I Gulrajani, D Gulrajani, Lopez-Paz","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hoffman et al., 2013] J. Hoffman, E. Tzeng, J. Donahue, Y. Jia, K. Saenko, and Trevor D. One-shot adaptation of supervised deep convolutional models. ArXiv, 2013.","title":"One-shot adaptation of supervised deep convolutional models","authors":"J Hoffman, E Tzeng, J Donahue, Y Jia, K Saenko, Trevor D","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Huang and Belongie, 2017] X. Huang and S. Belongie. Ar- bitrary style transfer in real-time with adaptive instance normalization. ICCV, 2017.","title":"Arbitrary style transfer in real-time with adaptive instance normalization","authors":"Belongie Huang, X Huang, S Belongie","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Huang et al., 2019] L. Huang, Y. Zhou, F. Zhu, L. Liu, and L. Shao. Iterative normalization: Beyond standardization towards efficient whitening. CVPR, 2019.","title":"Iterative normalization: Beyond standardization towards efficient whitening","authors":"L Huang, Y Zhou, F Zhu, L Liu, L Shao","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kim et al., 2020] Y. Kim, S. Hong, D. Cho, H. Park, and P. Panda. Domain adaptation without source data. ArXiv, 2020.","title":"Domain adaptation without source data","authors":"Y Kim, S Hong, D Cho, H Park, P Panda","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Koh et al., 2020] P. Koh, S. Sagawa, H. Marklund, S. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Yasunaga, R. Phillips, S. Beery, J. Leskovec, A. Kundaje, E. Pierson, S. Levine, C. Finn, and P. Liang. Wilds: A benchmark of in-the-wild distribution shifts. ArXiv, 2020.","title":"Wilds: A benchmark of in-the-wild distribution shifts","authors":"P Koh, S Sagawa, H Marklund, S Xie, M Zhang, A Balsubramani, W Hu, M Yasunaga, R Phillips, S Beery, J Leskovec, A Kundaje, E Pierson, S Levine, C Finn, P Liang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kundu et al., 2020] J. N. Kundu, N. Venkat, R. Ambareesh, R. M. V., and R. V. Babu. Towards inheritable models for open-set domain adaptation. CVPR, 2020.","title":"Towards inheritable models for open-set domain adaptation","authors":"J N Kundu, N Venkat, R Ambareesh, R M V, R V Babu","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kurmi et al., 2021] V. Kurmi, Venkatesh K. Subramanian, and Vinay P. Namboodiri. Domain impression: A source data free domain adaptation method. WACV, 2021.","title":"Domain impression: A source data free domain adaptation method","authors":"V Kurmi, K Venkatesh, Vinay P Subramanian, Namboodiri","year":"2021","venue":"WACV","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2016] Y. Li, N. Wang, J. Shi, J. Liu, and X. Hou. Revisiting batch normalization for practical domain adap- tation. Pattern Recognition, 80, 03 2016.","title":"Revisiting batch normalization for practical domain adaptation","authors":"Y Li, N Wang, J Shi, J Liu, X Hou","year":"2016","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2018] X. Li, Y. Grandvalet, and F. Davoine. Ex- plicit inductive bias for transfer learning with convolu- tional networks. ICML, 2018.","title":"Explicit inductive bias for transfer learning with convolutional networks","authors":"Li","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2019] X. Li, H. Xiong, H. Wang, Y. Rao, L. Liu, and J. Huan. DELTA: Deep learning transfer using fea- ture map with attention for convolutional networks. ICLR, 2019.","title":"DELTA: Deep learning transfer using feature map with attention for convolutional networks","authors":"Li","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2020] R. Li, Q. Jiao, W. Cao, H.-S. Wong, and S. Wu. Model adaptation: Unsupervised domain adap- tation without source data. CVPR, 2020.","title":"Model adaptation: Unsupervised domain adaptation without source data","authors":"R Li, Q Jiao, W Cao, H.-S Wong, S Wu","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liang et al., 2020] J. Liang, D. Hu, and J. Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. ICML, 2020.","title":"Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation","authors":"Liang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nado et al., 2020] Z. Nado, S. Padhy, D. Sculley, A. D'Amour, B. Lakshminarayanan, and J. Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. ArXiv, 2020.","title":"Evaluating prediction-time batch normalization for robustness under covariate shift","authors":"Z Nado, S Padhy, D Sculley, A D'amour, B Lakshminarayanan, J Snoek","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nam and Kim, 2018] H. Nam and H.-E. Kim. Batch- instance normalization for adaptively style-invariant neu- ral networks. NeurIPS, 2018.","title":"Batchinstance normalization for adaptively style-invariant neural networks","authors":"H Nam, H.-E Kim","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pan and Yang, 2010] S. J. Pan and Q. Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):1345-1359, 2010.","title":"A survey on transfer learning","authors":"S J Pan, Q Yang","year":"2010","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pan et al., 2018] X. Pan, P. Luo, J. Shi, and X. Tang. Two at once: Enhancing learning and generalization capacities via ibn-net. ECCV, 2018.","title":"Two at once: Enhancing learning and generalization capacities via ibn-net","authors":"X Pan, P Luo, J Shi, X Tang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pan et al., 2019] X. Pan, X. Zhan, J. Shi, X. Tang, and P. Luo. Switchable whitening for deep representation learning. ICCV, 2019.","title":"Switchable whitening for deep representation learning","authors":"X Pan, X Zhan, J Shi, X Tang, P Luo","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Qiu et al., 2021] Z. Qiu, Y. Zhang, H. Lin, S. Niu, Y. Liu, Q. Du, and M. Tan. Source-free domain adaptation via avatar prototype generation and adaptation. IJCAI, 2021.","title":"Source-free domain adaptation via avatar prototype generation and adaptation","authors":"Z Qiu, Y Zhang, H Lin, S Niu, Y Liu, Q Du, M Tan","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2021/402"},{"position":23,"raw":"[Schneider et al., 2020] S. Schneider, E. Rusak, L. Eck, O. Bringmann, W. Brendel, and M. Bethge. Improving robustness against common corruptions by covariate shift adaptation. NeurIPS, 2020.","title":"Improving robustness against common corruptions by covariate shift adaptation","authors":"S Schneider, E Rusak, L Eck, O Bringmann, W Brendel, M Bethge","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Scott et al., 2018] T. R. Scott, K. Ridgeway, and M. Mozer. Adapted deep embeddings: A synthesis of methods for k- shot inductive transfer learning. NeurIPS, 2018.","title":"Adapted deep embeddings: A synthesis of methods for kshot inductive transfer learning","authors":"T R Scott, K Ridgeway, M Mozer","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sun et al., 2020] Y. Sun, X. Wang, Z. Liu, J. Miller, A. A. Efros, and M. Hardt. Test-time training for out-of- distribution generalization. ICML, 2020.","title":"Test-time training for out-ofdistribution generalization","authors":"Y Sun, X Wang, Z Liu, J Miller, A A Efros, M Hardt","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Triantafillou et al., 2021] E. Triantafillou, H. Larochelle, R. S. Zemel, and V. Dumoulin. Learning a universal tem- plate for few-shot dataset generalization. ICML, 2021.","title":"Learning a universal template for few-shot dataset generalization","authors":"E Triantafillou, H Larochelle, R S Zemel, V Dumoulin","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2021] D. Wang, E. Shelhamer, S. Liu, B. Ol- shausen, and T. Darrell. Tent: Fully test-time adaptation by entropy minimization. ICLR, 2021.","title":"Tent: Fully test-time adaptation by entropy minimization","authors":"D Wang, E Shelhamer, S Liu, B Olshausen, T Darrell","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wilson and Cook, 2020] G. Wilson and D. J. Cook. A sur- vey of unsupervised deep domain adaptation. ACM Trans. Intell. Syst. Technol., 11(5), July 2020.","title":"A survey of unsupervised deep domain adaptation","authors":"G Wilson, D J Cook","year":"2020","venue":"ACM Trans. Intell. Syst. Technol","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2021] S. Yang, Y. Wang, J. Weijer, L. Herranz, and S. Jui. Generalized source-free domain adaptation. ICCV, 2021.","title":"Generalized source-free domain adaptation","authors":"S Yang, Y Wang, J Weijer, L Herranz, S Jui","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yoo et al., 2018] D. Yoo, H. Fan, V. N. Boddeti, and K. M. Kitani. Efficient k-shot learning with regularized deep net- works. AAAI, 2018.","title":"Efficient k-shot learning with regularized deep networks","authors":"D Yoo, H Fan, V N Boddeti, K M Kitani","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yosinski et al., 2014] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson. How transferable are features in deep neural networks? NIPS, 2014.","title":"How transferable are features in deep neural networks? NIPS","authors":"J Yosinski, J Clune, Y Bengio, H Lipson","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhou et al., 2021] K. Zhou, Y. Yang, Y. Qiao, and T. Xiang. Domain generalization with mixstyle. ICLR, 2021.","title":"Domain generalization with mixstyle","authors":"K Zhou, Y Yang, Y Qiao, T Xiang","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}