{"type":"ScholarlyArticle","title":"Searching for Optimal Subword Tokenization in Cross-domain NER","authors":[{"name":"Ruotian Ma","slug":"ruotian-ma","given":"Ruotian","family":"Ma","affiliation":"School of Computer Science , Fudan University , Shanghai , China"},{"name":"Yiding Tan","slug":"yiding-tan","given":"Yiding","family":"Tan","affiliation":"School of Computer Science , Fudan University , Shanghai , China"},{"name":"Xin Zhou","slug":"xin-zhou","given":"Xin","family":"Zhou","affiliation":"School of Computer Science , Fudan University , Shanghai , China"},{"name":"Xuanting Chen","slug":"xuanting-chen","given":"Xuanting","family":"Chen","affiliation":"School of Computer Science , Fudan University , Shanghai , China"},{"name":"Di Liang","slug":"di-liang","given":"Di","family":"Liang","affiliation":"Meituan Inc. , Beijing , China"},{"name":"Sirui Wang","slug":"sirui-wang","given":"Sirui","family":"Wang","affiliation":"Meituan Inc. , Beijing , China"},{"name":"Wei Wu","slug":"wei-wu","given":"Wei","family":"Wu","affiliation":"Meituan Inc. , Beijing , China"},{"name":"Tao Gui","slug":"tao-gui","given":"Tao","family":"Gui","affiliation":"Institute of Modern Languages and Linguistics , Fudan University , Shanghai , China"}],"abstract":"Input distribution shift is one of the vital problems in unsupervised domain adaptation (UDA). The most popular UDA approaches focus on domain-invariant representation learning, trying to align the features from different domains into a similar feature distribution. However, these approaches ignore the direct alignment of input word distributions between domains, which is a vital factor in word-level classification tasks such as cross-domain NER. In this work, we shed new light on cross-domain NER by introducing a subword-level solution, X-Piece, for input word-level distribution shift in NER. Specifically, we re-tokenize the input words of the source domain to approach the target subword distribution, which is formulated and solved as an optimal transport problem. As this approach focuses on the input level, it can also be combined with previous DIRL methods for further improvement. Experimental results show the effectiveness of the proposed method based on BERT-tagger on four benchmark NER datasets. Also, the proposed method is proved to benefit DIRL methods such as DANN.","doi":"10.24963/ijcai.2022/595","url":"https://ijcai.org/proceedings/2022/595","pdf":"https://ijcai.org/proceedings/2022/0595.pdf","markdown":"https://ijcai.org/proceedings/2022/595.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/595","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":595,"pages":{"first":4289,"last":4295},"keywords":["Natural Language Processing: Information Extraction","Natural Language Processing: Tagging, Chunking, and Parsing"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987111","length_sec":84,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985254","length_sec":718,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Chen and Moschitti, 2019] Lingzhen Chen and Alessandro Moschitti. Transfer learning for sequence labeling using source model and target data. In AAAI, 2019.","title":"Transfer learning for sequence labeling using source model and target data","authors":"Chen, Moschitti ;, Lingzhen Chen, Alessandro Moschitti","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cortes et al., 2008] Corinna Cortes, Mehryar Mohri, Michael Riley, and Afshin Rostamizadeh. Sample selection bias correction theory. In ALT, pages 38-53, 2008.","title":"Sample selection bias correction theory","authors":"Corinna Cortes, Mehryar Mohri, Michael Riley, Afshin Rostamizadeh","year":"2008","venue":"ALT","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, pages 4171-4186, June 2019.","title":"BERT: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fu et al., 2017] Lisheng Fu, Thien Huu Nguyen, Bonan Min, and Ralph Grishman. Domain adaptation for relation extrac- tion with domain adversarial neural network. In IJCNLP (Short Papers), pages 425-429, November 2017.","title":"Domain adaptation for relation extraction with domain adversarial neural network","authors":"Lisheng Fu, Thien Huu Nguyen, Bonan Min, Ralph Grishman","year":"2017","venue":"IJCNLP (Short Papers)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fu et al., 2020] Jinlan Fu, Pengfei Liu, and Qi Zhang. Re- thinking generalization of neural models: A named entity recognition case study. In AAAI, pages 7732-7739, 2020.","title":"Rethinking generalization of neural models: A named entity recognition case study","authors":"Fu","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ganin et al., 2016] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The jour- nal of machine learning research, 17(1):2096-2030, 2016.","title":"Domain-adversarial training of neural networks","authors":"Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, Victor Lempitsky","year":"2016","venue":"The journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jia and Zhang, 2020] Chen Jia and Yue Zhang. Multi-cell compositional LSTM for NER domain adaptation. In ACL, pages 5906-5917, July 2020.","title":"Multi-cell compositional LSTM for NER domain adaptation","authors":"Chen Jia, Yue Zhang","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jiang and Zhai, 2007] Jing Jiang and ChengXiang Zhai. In- stance weighting for domain adaptation in NLP. In ACL, pages 264-271, June 2007.","title":"Instance weighting for domain adaptation in NLP","authors":"Zhai ; ] Jing Jiang, Chengxiang Jiang, Zhai","year":"2007","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kudo, 2018] Taku Kudo. Subword regularization: Improv- ing neural network translation models with multiple sub- word candidates. In ACL, pages 66-75, July 2018.","title":"Subword regularization: Improving neural network translation models with multiple subword candidates","authors":"Taku Kudo, Kudo","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2021] Yangming Li, lemao liu, and Shuming Shi. Empirical analysis of unlabeled entity problem in named entity recognition. In ICLR, 2021.","title":"Empirical analysis of unlabeled entity problem in named entity recognition","authors":"Li","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liang et al., 2020] Chen Liang, Yue Yu, Haoming Jiang, Siawpeng Er, Ruijia Wang, Tuo Zhao, and Chao Zhang. Bond: Bert-assisted open-domain named entity recogni- tion with distant supervision. In ACM SIGKDD, page 1054-1064, 2020.","title":"Bond: Bert-assisted open-domain named entity recognition with distant supervision","authors":"Chen Liang, Yue Yu, Haoming Jiang, Siawpeng Er, Ruijia Wang, Tuo Zhao, Chao Zhang","year":"2020","venue":"ACM SIGKDD","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.","title":"Roberta: A robustly optimized bert pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2021a] Xin Liu, Baosong Yang, Dayiheng Liu, Haibo Zhang, Weihua Luo, Min Zhang, Haiying Zhang, and Jinsong Su. Bridging subword gaps in pretrain-finetune paradigm for natural language generation. In ACL, pages 6001-6011, August 2021.","title":"Bridging subword gaps in pretrain-finetune paradigm for natural language generation","authors":"Liu","year":"2021","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2021b] Zihan Liu, Yan Xu, Tiezheng Yu, Wen- liang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. Crossner: Evaluating cross-domain named entity recognition. AAAI, pages 13452-13460, May 2021.","title":"Crossner: Evaluating cross-domain named entity recognition","authors":"Liu","year":"2021","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Long et al., 2017] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I. Jordan. Deep transfer learning with joint adaptation networks. In ICML, volume 70 of Pro- ceedings of Machine Learning Research, pages 2208-2217, 06-11 Aug 2017.","title":"Deep transfer learning with joint adaptation networks","authors":"Mingsheng Long, Han Zhu, Jianmin Wang, Michael I Jordan","year":"2017","venue":"Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pan et al., 2010] Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE transactions on neural networks, 22(2):199-210, 2010.","title":"Domain adaptation via transfer component analysis","authors":"Ivor W Sinno Jialin Pan, James T Tsang, Qiang Kwok, Yang","year":"2010","venue":"IEEE transactions on neural networks","doi":null,"url":null,"internal_url":"/proceedings/2009/200"},{"position":17,"raw":"[Peng et al., 2018] Minlong Peng, Qi Zhang, Yu-gang Jiang, and Xuanjing Huang. Cross-domain sentiment classifica- tion with target domain specific information. In ACL, pages 2505-2513, July 2018.","title":"Cross-domain sentiment classification with target domain specific information","authors":"Peng","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Provilkov et al., 2020] Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. BPE-dropout: Simple and effective sub- word regularization. In ACL, pages 1882-1892, July 2020.","title":"BPE-dropout: Simple and effective subword regularization","authors":"Ivan Provilkov, Dmitrii Emelianenko, Elena Voita","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sato et al., 2020] Shoetsu Sato, Jin Sakuma, Naoki Yoshi- naga, Masashi Toyoda, and Masaru Kitsuregawa. Vocab- ulary adaptation for domain adaptation in neural machine translation. In Findings of EMNLP, pages 4269-4279, November 2020.","title":"Vocabulary adaptation for domain adaptation in neural machine translation","authors":"Sato","year":"2020","venue":"Findings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schick and Schütze, 2020] Timo Schick and Hinrich Schütze. BERTRAM: Improved word embeddings have big impact on contextualized model performance. In ACL, pages 3996-4007, July 2020.","title":"BERTRAM: Improved word embeddings have big impact on contextualized model performance","authors":"Schütze ; ] Timo Schick, Hinrich Schick, Schütze","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shah et al., 2018] Darsh Shah, Tao Lei, Alessandro Mos- chitti, Salvatore Romeo, and Preslav Nakov. Adversarial domain adaptation for duplicate question detection. In EMNLP, pages 1056-1063, October-November 2018.","title":"Adversarial domain adaptation for duplicate question detection","authors":"Darsh Shah, Tao Lei, Alessandro Moschitti, Salvatore Romeo, Preslav Nakov","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yosinski et al., 2014] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In NIPS, volume 27, 2014.","title":"How transferable are features in deep neural networks?","authors":"Jason Yosinski, Jeff Clune, Yoshua Bengio, Hod Lipson","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zellinger et al., 2017] Werner Zellinger, Thomas Grubinger, Edwin Lughofer, Thomas Natschläger, and Susanne Saminger-Platz. Central moment discrepancy (cmd) for domain-invariant representation learning. arXiv preprint arXiv:1702.08811, 2017.","title":"Central moment discrepancy (cmd) for domain-invariant representation learning","authors":"Werner Zellinger, Thomas Grubinger, Edwin Lughofer, Thomas Natschläger, Susanne Saminger-Platz","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2021] Xin Zhang, Guangwei Xu, Yueheng Sun, Meishan Zhang, and Pengjun Xie. Crowdsourcing learn- ing as domain adaptation: A case study on named entity recognition. In ACL, pages 5558-5570, August 2021.","title":"Crowdsourcing learning as domain adaptation: A case study on named entity recognition","authors":"Xin Zhang, Guangwei Xu, Yueheng Sun, Meishan Zhang, Pengjun Xie","year":"2021","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zou et al., 2021] Han Zou, Jianfei Yang, and Xiaojian Wu. Unsupervised energy-based adversarial domain adaptation for cross-domain text classification. In Findings of ACL, pages 1208-1218, August 2021.","title":"Unsupervised energy-based adversarial domain adaptation for cross-domain text classification","authors":"Han Zou, Jianfei Yang, Xiaojian Wu","year":"2021","venue":"Findings of ACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}