{"type":"ScholarlyArticle","title":"Domain-Adaptive Text Classification with Structured Knowledge from Unlabeled Data","authors":[{"name":"Tian Li","slug":"tian-li","given":"Tian","family":"Li","affiliation":"Peking University ,"},{"name":"Xiang Chen","slug":"xiang-chen","given":"Xiang","family":"Chen","affiliation":"Peking University ,"},{"name":"Zhen Dong","slug":"zhen-dong","given":"Zhen","family":"Dong","affiliation":"University of California , Berkeley ,"},{"name":"Kurt Keutzer","slug":"kurt-keutzer","given":"Kurt","family":"Keutzer","affiliation":"University of California , Berkeley ,"},{"name":"Shanghang Zhang","slug":"shanghang-zhang","given":"Shanghang","family":"Zhang","affiliation":"Peking University ,"}],"abstract":"Domain adaptive text classification is a challenging problem for the large-scale pretrained language models because they often require expensive additional labeled data to adapt to new domains. Existing works usually fails to leverage the implicit relationships among words across domains. In this paper, we propose a novel method, called Domain Adaptation with Structured Knowledge (DASK), to enhance domain adaptation by exploiting word-level semantic relationships. DASK first builds a knowledge graph to capture the relationship between pivot terms (domain-independent words) and non-pivot terms in the target domain. Then during training, DASK injects pivot-related knowledge graph information into source domain texts. For the downstream task, these knowledge-injected texts are fed into a BERT variant capable of processing knowledge-injected textual data. Thanks to the knowledge injection, our model learns domain-invariant features for non-pivots according to their relationships with pivots. DASK ensures the pivots to have domain-invariant behaviors by dynamically inferring via the polarity scores of candidate pivots during training with pseudo-labels. We validate DASK on a wide range of cross-domain sentiment classification tasks and observe up to 2.9% absolute performance improvement over baselines for 20 different domain pairs. Code is available at https://github.com/hikaru-nara/DASK.","doi":"10.24963/ijcai.2022/585","url":"https://ijcai.org/proceedings/2022/585","pdf":"https://ijcai.org/proceedings/2022/0585.pdf","markdown":"https://ijcai.org/proceedings/2022/585.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/585","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":585,"pages":{"first":4216,"last":4222},"keywords":["Natural Language Processing: Knowledge Extraction","Natural Language Processing: Sentiment Analysis and Text Mining","Natural Language Processing: Text Classification","Natural Language Processing: Natural Language Semantics"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987076","length_sec":85,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985219","length_sec":665,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Blitzer et al., 2006] John Blitzer, Ryan McDonald, and Fer- nando Pereira. Domain adaptation with structural corre- spondence learning. In Proceedings of the 2006 confer- ence on empirical methods in natural language process- ing, 2006.","title":"Domain adaptation with structural correspondence learning","authors":"John Blitzer, Ryan Mcdonald, Fernando Pereira","year":"2006","venue":"Proceedings of the 2006 conference on empirical methods in natural language processing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Blitzer et al., 2007] John Blitzer, Mark Dredze, and Fer- nando Pereira. Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th Annual Meeting of the Associ- ation of Computational Linguistics, 2007.","title":"Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification","authors":"John Blitzer, Mark Dredze, Fernando Pereira","year":"2007","venue":"Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2021] Xiang Chen, Yue Cao, and Xiaojun Wan. Wind: Weighting instances differentially for model- agnostic domain adaptation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2366-2376, 2021.","title":"Wind: Weighting instances differentially for modelagnostic domain adaptation","authors":"Xiang Chen, Yue Cao, Xiaojun Wan","year":"2021","venue":"Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dong et al., 2014] Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. Knowledge vault: A web-scale approach to probabilistic knowledge fusion. In SIGKDD, 2014.","title":"Knowledge vault: A web-scale approach to probabilistic knowledge fusion","authors":"Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, Wei Zhang","year":"2014","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Du et al., 2020] Chunning Du, Haifeng Sun, Jingyu Wang, Qi Qi, and Jianxin Liao. Adversarial and domain-aware bert for cross-domain sentiment analysis. In ACL, 2020.","title":"Adversarial and domain-aware bert for cross-domain sentiment analysis","authors":"Du","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ganin et al., 2016] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Franc ¸ois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The jour- nal of machine learning research, 2016.","title":"Domain-adversarial training of neural networks","authors":"Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Mario Franc ¸ois Laviolette, Victor Marchand, Lempitsky","year":"2016","venue":"The journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ghosal et al., 2020] Deepanway Ghosal, Devamanyu Haz- arika, Abhinaba Roy, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. Kingdom: Knowledge-guided do- main adaptation for sentiment analysis. In ACL, 2020.","title":"Kingdom: Knowledge-guided domain adaptation for sentiment analysis","authors":"Ghosal","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Glorot et al., 2011] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Domain adaptation for large-scale sen- timent classification: A deep learning approach. In ICML, 2011.","title":"Domain adaptation for large-scale sentiment classification: A deep learning approach","authors":"Xavier Glorot, Antoine Bordes, Yoshua Bengio","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2018] Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. Adaptive semi-supervised learning for cross-domain sentiment classification. arXiv preprint arXiv:1809.00530, 2018.","title":"Adaptive semi-supervised learning for cross-domain sentiment classification","authors":"He","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lehmann et al., 2015] Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. Dbpedia-a large-scale, mul- tilingual knowledge base extracted from wikipedia. Se- mantic web, 2015.","title":"Dbpedia-a large-scale, multilingual knowledge base extracted from wikipedia. Semantic web","authors":"Lehmann","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2018] Zheng Li, Ying Wei, Yu Zhang, and Qiang Yang. Hierarchical attention transfer network for cross- domain sentiment classification. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.","title":"Hierarchical attention transfer network for crossdomain sentiment classification","authors":"Zheng Li, Ying Wei, Yu Zhang, Qiang Yang","year":"2018","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2021] Tian Li, Xiang Chen, Shanghang Zhang, Zhen Dong, and Kurt Keutzer. Cross-domain sentiment classification with contrastive learning and mutual infor- mation maximization. In ICASSP, 2021.","title":"Cross-domain sentiment classification with contrastive learning and mutual information maximization","authors":"Tian Li, Xiang Chen, Shanghang Zhang, Zhen Dong, Kurt Keutzer","year":"2021","venue":"ICASSP","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.","title":"Roberta: A robustly optimized bert pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2020] Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. K-bert: En- abling language representation with knowledge graph. In Proc. of AAAI, 2020.","title":"K-bert: Enabling language representation with knowledge graph","authors":"Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, Ping Wang","year":"2020","venue":"Proc. of AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Long et al., 2015] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In ICML, 2015.","title":"Learning transferable features with deep adaptation networks","authors":"Mingsheng Long, Yue Cao, Jianmin Wang, Michael Jordan","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shiv and Quirk, 2019] Vighnesh Leonardo Shiv and Chris Quirk. Novel positional encodings to enable tree-based transformers. In NeurIPS, 2019.","title":"Vighnesh Leonardo Shiv and Chris Quirk. Novel positional encodings to enable tree-based transformers","authors":"Quirk Shiv","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sun et al., 2016] Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In AAAI, 2016.","title":"Return of frustratingly easy domain adaptation","authors":"Sun","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sun et al., 2020] Zeyu Sun, Qihao Zhu, Yingfei Xiong, Yi- can Sun, Lili Mou, and Lu Zhang. Treegen: A tree-based transformer architecture for code generation. In AAAI, 2020.","title":"Treegen: A tree-based transformer architecture for code generation","authors":"Sun","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tzeng et al., 2017] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In CVPR, 2017.","title":"Adversarial discriminative domain adaptation","authors":"Eric Tzeng, Judy Hoffman, Kate Saenko, Trevor Darrell","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vrandečić and Krötzsch, 2014] Denny Vrandečić and Markus Krötzsch. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 2014.","title":"Wikidata: a free collaborative knowledgebase","authors":"Krötzsch Vrandečić, Denny Vrandečić, Markus Krötzsch","year":"2014","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2020a] Chenguang Wang, Xiao Liu, and Dawn Song. Language models are open knowledge graphs. arXiv preprint arXiv:2010.11967, 2020.","title":"Language models are open knowledge graphs","authors":"Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2020b] Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Cuihong Cao, Daxin Jiang, Ming Zhou, et al. K-adapter: Infusing knowl- edge into pre-trained models with adapters. arXiv preprint arXiv:2002.01808, 2020.","title":"K-adapter: Infusing knowledge into pre-trained models with adapters","authors":"Wang","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2021] Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. Kepler: A unified model for knowledge embedding and pre-trained language representation. ACL, 2021.","title":"Kepler: A unified model for knowledge embedding and pre-trained language representation","authors":"Wang","year":"2021","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yu and Jiang, 2016] Jianfei Yu and Jing Jiang. Learn- ing sentence embeddings with auxiliary tasks for cross- domain sentiment classification. In EMNLP, 2016.","title":"Learning sentence embeddings with auxiliary tasks for crossdomain sentiment classification","authors":"Jianfei Yu, Jing Jiang","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2019] Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. Ernie: Enhanced language representation with informative entities. arXiv preprint arXiv:1905.07129, 2019.","title":"Ernie: Enhanced language representation with informative entities","authors":"Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, Qun Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ziser and Reichart, 2018] Yftah Ziser and Roi Reichart. Pivot based language modeling for improved neural do- main adaptation. In Proc. of ACL, 2018.","title":"Yftah Ziser and Roi Reichart. Pivot based language modeling for improved neural domain adaptation","authors":"Reichart Ziser","year":"2018","venue":"Proc. of ACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}