{"type":"ScholarlyArticle","title":"Improving Low Resource Named Entity Recognition using Cross-lingual Knowledge Transfer","authors":[{"name":"Xiaocheng Feng","slug":"xiaocheng-feng","given":"Xiaocheng","family":"Feng","affiliation":"Harbin Institute of Technology , China"},{"name":"Xiachong Feng","slug":"xiachong-feng","given":"Xiachong","family":"Feng","affiliation":"Harbin Institute of Technology , China"},{"name":"Bing Qin","slug":"bing-qin","given":"Bing","family":"Qin","affiliation":"Harbin Institute of Technology , China"},{"name":"Zhangyin Feng","slug":"zhangyin-feng","given":"Zhangyin","family":"Feng","affiliation":"Harbin Institute of Technology , China"},{"name":"Ting Liu","slug":"ting-liu","given":"Ting","family":"Liu","affiliation":"Harbin Institute of Technology , China"}],"abstract":"Neural networks have been widely used for high resource language (e.g. English) named entity recognition (NER) and have shown state-of-the-art results.However, for low resource languages, such as Dutch, Spanish, due to the limitation of resources and lack of annotated data, taggers tend to have lower performances.To narrow this gap, we propose three novel strategies to enrich the semantic representations of low resource languages: we first develop neural networks to improve low resource word representations by knowledge transfer from high resource language using bilingual lexicons. Further, a lexicon extension strategy is designed to address out-of lexicon problem by automatically learning semantic projections.Thirdly, we regard word-level entity type distribution features as an external language-independent knowledge and incorporate them into our neural architecture. Experiments on two low resource languages (including Dutch and Spanish) demonstrate the effectiveness of these additional semantic representations (average 4.8\\% improvement). Moreover, on Chinese OntoNotes 4.0 dataset, our approach achieved an F-score of 83.07\\% with 2.91\\% absolute gain compared to the state-of-the-art results.","doi":"10.24963/ijcai.2018/566","url":"https://ijcai.org/proceedings/2018/566","pdf":"https://ijcai.org/proceedings/2018/0566.pdf","markdown":"https://ijcai.org/proceedings/2018/566.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/566","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":566,"pages":{"first":4071,"last":4077},"keywords":["Machine Learning: Neural Networks","Machine Learning: Deep Learning","Natural Language Processing: Named Entities","Natural Language Processing: Information Extraction"],"videos":[],"references":[{"position":1,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. CoRR, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2015] Yoshua Bengio, Ian J Goodfellow, and Aaron Courville. Deep learning. Nature, 2015.","title":"Deep learning","authors":"Yoshua Bengio, Ian J Goodfellow, Aaron Courville","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Burkett et al., 2010] David Burkett, Slav Petrov, John Blitzer, and Dan Klein. Learning better monolingual mod- els with unannotated bilingual text. In CONLL, pages 46- 54, 2010.","title":"Learning better monolingual models with unannotated bilingual text","authors":"David Burkett, Slav Petrov, John Blitzer, Dan Klein","year":"2010","venue":"CONLL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Che et al., 2013] Wanxiang Che, Mengqiu Wang, Christo- pher D Manning, and Ting Liu. Named entity recognition with bilingual constraints. In HLT-NAACL, 2013.","title":"Named entity recognition with bilingual constraints","authors":"Wanxiang Che, Mengqiu Wang, Christopher D Manning, Ting Liu","year":"2013","venue":"HLT-NAACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2010] Yufeng Chen, Chengqing Zong, and Keh-Yih Su. On jointly recognizing and aligning bilingual named entities. In ACL, pages 631-639, 2010.","title":"On jointly recognizing and aligning bilingual named entities","authors":"Yufeng Chen, Chengqing Zong, Keh-Yih Su","year":"2010","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chieu and Ng, 2002] Hai Leong Chieu and Hwee Tou Ng. Named entity recognition: a maximum entropy approach using global information. In COLING, pages 1-7, 2002.","title":"Named entity recognition: a maximum entropy approach using global information","authors":"Ng ; ] Chieu, Leong Hai, Hwee Tou Chieu, Ng","year":"2002","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chiu and Nichols, 2015] Jason PC Chiu and Eric Nichols. Named entity recognition with bidirectional lstm-cnns. arXiv preprint arXiv:1511.08308, 2015.","title":"Named entity recognition with bidirectional lstm-cnns","authors":"Jason Pc Chiu, Eric Nichols","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Florian et al., 2003] Radu Florian, Abe Ittycheriah, Hongyan Jing, and Tong Zhang. Named entity recognition through classifier combination. In HLT-NAACL, pages 168-171, 2003.","title":"Named entity recognition through classifier combination","authors":"Radu Florian, Abe Ittycheriah, Hongyan Jing, Tong Zhang","year":"2003","venue":"HLT-NAACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gers et al., 1999] Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to forget: Continual prediction with lstm. 1999.","title":"Learning to forget: Continual prediction with lstm","authors":"Jürgen Felix A Gers, Fred Schmidhuber, Cummins","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2015] Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991, 2015.","title":"Bidirectional lstm-crf models for sequence tagging","authors":"Zhiheng Huang, Wei Xu, Kai Yu","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jurafsky and Martin, 2000] Dan Jurafsky and Jameás H Martin. Speech & language processing. Pearson Educa- tion India, 2000.","title":"Speech & language processing","authors":"Martin ; Jurafsky, Dan Jurafsky, Jameás H Martin","year":"2000","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim et al., 2012] Sungchul Kim, Kristina Toutanova, and Hwanjo Yu. Multilingual named entity recognition using parallel data and metadata from wikipedia. In ACL, 2012.","title":"Multilingual named entity recognition using parallel data and metadata from wikipedia","authors":"Kim","year":"2012","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lafferty et al., 2001] John Lafferty, Andrew McCallum, and Fernando Pereira. Conditional random fields: Proba- bilistic models for segmenting and labeling sequence data. In ICML, volume 1, pages 282-289, 2001.","title":"Conditional random fields: Probabilistic models for segmenting and labeling sequence data","authors":"Lafferty","year":"2001","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lample et al., 2016] Guillaume Lample, Miguel Balles- teros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. Neural architectures for named entity recog- nition. arXiv preprint arXiv:1603.01360, 2016.","title":"Neural architectures for named entity recognition","authors":"Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, Chris Dyer","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2012] Qi Li, Haibo Li, Heng Ji, Wen Wang, Jing Zheng, and Fei Huang. Joint bilingual name tagging for parallel corpora. In CIKM, pages 1727-1731. ACM, 2012.","title":"Joint bilingual name tagging for parallel corpora","authors":"Qi Li, Haibo Li, Heng Ji, Wen Wang, Jing Zheng, Fei Huang","year":"2012","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2017a] Liyuan Liu, Jingbo Shang, Frank Xu, Xi- ang Ren, Huan Gui, Jian Peng, and Jiawei Han. Empower sequence labeling with task-aware neural language model. arXiv preprint arXiv:1709.04109, 2017.","title":"Empower sequence labeling with task-aware neural language model","authors":"Liu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2017b] Tianyu Liu, Kexiang Wang, Lei Sha, Baobao Chang, and Zhifang Sui. Table-to-text generation by structure-aware seq2seq learning. CoRR, 2017.","title":"Table-to-text generation by structure-aware seq2seq learning","authors":"Liu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ma and Hovy, 2016] Xuezhe Ma and Eduard Hovy. End- to-end sequence labeling via bi-directional lstm-cnns-crf. arXiv preprint arXiv:1603.01354, 2016.","title":"Xuezhe Ma and Eduard Hovy. Endto-end sequence labeling via bi-directional lstm-cnns-crf","authors":"Hovy Ma","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Manning and Schütze, 1999] Christopher D Manning and Hinrich Schütze. Foundations of statistical natural lan- guage processing. MIT press, 1999.","title":"Foundations of statistical natural language processing","authors":"D Christopher, Hinrich Manning, Schütze","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed rep- resentations of words and phrases and their composition- ality. In NIPS, pages 3111-3119, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ni and Florian, 2017] Jian Ni and Radu Florian. Improving multilingual named entity recognition with wikipedia en- tity type mapping. EMNLP, 2017.","title":"Improving multilingual named entity recognition with wikipedia entity type mapping","authors":"Jian Ni, Radu Florian","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ni et al., 2017] Jian Ni, Georgiana Dinu, and Radu Flo- rian. Weakly supervised cross-lingual named entity recog- nition via effective annotation and representation projec- tion. ACL, page 1470-1480, 2017.","title":"Weakly supervised cross-lingual named entity recognition via effective annotation and representation projection","authors":"Jian Ni, Georgiana Dinu, Radu Florian","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Peters et al., 2017] Matthew E Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. Semi- supervised sequence tagging with bidirectional language models. arXiv preprint arXiv:1705.00108, 2017.","title":"Semisupervised sequence tagging with bidirectional language models","authors":"Waleed Matthew E Peters, Chandra Ammar, Russell Bhagavatula, Power","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ratinov and Roth, 2009] Lev Ratinov and Dan Roth. De- sign challenges and misconceptions in named entity recog- nition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning, pages 147- 155. Association for Computational Linguistics, 2009.","title":"Lev Ratinov and Dan Roth. Design challenges and misconceptions in named entity recognition","authors":"Roth Ratinov","year":"2009","venue":"Proceedings of the Thirteenth Conference on Computational Natural Language Learning","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Santos and Guimarães, 2015] Cicero Nogueira dos Santos and Victor Guimarães. Boosting named entity recogni- tion with neural character embeddings. arXiv preprint arXiv:1505.05008, 2015.","title":"Cicero Nogueira dos Santos and Victor Guimarães. Boosting named entity recognition with neural character embeddings","authors":"Guimarães Santos","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sukhbaatar et al., 2015] Sainbayar Sukhbaatar, Jason We- ston, Rob Fergus, et al. End-to-end memory networks. In NIPS, pages 2440-2448, 2015.","title":"End-to-end memory networks","authors":"Sukhbaatar","year":"2015","venue":"Sainbayar Sukhbaatar","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2013] Mengqiu Wang, Wanxiang Che, and Christopher D Manning. Effective bilingual constraints for semi-supervised learning of named entity recognizers. In AAAI. Citeseer, 2013.","title":"Effective bilingual constraints for semi-supervised learning of named entity recognizers","authors":"Wang","year":"2013","venue":"AAAI. Citeseer","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yang et al., 2017] Zhilin Yang, Ruslan Salakhutdinov, and William W Cohen. Transfer learning for sequence tagging with hierarchical recurrent networks. CoRR, 2017.","title":"Transfer learning for sequence tagging with hierarchical recurrent networks","authors":"Zhilin Yang, Ruslan Salakhutdinov, William W Cohen","year":"2017","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2016] Boliang Zhang, Xiaoman Pan, Tianlu Wang, Ashish Vaswani, Heng Ji, Kevin Knight, and Daniel Marcu. Name tagging for low-resource incident languages based on expectation-driven learning. In Proceedings of NAACL-HLT, pages 249-259, 2016.","title":"Name tagging for low-resource incident languages based on expectation-driven learning","authors":"Zhang","year":"2016","venue":"Proceedings of NAACL-HLT","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}