{"type":"ScholarlyArticle","title":"Coreference Aware Representation Learning for Neural Named Entity Recognition","authors":[{"name":"Zeyu Dai","slug":"zeyu-dai","given":"Zeyu","family":"Dai","affiliation":"Cognitive Computing Lab , Baidu Research USA"},{"name":"Hongliang Fei","slug":"hongliang-fei","given":"Hongliang","family":"Fei","affiliation":"Cognitive Computing Lab , Baidu Research USA"},{"name":"Ping Li","slug":"ping-li","given":"Ping","family":"Li","affiliation":"Cognitive Computing Lab , Baidu Research USA"}],"abstract":"Recent neural network models have achieved state-of-the-art performance on the task of named entity recognition (NER). \r\nHowever, previous neural network models typically treat the input sentences as a linear sequence of words but ignore rich structural information, such as the coreference relations among non-adjacent words, phrases or entities. In this paper, we propose a novel approach to learn coreference-aware word representations for the NER task at the document level. In particular, we enrich the well-known neural architecture ``CNN-BiLSTM-CRF'' with a coreference layer on top of the BiLSTM layer to incorporate coreferential relations. Furthermore, we introduce the coreference regularization to ensure the coreferential entities to share similar representations and consistent predictions within the same coreference cluster. Our proposed model achieves new state-of-the-art performance on two NER benchmarks: CoNLL-2003 and OntoNotes v5.0. More importantly, we demonstrate that our framework does not rely on gold coreference knowledge, and can still work well even when the coreferential relations are generated by a third-party toolkit.","doi":"10.24963/ijcai.2019/687","url":"https://ijcai.org/proceedings/2019/687","pdf":"https://ijcai.org/proceedings/2019/0687.pdf","markdown":"https://ijcai.org/proceedings/2019/687.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/687","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":687,"pages":{"first":4946,"last":4953},"keywords":["Natural Language Processing: Natural Language Processing","Natural Language Processing: Tagging, chunking, and parsing","Natural Language Processing: Information Extraction","Natural Language Processing: Named Entities"],"videos":[],"references":[{"position":1,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chiu and Nichols, 2016] Jason Chiu and Eric Nichols. Named en- tity recognition with bidirectional LSTM-CNNs. TACL, 4:357- 370, 2016.","title":"Named entity recognition with bidirectional LSTM-CNNs","authors":"Nichols ; Chiu, Jason Chiu, Eric Nichols","year":"2016","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Clark et al., 2018] Kevin Clark, Minh-Thang Luong, Christopher Manning, and Quoc Le. Semi-supervised sequence modeling with cross-view training. In EMNLP, pages 1914-1925, 2018.","title":"Semi-supervised sequence modeling with cross-view training","authors":"Kevin Clark, Minh-Thang Luong, Christopher Manning, Quoc Le","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Collobert et al., 2011] Ronan Collobert, Jason Weston, Léon Bot- tou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Nat- ural language processing (almost) from scratch. Journal of Ma- chine Learning Research, 12:2493-2537, 2011.","title":"Natural language processing (almost) from scratch","authors":"Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, Pavel Kuksa","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dhingra et al., 2017] Bhuwan Dhingra, Zhilin Yang, William Co- hen, and Ruslan Salakhutdinov. Linguistic knowledge as memory for recurrent neural networks. arXiv preprint arXiv:1703.02620, 2017.","title":"Linguistic knowledge as memory for recurrent neural networks","authors":"Bhuwan Dhingra, Zhilin Yang, William Cohen, Ruslan Salakhutdinov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dhingra et al., 2018] Bhuwan Dhingra, Qiao Jin, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. Neural models for reasoning over multiple mentions using coreference. In NAACL- HLT, pages 42-48, 2018.","title":"Neural models for reasoning over multiple mentions using coreference","authors":"Bhuwan Dhingra, Qiao Jin, Zhilin Yang, William Cohen, Ruslan Salakhutdinov","year":"2018","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Dong et al., 2014] Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. Knowledge vault: a web-scale approach to probabilistic knowledge fusion. In ACM SIGKDD, pages 601-610, 2014.","title":"Knowledge vault: a web-scale approach to probabilistic knowledge fusion","authors":"Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, Wei Zhang","year":"2014","venue":"ACM SIGKDD","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dos Santos and Zadrozny, 2014] Cicero Nogueira Dos Santos and Bianca Zadrozny. Learning character-level representations for part-of-speech tagging. In ICML, pages 1818-1826, 2014.","title":"Cicero Nogueira Dos Santos and Bianca Zadrozny. Learning character-level representations for part-of-speech tagging","authors":"Dos Santos, Zadrozny","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Durrett and Klein, 2014] Greg Durrett and Dan Klein. A joint model for entity analysis: Coreference, typing, and linking. TACL, 2:477-490, 2014.","title":"A joint model for entity analysis: Coreference, typing, and linking","authors":"Klein Durrett, Greg Durrett, Dan Klein","year":"2014","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Fei et al., 2019] Hongliang Fei, Xu Li, Dingcheng Li, and Ping Li. End-to-end deep reinforcement learning based coreference reso- lution. In ACL, 2019.","title":"End-to-end deep reinforcement learning based coreference resolution","authors":"Hongliang Fei, Xu Li, Dingcheng Li, Ping Li","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ghaddar and Langlais, 2018] Abbas Ghaddar and Philippe Langlais. Robust lexical features for improved neural network named-entity recognition. In COLING, pages 1896-1907, 2018.","title":"Robust lexical features for improved neural network named-entity recognition","authors":"Langlais Ghaddar, Abbas Ghaddar, Philippe Langlais","year":"2018","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Huang et al., 2015] Zhiheng Huang, Wei Xu, and Kai Yu. Bidirec- tional LSTM-CRF models for sequence tagging. arXiv preprint arXiv:1508.01991, 2015.","title":"Bidirectional LSTM-CRF models for sequence tagging","authors":"Zhiheng Huang, Wei Xu, Kai Yu","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jie et al., 2017] Zhanming Jie, Aldrian Obaja Muis, and Wei Lu. Efficient dependency-guided named entity recognition. In AAAI, pages 3457-3465, 2017.","title":"Efficient dependency-guided named entity recognition","authors":"Jie","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lample et al., 2016] Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. Neural architectures for named entity recognition. In NAACL- HLT, pages 260-270, 2016.","title":"Neural architectures for named entity recognition","authors":"Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, Chris Dyer","year":"2016","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lee et al., 2018] Kenton Lee, Luheng He, and Luke Zettlemoyer. Higher-order coreference resolution with coarse-to-fine infer- ence. In NAACL-HLT, pages 687-692, 2018.","title":"Higher-order coreference resolution with coarse-to-fine inference","authors":"Kenton Lee, Luheng He, Luke Zettlemoyer","year":"2018","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li and Ji, 2014] Qi Li and Heng Ji. Incremental joint extraction of entity mentions and relations. In ACL, pages 402-412, 2014.","title":"Incremental joint extraction of entity mentions and relations","authors":"Ji ; ] Qi Li, Heng Li, Ji","year":"2014","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2017] Peng-Hsuan Li, Ruo-Ping Dong, Yu-Siang Wang, Ju-Chieh Chou, and Wei-Yun Ma. Leveraging linguistic struc- tures for named entity recognition with bidirectional recursive neural networks. In EMNLP, pages 2664-2669, 2017.","title":"Leveraging linguistic structures for named entity recognition with bidirectional recursive neural networks","authors":"Peng-Hsuan, Ruo-Ping Li, Yu-Siang Dong, Ju-Chieh Wang, Wei-Yun Chou, Ma","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Liu et al., 2018] Liyuan Liu, Jingbo Shang, Xiang Ren, Frank Fangzheng Xu, Huan Gui, Jian Peng, and Jiawei Han. Empower sequence labeling with task-aware neural language model. In AAAI, 2018.","title":"Empower sequence labeling with task-aware neural language model","authors":"Liyuan Liu, Jingbo Shang, Xiang Ren, Frank Fangzheng Xu, Huan Gui, Jian Peng, Jiawei Han","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Luan et al., 2018] Yi Luan, Luheng He, Mari Ostendorf, and Han- naneh Hajishirzi. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In EMNLP, pages 3219-3232, 2018.","title":"Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction","authors":"Yi Luan, Luheng He, Mari Ostendorf, Hannaneh Hajishirzi","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Luo et al., 2015] Gang Luo, Xiaojiang Huang, Chin-Yew Lin, and Zaiqing Nie. Joint entity recognition and disambiguation. In EMNLP, pages 879-888, 2015.","title":"Joint entity recognition and disambiguation","authors":"Gang Luo, Xiaojiang Huang, Chin-Yew Lin, Zaiqing Nie","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ma and Hovy, 2016] Xuezhe Ma and Eduard Hovy. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In ACL, pages 1064-1074, 2016.","title":"Xuezhe Ma and Eduard Hovy. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF","authors":"Hovy Ma","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Manning et al., 2014] Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. The stanford corenlp natural language processing toolkit. In ACL: system demonstrations, pages 55-60, 2014.","title":"The stanford corenlp natural language processing toolkit","authors":"Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, David Mcclosky","year":"2014","venue":"ACL: system demonstrations","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Peng et al., 2017] Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, and Wen-tau Yih. Cross-sentence n-ary rela- tion extraction with graph LSTMs. TACL, 5(1):101-115, 2017.","title":"Cross-sentence n-ary relation extraction with graph LSTMs","authors":"Peng","year":"2017","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Peters et al., 2017] Matthew Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. Semi-supervised sequence tag- ging with bidirectional language models. In ACL, pages 1756- 1765, 2017.","title":"Semi-supervised sequence tagging with bidirectional language models","authors":"Matthew Peters, Waleed Ammar, Chandra Bhagavatula, Russell Power","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Peters et al., 2018] Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettle- moyer. Deep contextualized word representations. In NAACL- HLT, pages 2227-2237, 2018.","title":"Deep contextualized word representations","authors":"Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer","year":"2018","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Pradhan et al., 2011] Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. Conll-2011 shared task: Modeling unrestricted coreference in ontonotes. In CoNLL: Shared Task, pages 1-27, 2011.","title":"Conll-2011 shared task: Modeling unrestricted coreference in ontonotes","authors":"Lance Sameer Pradhan, Mitchell Ramshaw, Martha Marcus, Ralph Palmer, Nianwen Weischedel, Xue","year":"2011","venue":"CoNLL: Shared Task","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Ratinov and Roth, 2009] Lev Ratinov and Dan Roth. Design chal- lenges and misconceptions in named entity recognition. In Pro- ceedings of the thirteenth conference on computational natural language learning, pages 147-155. Association for Computa- tional Linguistics, 2009.","title":"Lev Ratinov and Dan Roth. Design challenges and misconceptions in named entity recognition","authors":"Roth Ratinov","year":"2009","venue":"Proceedings of the thirteenth conference on computational natural language learning","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sang and De Meulder, 2003] Erik Tjong Kim Sang and Fien De Meulder. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In NAACL- HLT, 2003.","title":"Introduction to the conll-2003 shared task: Language-independent named entity recognition","authors":"Sang, De Meulder, Erik Tjong, Kim Sang, Fien De, Meulder","year":"2003","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Shen et al., 2018] Yanyao Shen, Hyokun Yun, Zachary Lipton, Yakov Kronrod, and Animashree Anandkumar. Deep active learning for named entity recognition. In ICLR, 2018.","title":"Deep active learning for named entity recognition","authors":"Shen","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Singh et al., 2013] Sameer Singh, Sebastian Riedel, Brian Martin, Jiaping Zheng, and Andrew McCallum. Joint inference of enti- ties, relations, and coreference. In AKBC@CIKM 2013, 2013.","title":"Joint inference of entities, relations, and coreference","authors":"Sameer Singh, Sebastian Riedel, Brian Martin, Jiaping Zheng, Andrew Mccallum","year":"2013","venue":"AKBC@CIKM 2013","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Socher et al., 2013] Richard Socher, Danqi Chen, Christopher Manning, and Andrew Ng. Reasoning with neural tensor net- works for knowledge base completion. In NIPS, 2013.","title":"Reasoning with neural tensor networks for knowledge base completion","authors":"Richard Socher, Danqi Chen, Christopher Manning, Andrew Ng","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Strubell et al., 2017] Emma Strubell, Patrick Verga, David Be- langer, and Andrew McCallum. Fast and accurate entity recogni- tion with iterated dilated convolutions. In EMNLP, 2017.","title":"Fast and accurate entity recognition with iterated dilated convolutions","authors":"Emma Strubell, Patrick Verga, David Belanger, Andrew Mccallum","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Swayamdipta et al., 2018] Swabha Swayamdipta, Ankur Parikh, and Tom Kwiatkowski. Multi-mention learning for reading com- prehension with neural cascades. ICLR, 2018.","title":"Multi-mention learning for reading comprehension with neural cascades","authors":"Swayamdipta","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Tran et al., 2017] Quan Tran, Andrew MacKinlay, and Antonio Ji- meno Yepes. Named entity recognition with stack residual LSTM and trainable bias decoding. In IJCNLP, pages 566-575, 2017.","title":"Named entity recognition with stack residual LSTM and trainable bias decoding","authors":"Quan Tran, Andrew Mackinlay, Antonio Jimeno, Yepes","year":"2017","venue":"IJCNLP","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Yang and Mitchell, 2017] Bishan Yang and Tom Mitchell. Lever- aging knowledge bases in LSTMs for improving machine read- ing. In ACL, pages 1436-1446, 2017.","title":"Leveraging knowledge bases in LSTMs for improving machine reading","authors":"Yang, Mitchell ;, Bishan Yang, Tom Mitchell","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Yang and Zhang, 2018] Jie Yang and Yue Zhang. Ncrf++: An open-source neural sequence labeling toolkit. In ACL, 2018.","title":"Ncrf++: An open-source neural sequence labeling toolkit","authors":"Yang, Zhang ; ] Jie Yang, Yue Zhang","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Yang et al., 2017] Zhilin Yang, Ruslan Salakhutdinov, and William Cohen. Transfer learning for sequence tagging with hierarchical recurrent networks. In ICLR, 2017.","title":"Transfer learning for sequence tagging with hierarchical recurrent networks","authors":"Zhilin Yang, Ruslan Salakhutdinov, William Cohen","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Yang et al., 2018] Jie Yang, Shuailong Liang, and Yue Zhang. De- sign challenges and misconceptions in neural sequence labeling. In COLING, 2018.","title":"Design challenges and misconceptions in neural sequence labeling","authors":"Jie Yang, Shuailong Liang, Yue Zhang","year":"2018","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Ye and Ling, 2018] Zhixiu Ye and Zhen-Hua Ling. Hybrid semi- markov CRF for neural sequence labeling. In ACL, pages 235- 240, 2018.","title":"Zhixiu Ye and Zhen-Hua Ling. Hybrid semimarkov CRF for neural sequence labeling","authors":"Ling Ye","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}