{"type":"ScholarlyArticle","title":"Leveraging Document-Level Label Consistency for Named Entity Recognition","authors":[{"name":"Tao Gui","slug":"tao-gui","given":"Tao","family":"Gui","affiliation":"School of Computer Science , Fudan University , Shanghai , China"},{"name":"Jiacheng Ye","slug":"jiacheng-ye","given":"Jiacheng","family":"Ye","affiliation":"School of Computer Science , Fudan University , Shanghai , China"},{"name":"Qi Zhang","slug":"qi-zhang","given":"Qi","family":"Zhang","affiliation":"School of Computer Science , Fudan University , Shanghai , China"},{"name":"Yaqian Zhou","slug":"yaqian-zhou","given":"Yaqian","family":"Zhou","affiliation":"School of Computer Science , Fudan University , Shanghai , China"},{"name":"Yeyun Gong","slug":"yeyun-gong","given":"Yeyun","family":"Gong","affiliation":"Microsoft Research Asia"},{"name":"Xuanjing Huang","slug":"xuanjing-huang","given":"Xuanjing","family":"Huang","affiliation":"School of Computer Science , Fudan University , Shanghai , China"}],"abstract":"Document-level label consistency is an effective indicator that different occurrences of a particular token sequence are very likely to have the same entity types. Previous work focused on better context representations and used the CRF for label decoding. However, CRF-based methods are inadequate for modeling document-level label consistency. This work introduces a novel two-stage label refinement approach to handle document-level label consistency, where a key-value memory network is first used to record draft labels predicted by the base model, and then a multi-channel Transformer makes refinements on these draft predictions based on the explicit co-occurrence relationship derived from the memory network. In addition, in order to mitigate the side effects of incorrect draft labels, Bayesian neural networks are used to indicate the labels with a high probability of being wrong, which can greatly assist in preventing the incorrect refinement of correct draft labels. The experimental results on three named entity recognition benchmarks demonstrated that the proposed method significantly outperformed the state-of-the-art methods.","doi":"10.24963/ijcai.2020/550","url":"https://ijcai.org/proceedings/2020/550","pdf":"https://ijcai.org/proceedings/2020/0550.pdf","markdown":"https://ijcai.org/proceedings/2020/550.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/550","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":550,"pages":{"first":3976,"last":3982},"keywords":["Natural Language Processing: Named Entities","Natural Language Processing: Tagging, chunking, and parsing"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24887/file_24887.m3u8","length_sec":292,"provider":"hls"}],"references":[{"position":1,"raw":"[Chen et al., 2019] Hui Chen, Zijia Lin, Guiguang Ding, Jian-Guang Lou, Yusen Zhang, and Börje F. Karlsson. Grn: Gated relation network to enhance convolutional neural network for named entity recognition. In AAAI, 2019.","title":"Grn: Gated relation network to enhance convolutional neural network for named entity recognition","authors":"Hui Chen, Zijia Lin, Guiguang Ding, Jian-Guang Lou, Yusen Zhang, Börje F Karlsson","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chiu and Nichols, 2016] Jason PC Chiu and Eric Nichols. Named entity recognition with bidirectional lstm-cnns. TACL, 4:357-370, 2016.","title":"Named entity recognition with bidirectional lstm-cnns","authors":"Jason Pc Chiu, Eric Nichols","year":"2016","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cui and Zhang, 2019] Leyang Cui and Yue Zhang. Hierarchically-refined label attention network for sequence labeling. In EMNLP-IJCNLP, 2019.","title":"Hierarchically-refined label attention network for sequence labeling","authors":"Zhang Cui, Leyang Cui, Yue Zhang","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dai et al., 2019] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. In ACL, 2019.","title":"Transformer-XL: Attentive language models beyond a fixed-length context","authors":"Dai","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL, pages 4171-4186, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gal and Ghahramani, 2016a] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML, pages 1050-1059, 2016.","title":"Dropout as a bayesian approximation: Representing model uncertainty in deep learning","authors":"Ghahramani Gal, Yarin Gal, Zoubin Ghahramani","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gal and Ghahramani, 2016b] Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. In NeurIPS, 2016.","title":"A theoretically grounded application of dropout in recurrent neural networks","authors":"Ghahramani Gal, Yarin Gal, Zoubin Ghahramani","year":"2016","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2019] Anwen Hu, Zhicheng Dou, Jian-Yun Nie, and Ji-Rong Wen. Leveraging multi-token entities in document-level named entity recognition. In AAAI, 2019.","title":"Leveraging multi-token entities in document-level named entity recognition","authors":"Anwen Hu, Zhicheng Dou, Jian-Yun Nie, Ji-Rong Wen","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kendall and Gal, 2017] Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In NeurIPS, pages 5574-5584, 2017.","title":"What uncertainties do we need in bayesian deep learning for computer vision?","authors":"Alex Kendall, Yarin Gal","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krallinger et al., 2015] Martin Krallinger, Florian Leitner, Obdulia Rabal, Miguel Vazquez, Julen Oyarzabal, and Alfonso Valencia. Chemdner: The drugs and chemical names extraction challenge. Journal of cheminformatics, 7(1):S1, 2015.","title":"Chemdner: The drugs and chemical names extraction challenge","authors":"Martin Krallinger, Florian Leitner, Obdulia Rabal, Miguel Vazquez, Julen Oyarzabal, Alfonso Valencia","year":"2015","venue":"Journal of cheminformatics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Krishnan and Manning, 2006] Vijay Krishnan and Christo- pher D Manning. An effective two-stage model for exploiting non-local dependencies in named entity recognition. In ACL, pages 1121-1128, 2006.","title":"An effective two-stage model for exploiting non-local dependencies in named entity recognition","authors":"Vijay Krishnan, Christopher D Manning","year":"2006","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lample et al., 2016] Guillaume Lample, Miguel Balles- teros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. Neural architectures for named entity recognition. In NAACL, pages 260-270, 2016.","title":"Neural architectures for named entity recognition","authors":"Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, Chris Dyer","year":"2016","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2019] Yijin Liu, Fandong Meng, Jinchao Zhang, Jinan Xu, Yufeng Chen, and Jie Zhou. Gcdt: A global context enhanced deep transition architecture for sequence labeling. arXiv preprint arXiv:1906.02437, 2019.","title":"Gcdt: A global context enhanced deep transition architecture for sequence labeling","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Luo et al., 2020] Ying Luo, Fengshun Xiao, and Hai Zhao. Hierarchical contextualized representation for named entity recognition. In AAAI, 2020.","title":"Hierarchical contextualized representation for named entity recognition","authors":"Ying Luo, Fengshun Xiao, Hai Zhao","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ma and Hovy, 2016] Xuezhe Ma and Eduard Hovy. End-to- end sequence labeling via bi-directional lstm-cnns-crf. In ACL, pages 1064-1074, 2016.","title":"Xuezhe Ma and Eduard Hovy. End-toend sequence labeling via bi-directional lstm-cnns-crf","authors":"Hovy Ma","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. In NeurIPS. 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Miller et al., 2016] Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. Key-value memory networks for directly reading documents. In EMNLP, pages 1400-1409, 2016.","title":"Key-value memory networks for directly reading documents","authors":"Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein, Antoine Karimi, Jason Bordes, Weston","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Peters et al., 2018] Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL-HLT, pages 2227-2237, 2018.","title":"Deep contextualized word representations","authors":"Mark Matthew E Peters, Mohit Neumann, Matt Iyyer, Christopher Gardner, Kenton Clark, Luke Lee, Zettlemoyer","year":"2018","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Qian et al., 2019] Yujie Qian, Enrico Santus, Zhijing Jin, Jiang Guo, and Regina Barzilay. Graphie: A graph-based framework for information extraction. In NAACL-HIT, pages 751-761, 2019.","title":"Graphie: A graph-based framework for information extraction","authors":"Yujie Qian, Enrico Santus, Zhijing Jin, Jiang Guo, Regina Barzilay","year":"2019","venue":"NAACL-HIT","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Strubell et al., 2017] Emma Strubell, Patrick Verga, David Belanger, and Andrew McCallum. Fast and accurate entity recognition with iterated dilated convolutions. In EMNLP, 2017.","title":"Fast and accurate entity recognition with iterated dilated convolutions","authors":"Emma Strubell, Patrick Verga, David Belanger, Andrew Mccallum","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tjong Kim Sang and De Meulder, 2003] Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In NAACL-HLT, pages 142-147, 2003.","title":"Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition","authors":"Tjong Kim, Sang, De Meulder, ; ] Erik, F Tjong, Kim Sang, Fien De, Meulder","year":"2003","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Weischedel et al., 2013] Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia, PA, 23, 2013.","title":"Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium","authors":"Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ye and Ling, 2018] Zhixiu Ye and Zhen-Hua Ling. Hybrid semi-markov crf for neural sequence labeling. In ACL, pages 235-240, 2018.","title":"Zhixiu Ye and Zhen-Hua Ling. Hybrid semi-markov crf for neural sequence labeling","authors":"Ling Ye","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2018a] Boliang Zhang, Spencer Whitehead, Lifu Huang, and Heng Ji. Global attention for name tagging. CoNLL, 2018.","title":"Global attention for name tagging","authors":"Zhang","year":"2018","venue":"CoNLL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2018b] Yuan Zhang, Hongshen Chen, Yihong Zhao, Qun Liu, and Dawei Yin. Learning tag dependencies for sequence tagging. In IJCAI, pages 4581-4587, 2018.","title":"Learning tag dependencies for sequence tagging","authors":"Zhang","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/637"},{"position":28,"raw":"[Zhang et al., 2018c] Yue Zhang, Qi Liu, and Linfeng Song. Sentence-state lstm for text representation. In ACL, 2018.","title":"Sentence-state lstm for text representation","authors":"Zhang","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}