{"type":"ScholarlyArticle","title":"Learning Tag Dependencies for Sequence Tagging","authors":[{"name":"Yuan Zhang","slug":"yuan-zhang","given":"Yuan","family":"Zhang","affiliation":"§ Data Science Lab , JD.com ♦ ADAPT centre , School of Computing , Dublin City University; Data Science Lab , JD.com."},{"name":"Hongshen Chen","slug":"hongshen-chen","given":"Hongshen","family":"Chen","affiliation":"‡ Key Laboratory of Intelligent Information Processing Institute of Computing Technology , Chinese Academy of Sciences"},{"name":"Yihong Zhao","slug":"yihong-zhao","given":"Yihong","family":"Zhao","affiliation":"‡ Key Laboratory of Intelligent Information Processing Institute of Computing Technology , Chinese Academy of Sciences"},{"name":"Qun Liu","slug":"qun-liu","given":"Qun","family":"Liu","affiliation":"University of Chinese Academy of Sciences"},{"name":"Dawei Yin","slug":"dawei-yin","given":"Dawei","family":"Yin","affiliation":"‡ Key Laboratory of Intelligent Information Processing Institute of Computing Technology , Chinese Academy of Sciences"}],"abstract":"Sequence tagging is the basis for multiple applications in natural language processing. Despite successes in learning long term token sequence dependencies with neural network, tag dependencies are rarely considered previously. Sequence tagging actually possesses complex dependencies and interactions among the input tokens and the output tags. We propose a novel multi-channel model, which handles different ranges of token-tag dependencies and their interactions simultaneously. A tag LSTM is augmented to manage the output tag dependencies and word-tag interactions, while three mechanisms are presented to efficiently incorporate token context representation and tag dependency. Extensive experiments on part-of-speech tagging and named entity recognition tasks show that  the proposed model outperforms the BiLSTM-CRF baseline by effectively incorporating the tag dependency feature.","doi":"10.24963/ijcai.2018/637","url":"https://ijcai.org/proceedings/2018/637","pdf":"https://ijcai.org/proceedings/2018/0637.pdf","markdown":"https://ijcai.org/proceedings/2018/637.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/637","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":637,"pages":{"first":4581,"last":4587},"keywords":["Natural Language Processing: Phonology, Morphology, and word segmentation","Natural Language Processing: Named Entities","Natural Language Processing: Tagging, chunking, and parsing"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2015] Xinchi Chen, Xipeng Qiu, Chenxi Zhu, Pengfei Liu, and Xuanjing Huang. Long short-term mem- ory neural networks for chinese word segmentation. In EMNLP, pages 1197-1206, 2015.","title":"Long short-term memory neural networks for chinese word segmentation","authors":"Chen","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2016] Hongshen Chen, Yue Zhang, and Qun Liu. Neural network for heterogeneous annotations. In EMNLP, pages 731-741, 2016.","title":"Neural network for heterogeneous annotations","authors":"Chen","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chiu and Nichols, 2015] Jason PC Chiu and Eric Nichols. Named entity recognition with bidirectional lstm-cnns. arXiv preprint arXiv:1511.08308, 2015.","title":"Named entity recognition with bidirectional lstm-cnns","authors":"Jason Pc Chiu, Eric Nichols","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, pages 1724-1734, 2014.","title":"Learning phrase representations using rnn encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Collins, 2002] Michael Collins. Discriminative training methods for hidden markov models: Theory and experi- ments with perceptron algorithms. In EMNLP, 2002.","title":"Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms","authors":"Michael Collins","year":"2002","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Collobert et al., 2011] Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. JMLR, 12(Aug):2493-2537, 2011.","title":"Natural language processing (almost) from scratch","authors":"Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, Pavel Kuksa","year":"2011","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cuong et al., 2014] Nguyen Viet Cuong, Nan Ye, Wee Sun Lee, and Hai Leong Chieu. Conditional random field with high-order dependencies for sequence labeling and seg- mentation. JMLR, 15(1):981-1009, 2014.","title":"Conditional random field with high-order dependencies for sequence labeling and segmentation","authors":"Nguyen Viet, Cuong, Nan Ye, Wee Sun Lee, Hai Leong Chieu","year":"2014","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Giménez and Marquez, 2004] Jesús Giménez and Lluis Marquez. Svmtool: A general pos tagger generator based on support vector machines. In ICLR, 2004.","title":"Svmtool: A general pos tagger generator based on support vector machines","authors":"Marquez Giménez, Jesús Giménez, Lluis Marquez","year":"2004","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Graves and Schmidhuber, 2005] Alex Graves and Jürgen Schmidhuber. Framewise phoneme classification with bidirectional lstm networks. In IJCNN, 2005.","title":"Framewise phoneme classification with bidirectional lstm networks","authors":"Schmidhuber Graves, Alex Graves, Jürgen Schmidhuber","year":"2005","venue":"IJCNN","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lafferty et al., 2001] John Lafferty, Andrew McCallum, and Fernando CN Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. 2001.","title":"Conditional random fields: Probabilistic models for segmenting and labeling sequence data","authors":"Lafferty","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lample et al., 2016] Guillaume Lample, Miguel Balles- teros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. Neural architectures for named entity recog- nition. In NAACL, pages 260-270, 2016.","title":"Neural architectures for named entity recognition","authors":"Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, Chris Dyer","year":"2016","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin and Wu, 2009] Dekang Lin and Xiaoyun Wu. Phrase clustering for discriminative learning. In ACL, 2009.","title":"Phrase clustering for discriminative learning","authors":"Dekang Lin, Xiaoyun Wu","year":"2009","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2016] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Recurrent neural network for text classification with multi-task learning. In IJCAI, 2016.","title":"Recurrent neural network for text classification with multi-task learning","authors":"Pengfei Liu, Xipeng Qiu, Xuanjing Huang","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/408"},{"position":15,"raw":"[Luo et al., 2015] Gang Luo, Xiaojiang Huang, Chin-Yew Lin, and Zaiqing Nie. Joint named entity recognition and disambiguation. In EMNLP, 2015.","title":"Joint named entity recognition and disambiguation","authors":"Gang Luo, Xiaojiang Huang, Chin-Yew Lin, Zaiqing Nie","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ma and Hovy, 2016] Xuezhe Ma and Eduard Hovy. End-to- end sequence labeling via bi-directional lstm-cnns-crf. In ACL, pages 1064-1074, 2016.","title":"Xuezhe Ma and Eduard Hovy. End-toend sequence labeling via bi-directional lstm-cnns-crf","authors":"Hovy Ma","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ma and Xia, 2014] Xuezhe Ma and Fei Xia. Unsupervised dependency parsing with transferring distribution via par- allel guidance and entropy regularization. In ACL, 2014.","title":"Xuezhe Ma and Fei Xia. Unsupervised dependency parsing with transferring distribution via parallel guidance and entropy regularization","authors":"Xia Ma","year":"2014","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Manning, 2011] Christopher D Manning. Part-of-speech tagging from 97% to 100%: is it time for some linguis- tics? In Cicling, pages 171-189, 2011.","title":"Part-of-speech tagging from 97% to 100%: is it time for some linguistics? In Cicling","authors":"D Christopher, Manning","year":"2011","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Marcus et al., 1993] Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313-330, 1993.","title":"Building a large annotated corpus of english: The penn treebank","authors":"Mary Mitchell P Marcus, Ann Marcinkiewicz, Beatrice Santorini","year":"1993","venue":"Computational linguistics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Passos et al., 2014] Alexandre Passos, Vineet Kumar, and Andrew McCallum. Lexicon infused phrase embeddings for named entity resolution. arXiv:1404.5367, 2014.","title":"Lexicon infused phrase embeddings for named entity resolution","authors":"Alexandre Passos, Vineet Kumar, Andrew Mccallum","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pei et al., 2014] Wenzhe Pei, Tao Ge, and Baobao Chang. Max-margin tensor neural network for chinese word seg- mentation. In ACL, pages 293-303, 2014.","title":"Max-margin tensor neural network for chinese word segmentation","authors":"Pei","year":"2014","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Reimers and Gurevych, 2017] Nils Reimers and Iryna Gurevych. Reporting score distributions makes a differ- ence: Performance study of lstm-networks for sequence tagging. In EMNLP, pages 338-348, 2017.","title":"Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging","authors":"Gurevych Reimers, Nils Reimers, Iryna Gurevych","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Santos and Zadrozny, 2014] Cicero D Santos and Bianca Zadrozny. Learning character-level representations for part-of-speech tagging. In ICML, pages 1818-1826, 2014.","title":"Learning character-level representations for part-of-speech tagging","authors":"Santos, Zadrozny ; Cicero, D Santos, Bianca Zadrozny","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shen et al., 2007] Libin Shen, Giorgio Satta, and Aravind Joshi. Guided learning for bidirectional sequence classifi- cation. In ACL, volume 7, pages 760-767, 2007.","title":"Guided learning for bidirectional sequence classification","authors":"Libin Shen, Giorgio Satta, Aravind Joshi","year":"2007","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Søgaard, 2011] Anders Søgaard. Semisupervised con- densed nearest neighbor for part-of-speech tagging. In ACL, pages 48-52, 2011.","title":"Semisupervised condensed nearest neighbor for part-of-speech tagging","authors":"Anders Søgaard, Søgaard","year":"2011","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sun, 2014] Xu Sun. Structure regularization for structured prediction. In NIPS, pages 2402-2410, 2014.","title":"Structure regularization for structured prediction","authors":"Xu Sun","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NIPS, pages 3104-3112, 2014.","title":"Quoc V Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Toutanova et al., 2003] Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. Feature-rich part-of-speech tagging with a cyclic dependency network. In NAACL, pages 173-180, 2003.","title":"Feature-rich part-of-speech tagging with a cyclic dependency network","authors":"Kristina Toutanova, Dan Klein, Christopher D Manning, Yoram Singer","year":"2003","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Vaswani et al., 2016] Ashish Vaswani, Yonatan Bisk, Kenji Sagae, and Ryan Musa. Supertagging with lstms. In NAACL, 2016.","title":"Supertagging with lstms","authors":"Ashish Vaswani, Yonatan Bisk, Kenji Sagae, Ryan Musa","year":"2016","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2015] Meishan Zhang, Yue Zhang, and Duy- Tin Vo. Neural networks for open domain targeted senti- ment. In EMNLP, pages 612-621, 2015.","title":"Neural networks for open domain targeted sentiment","authors":"Zhang","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zheng et al., 2013] Xiaoqing Zheng, Hanyang Chen, and Tianyu Xu. Deep learning for chinese word segmentation and pos tagging. In EMNLP, pages 647-657, 2013.","title":"Deep learning for chinese word segmentation and pos tagging","authors":"Xiaoqing Zheng, Hanyang Chen, Tianyu Xu","year":"2013","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zheng et al., 2017] Suncong Zheng, Feng Wang, Hongyun Bao, Yuexing Hao, Peng Zhou, and Bo Xu. Joint ex- traction of entities and relations based on a novel tagging scheme. In ACL, 2017.","title":"Joint extraction of entities and relations based on a novel tagging scheme","authors":"Zheng","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}