{"type":"ScholarlyArticle","title":"Learning Task-Specific Representation for Novel Words in Sequence Labeling","authors":[{"name":"Minlong Peng","slug":"minlong-peng","given":"Minlong","family":"Peng","affiliation":"School of Computer Science , Fudan University , Shanghai , China"},{"name":"Qi Zhang","slug":"qi-zhang","given":"Qi","family":"Zhang","affiliation":"School of Computer Science , Fudan University , Shanghai , China"},{"name":"Xiaoyu Xing","slug":"xiaoyu-xing","given":"Xiaoyu","family":"Xing","affiliation":"School of Computer Science , Fudan University , Shanghai , China"},{"name":"Tao Gui","slug":"tao-gui","given":"Tao","family":"Gui","affiliation":"School of Computer Science , Fudan University , Shanghai , China"},{"name":"Jinlan Fu","slug":"jinlan-fu","given":"Jinlan","family":"Fu","affiliation":"School of Computer Science , Fudan University , Shanghai , China"},{"name":"Xuanjing Huang","slug":"xuanjing-huang","given":"Xuanjing","family":"Huang","affiliation":"School of Computer Science , Fudan University , Shanghai , China"}],"abstract":"Word representation is a key component in neural-network-based sequence labeling systems. However, representations of unseen or rare words trained on the end task are usually poor for appreciable performance. This is commonly referred to as the out-of-vocabulary (OOV) problem. In this work, we address the OOV problem in sequence labeling using only training data of the task. To this end, we propose a novel method to predict representations for OOV words from their surface-forms (e.g., character sequence) and contexts. The method is specifically designed to avoid the error propagation problem suffered by existing approaches in the same paradigm. To evaluate its effectiveness, we performed extensive empirical studies on four part-of-speech tagging (POS) tasks and four named entity recognition (NER) tasks. Experimental results show that the proposed method can achieve better or competitive performance on the OOV problem compared with existing state-of-the-art methods.","doi":"10.24963/ijcai.2019/715","url":"https://ijcai.org/proceedings/2019/715","pdf":"https://ijcai.org/proceedings/2019/0715.pdf","markdown":"https://ijcai.org/proceedings/2019/715.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/715","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":715,"pages":{"first":5146,"last":5152},"keywords":["Natural Language Processing: Named Entities","Natural Language Processing: Tagging, chunking, and parsing","Natural Language Processing: Embeddings"],"videos":[],"references":[{"position":1,"raw":"[Akbik et al., 2018] Alan Akbik, Duncan Blythe, and Roland Vollgraf. Contextual string embeddings for se- quence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638- 1649, 2018.","title":"Contextual string embeddings for sequence labeling","authors":"Alan Akbik, Duncan Blythe, Roland Vollgraf","year":"2018","venue":"Proceedings of the 27th International Conference on Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bojarski et al., 2016] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self- driving cars. arXiv preprint arXiv:1604.07316, 2016.","title":"End to end learning for selfdriving cars","authors":"Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Caselli et al., 2015] Tommaso Caselli, Piek Vossen, Marieke van Erp, Antske Fokkens, Filip Ilievski, Rubén Izquierdo, Minh Le, Roser Morante, and Marten Postma. When it's all piling up: investigating error propagation in an nlp pipeline. In WNACP@ NLDB, 2015.","title":"When it's all piling up: investigating error propagation in an nlp pipeline","authors":"Tommaso Caselli, Piek Vossen, Marieke Van Erp, Antske Fokkens, Filip Ilievski, Rubén Izquierdo, Minh Le, Roser Morante, Marten Postma","year":"2015","venue":"WNACP@ NLDB","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Delėger et al., 2016] Louise Delėger, Robert Bossy, Es- telle Chaix, Mouhamadou Ba, Arnaud Ferrė, Philippe Bessieres, and Claire Nėdellec. Overview of the bacteria biotope task at bionlp shared task 2016. In Proceedings of the 4th BioNLP Shared Task Workshop, pages 12-22, 2016.","title":"Overview of the bacteria biotope task at bionlp shared task 2016","authors":"Louise Delėger, Robert Bossy, Estelle Chaix, Mouhamadou Ba, Arnaud Ferrė, Philippe Bessieres, Claire Nėdellec","year":"2016","venue":"Proceedings of the 4th BioNLP Shared Task Workshop","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Derczynski et al., 2013] Leon Derczynski, Alan Ritter, Sam Clark, and Kalina Bontcheva. Twitter part-of-speech tagging for all: Overcoming sparse and noisy data. In RANLP, pages 198-206, 2013.","title":"Twitter part-of-speech tagging for all: Overcoming sparse and noisy data","authors":"Leon Derczynski, Alan Ritter, Sam Clark, Kalina Bontcheva","year":"2013","venue":"RANLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gui et al., 2017] Tao Gui, Qi Zhang, Haoran Huang, Minlong Peng, and Xuanjing Huang. Part-of-speech tagging for twitter with adversarial neural networks. In EMNLP, pages 2411-2420, 2017.","title":"Part-of-speech tagging for twitter with adversarial neural networks","authors":"Tao Gui, Qi Zhang, Haoran Huang, Minlong Peng, Xuanjing Huang","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2015] Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991, 2015.","title":"Bidirectional lstm-crf models for sequence tagging","authors":"Zhiheng Huang, Wei Xu, Kai Yu","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Khodak et al., 2018] Mikhail Khodak, Nikunj Saunshi, Yingyu Liang, Tengyu Ma, Brandon Stewart, and Sanjeev Arora. A la carte embedding: Cheap but effective induction of semantic feature vectors. In ACL (Long Papers), volume 1, pages 12-22, 2018.","title":"A la carte embedding: Cheap but effective induction of semantic feature vectors","authors":"Mikhail Khodak, Nikunj Saunshi, Yingyu Liang, Tengyu Ma, Brandon Stewart, Sanjeev Arora","year":"2018","venue":"ACL (Long Papers)","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. EMNLP, 2014.","title":"Convolutional neural networks for sentence classification","authors":"Yoon Kim","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kinga and Adam, 2015] D Kinga and J Ba Adam. A method for stochastic optimization. In ICLR, volume 5, 2015.","title":"A method for stochastic optimization","authors":"D Kinga, J Ba, Adam","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lafferty et al., 2001] John Lafferty, Andrew McCallum, and Fernando CN Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. 2001.","title":"Conditional random fields: Probabilistic models for segmenting and labeling sequence data","authors":"Lafferty","year":"2001","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lazaridou et al., 2017] Angeliki Lazaridou, Marco Marelli, and Marco Baroni. Multimodal word meaning induction from minimal exposure to natural text. Cognitive science, 41:677-705, 2017.","title":"Multimodal word meaning induction from minimal exposure to natural text","authors":"Angeliki Lazaridou, Marco Marelli, Marco Baroni","year":"2017","venue":"Cognitive science","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ling et al., 2015] Wang Ling, Chris Dyer, Alan W Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. Finding function in form: Compositional character models for open vocabulary word representation. In EMNLP, pages 1520-1530, 2015.","title":"Finding function in form: Compositional character models for open vocabulary word representation","authors":"Wang Ling, Chris Dyer, Alan W Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, Tiago Luis","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ma and Hovy, 2016] Xuezhe Ma and Eduard Hovy. End-to- end sequence labeling via bi-directional lstm-cnns-crf. In ACL, volume 1, pages 1064-1074, 2016.","title":"Xuezhe Ma and Eduard Hovy. End-toend sequence labeling via bi-directional lstm-cnns-crf","authors":"Hovy Ma","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Madhyastha et al., 2016] Pranava Swaroop Madhyastha, Mohit Bansal, Kevin Gimpel, and Karen Livescu. Mapping unseen words to task-trained embedding spaces. In Proceedings of the 1st Workshop on Representation Learning for NLP, pages 100-110, 2016.","title":"Mapping unseen words to task-trained embedding spaces","authors":"Pranava Swaroop Madhyastha, Mohit Bansal, Kevin Gimpel, Karen Livescu","year":"2016","venue":"Proceedings of the 1st Workshop on Representation Learning for NLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Marcus et al., 1993] Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313-330, 1993.","title":"Building a large annotated corpus of english: The penn treebank","authors":"Mary Mitchell P Marcus, Ann Marcinkiewicz, Beatrice Santorini","year":"1993","venue":"Computational linguistics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Peters et al., 2018] Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018.","title":"Deep contextualized word representations","authors":"Mark Matthew E Peters, Mohit Neumann, Matt Iyyer, Christopher Gardner, Kenton Clark, Luke Lee, Zettlemoyer","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pinter et al., 2017] Yuval Pinter, Robert Guthrie, and Jacob Eisenstein. Mimicking word embeddings using subword rnns. In EMNLP, pages 102-112, 2017.","title":"Mimicking word embeddings using subword rnns","authors":"Pinter","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sang, 2002] Erik F. Tjong Kim Sang. Introduction to the conll-2002 shared task: Language-independent named entity recognition. Computer Science, pages 142-147, 2002.","title":"Introduction to the conll-2002 shared task: Language-independent named entity recognition","authors":"Erik F Tjong, Kim Sang","year":"2002","venue":"Computer Science","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Schick and Schütze, 2018] Timo Schick and Hinrich Schütze. Learning semantic representations for novel words: Leveraging both form and context. arXiv preprint arXiv:1811.03866, 2018.","title":"Learning semantic representations for novel words: Leveraging both form and context","authors":"Schütze Schick, Timo Schick, Hinrich Schütze","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tjong Kim Sang and De Meulder, 2003] Erik F Tjong Kim Sang and Fien De Meulder. Introduction to the conll- 2003 shared task: Language-independent named entity recognition. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003- Volume 4, pages 142-147. Association for Computational Linguistics, 2003.","title":"Introduction to the conll-2003 shared task: Language-independent named entity recognition","authors":"Tjong Kim, Sang, De Meulder, ; ] Erik, F Tjong, Kim Sang, Fien De, Meulder","year":"2003","venue":"Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Verginica Barbu Mititelu, 2016] Cenel-Augusto Perez Radu Ion Radu Simionescu Martin Popel Verginica Barbu Mititelu, Elena Irimia. The romanian treebank annotated according to universal dependencies. In Proceedings of The Tenth International Conference on Natural Language Processing (HrTAL2016), 2016.","title":"The romanian treebank annotated according to universal dependencies","authors":"Barbu Verginica, ; -Augusto Mititelu, Radu Perez, Radu Ion, Martin Popel Simionescu, Barbu Verginica, Elena Mititelu, Irimia","year":"2016","venue":"Proceedings of The Tenth International Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang and Yang, 2018] Yue Zhang and Jie Yang. Chinese ner using lattice lstm. In ACL, volume 1, pages 1554- 1564, 2018.","title":"Chinese ner using lattice lstm","authors":"Yue Zhang, Jie Yang","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2018] Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. Adaptive co-attention network for named entity recognition in tweets. In AAAI, 2018.","title":"Adaptive co-attention network for named entity recognition in tweets","authors":"Zhang","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zipf, 1949] George Kingsley Zipf. Human behavior and the principle of least effort. 1949.","title":"Human behavior and the principle of least effort","authors":"George Kingsley, Zipf","year":"1949","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}