Proceedings

Joint POS Tagging and Text Normalization for Informal Text

Chen Li, Yang Liu

Download BibTeX Cite

Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence. Main Track. Pages 1263-1268.

Abstract

Text normalization and part-of-speech (POS) tagging for social media data have been investigated recently, however, prior work has treated them separately. In this paper, we propose a joint Viterbi decoding process to determine each token’s POS tag and non-standard token’s correct form at the same time. In order to evaluate our approach, we create two new data sets with POS tag labels and non-standard tokens' correct forms. This is the first data set with such annotation. The experiment results demonstrate the effect of non-standard words on POS tagging, and also show that our proposed methods perform better than the state-of-the-art systems in both POS tagging and normalization.

Chen Li, and Yang Liu. 2015. Joint POS Tagging and Text Normalization for Informal Text. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-15. IJCAI, 1263–1268. https://ijcai.org/proceedings/2015/182
Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards