{"type":"ScholarlyArticle","title":"Combining Knowledge with Deep Convolutional Neural Networks for Short Text Classification","authors":[{"name":"Jin Wang","slug":"jin-wang","given":"Jin","family":"Wang","affiliation":"† Computer Science Department , University of California , Los Angeles ."},{"name":"Zhongyuan Wang","slug":"zhongyuan-wang","given":"Zhongyuan","family":"Wang","affiliation":"‡ Facebook Inc. Microsoft Research , Beijing , China."},{"name":"Dawei Zhang","slug":"dawei-zhang","given":"Dawei","family":"Zhang","affiliation":""},{"name":"Jun Yan","slug":"jun-yan","given":"Jun","family":"Yan","affiliation":""}],"abstract":"Text classification is a fundamental task in NLP applications. Most existing work relied on either explicit or implicit text representation to address this problem. While these techniques work well for sentences, they can not easily be applied to short text because of its shortness and sparsity. In this paper, we propose a framework based on convolutional neural networks that combines explicit and implicit representations of short text for classification. We first conceptualize a short text as a set of relevant concepts using a large taxonomy knowledge base. We then obtain the embedding of short text by coalescing the words and relevant concepts on top of pre-trained word vectors. We further incorporate character level features into our model to capture fine-grained subword information. Experimental results on five commonly used datasets show that our proposed method significantly outperforms state-of-the-art methods.","doi":"10.24963/ijcai.2017/406","url":"https://ijcai.org/proceedings/2017/406","pdf":"https://ijcai.org/proceedings/2017/0406.pdf","markdown":"https://ijcai.org/proceedings/2017/406.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/406","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":406,"pages":{"first":2915,"last":2921},"keywords":["Machine Learning: Deep Learning","Natural Language Processing: Text Classification"],"videos":[],"references":[{"position":1,"raw":"[Bengio et al., 2003] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural proba- bilistic language model. Journal of Machine Learning Re- search, 3:1137-1155, 2003.","title":"A neural proba-bilistic language model","authors":"Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Janvin","year":"2003","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cheng et al., 2015] Jianpeng Cheng, Zhongyuan Wang, Ji- Rong Wen, Jun Yan, and Zheng Chen. Contextual text understanding in distributional semantic space. In CIKM, pages 133-142, 2015.","title":"Contextual text understanding in distributional semantic space","authors":"Cheng","year":"2015","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Collobert et al., 2011] Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493- 2537, 2011.","title":"Natural language processing (almost) from scratch","authors":"Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, Pavel P Kuksa","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Conneau et al., 2016] Alexis Conneau, Holger Schwenk, Loïc Barrault, and Yann LeCun. Very deep convolu- tional networks for natural language processing. CoRR, abs/1606.01781, 2016.","title":"Very deep convolutional networks for natural language processing","authors":"Alexis Conneau, Holger Schwenk, Loïc Barrault, Yann Lecun","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Duchi et al., 2011] John C. Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121-2159, 2011.","title":"Adaptive subgradient methods for online learning and stochastic optimization","authors":"John C Duchi, Elad Hazan, Yoram Singer","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hill et al., 2016] Felix Hill, KyungHyun Cho, Anna Korho- nen, and Yoshua Bengio. Learning to understand phrases by embedding the dictionary. TACL, 4:17-30, 2016.","title":"Learning to understand phrases by embedding the dictionary","authors":"Felix Hill, Kyunghyun Cho, Anna Korhonen, Yoshua Bengio","year":"2016","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hinton et al., 2012] Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhut- dinov. Improving neural networks by preventing co- adaptation of feature detectors. CoRR, abs/1207.0580, 2012.","title":"Improving neural networks by preventing coadaptation of feature detectors","authors":"Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hu et al., 2016] Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard H. Hovy, and Eric P. Xing. Harnessing deep neural networks with logic rules. In ACL, 2016.","title":"Harnessing deep neural networks with logic rules","authors":"Hu","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hua et al., 2015] Wen Hua, Zhongyuan Wang, Haixun Wang, Kai Zheng, and Xiaofang Zhou. Short text un- derstanding through lexical-semantic analysis. In ICDE, pages 495-506, 2015.","title":"Short text understanding through lexical-semantic analysis","authors":"Wen Hua, Zhongyuan Wang, Haixun Wang, Kai Zheng, Xiaofang Zhou","year":"2015","venue":"ICDE","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2012] Eric H. Huang, Richard Socher, Christopher D. Manning, and Andrew Y. Ng. Improving word representations via global context and multiple word prototypes. In ACL, pages 873-882, 2012.","title":"Improving word representations via global context and multiple word prototypes","authors":"Eric H Huang, Richard Socher, Christopher D Manning, Andrew Y Ng","year":"2012","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kalchbrenner et al., 2014] Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. A convolutional neural network for modelling sentences. In ACL, pages 655-665, 2014.","title":"A convolutional neural network for modelling sentences","authors":"Nal Kalchbrenner, Edward Grefenstette, Phil Blunsom","year":"2014","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kenter and de Rijke, 2015] Tom Kenter and Maarten de Ri- jke. Short text similarity with word embeddings. In CIKM, pages 1411-1420, 2015.","title":"Short text similarity with word embeddings","authors":"Kenter, Tom De Rijke, Maarten Kenter, De Rijke","year":"2015","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kim et al., 2016] Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. Character-aware neural language models. In AAAI, pages 2741-2749, 2016.","title":"Character-aware neural language models","authors":"Yoon Kim, Yacine Jernite, David Sontag, Alexander M Rush","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, pages 1746-1751, 2014.","title":"Convolutional neural networks for sentence classification","authors":"Yoon Kim","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Le and Mikolov, 2014] Quoc V. Le and Tomas Mikolov. Distributed representations of sentences and documents. In ICML, pages 1188-1196, 2014.","title":"Distributed representations of sentences and documents","authors":"V Quoc, Tomas Le, Mikolov","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2013] Pei-Pei Li, Haixun Wang, Kenny Q. Zhu, Zhongyuan Wang, and Xindong Wu. Computing term similarity by large probabilistic isa knowledge. In CIKM, pages 1401-1410, 2013.","title":"Computing term similarity by large probabilistic isa knowledge","authors":"Pei-Pei Li, Haixun Wang, Kenny Q Zhu, Zhongyuan Wang, Xindong Wu","year":"2013","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Palangi et al., 2016] Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song, and Rabab K. Ward. Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval. IEEE/ACM Trans. Audio, Speech & Language Processing, 24(4):694-707, 2016.","title":"Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval","authors":"Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song, Rabab K Ward","year":"2016","venue":"IEEE/ACM Trans. Audio, Speech & Language Processing","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pang and Lee, 2005] Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL, 2005.","title":"Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales","authors":"Lee ; Pang, Bo Pang, Lillian Lee","year":"2005","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Post and Bergsma, 2013] Matt Post and Shane Bergsma. Explicit and implicit syntactic features for text classifica- tion. In ACL, pages 866-872, 2013.","title":"Matt Post and Shane Bergsma. Explicit and implicit syntactic features for text classification","authors":"Bergsma Post","year":"2013","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Shen et al., 2006] Dou Shen, Rong Pan, Jian-Tao Sun, Jef- frey Junfeng Pan, Kangheng Wu, Jie Yin, and Qiang Yang. Query enrichment for web-query classification. ACM Trans. Inf. Syst., 24(3):320-352, 2006.","title":"Query enrichment for web-query classification","authors":"Dou Shen, Rong Pan, Jian-Tao Sun, Jeffrey Junfeng Pan, Kangheng Wu, Jie Yin, Qiang Yang","year":"2006","venue":"ACM Trans. Inf. Syst","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Socher et al., 2011] Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, and Christopher D. Manning. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In NIPS, pages 801-809, 2011.","title":"Dynamic pooling and unfolding recursive autoencoders for paraphrase detection","authors":"Richard Socher, Eric H Huang, Jeffrey Pennington, Andrew Y Ng, Christopher D Manning","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Socher et al., 2013] Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pages 1631-1642. Citeseer, 2013.","title":"Recursive deep models for semantic compositionality over a sentiment treebank","authors":"Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, Christopher Potts","year":"2013","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Song et al., 2011] Yangqiu Song, Haixun Wang, Zhongyuan Wang, Hongsong Li, and Weizhu Chen. Short text conceptualization using a probabilistic knowl- edgebase. In IJCAI, pages 2330-2336, 2011.","title":"Short text conceptualization using a probabilistic knowledgebase","authors":"Song","year":"2011","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2011/388"},{"position":25,"raw":"[Wang and Manning, 2012] Sida I. Wang and Christopher D. Manning. Baselines and bigrams: Simple, good sentiment and topic classification. In ACL, pages 90-94, 2012.","title":"Baselines and bigrams: Simple, good sentiment and topic classification","authors":"Wang, Manning ; ] Sida, I Wang, Christopher D Manning","year":"2012","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang and Wang, 2016] Zhongyuan Wang and Haixun Wang. Understanding short texts(tutorial). In ACL, 2016.","title":"Understanding short texts(tutorial)","authors":"Wang, Wang ;, Zhongyuan Wang, Haixun Wang","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2014] Fang Wang, Zhongyuan Wang, Zhoujun Li, and Ji-Rong Wen. Concept-based short text classifica- tion and ranking. In CIKM, pages 1069-1078, 2014.","title":"Concept-based short text classification and ranking","authors":"Fang Wang, Zhongyuan Wang, Zhoujun Li, Ji-Rong Wen","year":"2014","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wang et al., 2015] Zhongyuan Wang, Kejun Zhao, Haixun Wang, Xiaofeng Meng, and Ji-Rong Wen. Query under- standing through knowledge-based conceptualization. In IJCAI, pages 3264-3270, 2015.","title":"Query understanding through knowledge-based conceptualization","authors":"Zhongyuan Wang, Kejun Zhao, Haixun Wang, Xiaofeng Meng, Ji-Rong Wen","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/460"},{"position":29,"raw":"[Wu et al., 2012] Wentao Wu, Hongsong Li, Haixun Wang, and Kenny Qili Zhu. Probase: a probabilistic taxonomy for text understanding. In SIGMOD, pages 481-492, 2012.","title":"Probase: a probabilistic taxonomy for text understanding","authors":"Wentao Wu, Hongsong Li, Haixun Wang, Kenny Qili, Zhu","year":"2012","venue":"SIGMOD","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2015] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, pages 649-657, 2015.","title":"Character-level convolutional networks for text classification","authors":"Xiang Zhang, Junbo Zhao, Yann Lecun","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}