{"type":"ScholarlyArticle","title":"A Generalized Recurrent Neural Architecture for Text Classification with Multi-Task Learning","authors":[{"name":"Honglun Zhang","slug":"honglun-zhang","given":"Honglun","family":"Zhang","affiliation":"State Key Lab of Advanced Optical Communication System and Network"},{"name":"Liqiang Xiao","slug":"liqiang-xiao","given":"Liqiang","family":"Xiao","affiliation":"State Key Lab of Advanced Optical Communication System and Network"},{"name":"Yongkun Wang","slug":"yongkun-wang","given":"Yongkun","family":"Wang","affiliation":"Network and Information Center Shanghai Jiao Tong University"},{"name":"Yaohui Jin","slug":"yaohui-jin","given":"Yaohui","family":"Jin","affiliation":"State Key Lab of Advanced Optical Communication System and Network; Network and Information Center Shanghai Jiao Tong University"}],"abstract":"Multi-task learning leverages potential correlations among related tasks to extract common features and yield performance gains. However, most previous works only consider simple or weak interactions, thereby failing to model complex correlations among three or more tasks. In this paper, we propose a multi-task learning architecture with four types of recurrent neural layers to fuse information across multiple related tasks. The architecture is structurally flexible and considers various interactions among tasks, which can be regarded as a generalized case of many previous works. Extensive experiments on five benchmark datasets for text classification show that our model can significantly improve performances of related tasks with additional information from others.","doi":"10.24963/ijcai.2017/473","url":"https://ijcai.org/proceedings/2017/473","pdf":"https://ijcai.org/proceedings/2017/0473.pdf","markdown":"https://ijcai.org/proceedings/2017/473.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/473","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":473,"pages":{"first":3385,"last":3391},"keywords":["Machine Learning: Deep Learning","Machine Learning: Transfer, Adaptation, Multi-task Learning","Natural Language Processing: Text Classification"],"videos":[],"references":[{"position":1,"raw":"[Amari, 1993] Shun-ichi Amari. Backpropagation and stochastic gradient descent method. Neurocomputing, 5(3):185-196, 1993.","title":"Shun-ichi Amari. Backpropagation and stochastic gradient descent method","authors":"Amari","year":"1993","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Apté et al., 1994] Chidanand Apté, Fred Damerau, and Sholom M. Weiss. Automated Learning of Decision Rules for Text Categorization. ACM Trans. Inf. Syst., 12(3):233- 251, 1994.","title":"Automated Learning of Decision Rules for Text Categorization","authors":"Chidanand Apté, Fred Damerau, Sholom M Weiss","year":"1994","venue":"ACM Trans. Inf. Syst","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bengio et al., 2013] Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Representation Learning: A Review and New Perspectives. IEEE Trans. Pattern Anal. Mach. Intell., 35(8):1798-1828, 2013.","title":"Representation Learning: A Review and New Perspectives","authors":"Yoshua Bengio, Aaron C Courville, Pascal Vincent","year":"2013","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Blitzer et al., 2007] John Blitzer, Mark Dredze, and Fer- nando Pereira. Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classifica- tion. In ACL, 2007.","title":"Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification","authors":"John Blitzer, Mark Dredze, Fernando Pereira","year":"2007","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Caruana, 1997] Rich Caruana. Multitask Learning. Ma- chine Learning, 28(1):41-75, 1997.","title":"Multitask Learning","authors":"Rich Caruana","year":"1997","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2015] Xinchi Chen, Xipeng Qiu, Chenxi Zhu, Shiyu Wu, and Xuanjing Huang. Sentence Modeling with Gated Recursive Neural Network. In EMNLP, pages 793- 798, 2015.","title":"Sentence Modeling with Gated Recursive Neural Network","authors":"Chen","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Collobert and Weston, 2008] Ronan Collobert and Jason Weston. A unified architecture for natural language pro- cessing: deep neural networks with multitask learning. In ICML, pages 160-167, 2008.","title":"A unified architecture for natural language processing: deep neural networks with multitask learning","authors":"Weston ; Collobert, Ronan Collobert, Jason Weston","year":"2008","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Graves, 2013] Alex Graves. Generating Sequences With Recurrent Neural Networks. CoRR, abs/1308.0850, 2013.","title":"Generating Sequences With Recurrent Neural Networks","authors":"Alex Graves","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1735-1780, 1997.","title":"Long Short-Term Memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Le and Mikolov, 2014] Quoc V. Le and Tomas Mikolov. Distributed Representations of Sentences and Documents. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, pages 1188-1196, 2014.","title":"Distributed Representations of Sentences and Documents","authors":"V Quoc, Tomas Le, Mikolov","year":"2014","venue":"Proceedings of the 31th International Conference on Machine Learning, ICML 2014","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li and Roth, 2002] Xin Li and Dan Roth. Learning Ques- tion Classifiers. In COLING, 2002.","title":"Learning Question Classifiers","authors":"Xin Li, Dan Roth","year":"2002","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2015a] Pengfei Liu, Xipeng Qiu, Xinchi Chen, Shiyu Wu, and Xuanjing Huang. Multi-Timescale Long Short-Term Memory Neural Network for Modelling Sen- tences and Documents. In EMNLP, pages 2326-2335, 2015.","title":"Multi-Timescale Long Short-Term Memory Neural Network for Modelling Sentences and Documents","authors":"Liu","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2015b] Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye-Yi Wang. Represen- tation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval. In NAACL HLT, pages 912-921, 2015.","title":"Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval","authors":"Liu","year":"2015","venue":"NAACL HLT","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2016a] Pengfei Liu, Xipeng Qiu, Jifan Chen, and Xuanjing Huang. Deep Fusion LSTMs for Text Semantic Matching. In ACL, 2016.","title":"Deep Fusion LSTMs for Text Semantic Matching","authors":"Liu","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2016b] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Deep Multi-Task Learning with Shared Memory for Text Classification. In EMNLP, pages 118-127, 2016.","title":"Deep Multi-Task Learning with Shared Memory for Text Classification","authors":"Liu","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2016c] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Recurrent Neural Network for Text Classification with Multi-Task Learning. In IJCAI, pages 2873-2879, 2016.","title":"Recurrent Neural Network for Text Classification with Multi-Task Learning","authors":"Liu","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/408"},{"position":17,"raw":"[Liu et al., 2016d] Pengfei Liu, Xipeng Qiu, Yaqian Zhou, Jifan Chen, and Xuanjing Huang. Modelling Interaction of Sentence Pair with Coupled-LSTMs. In EMNLP, pages 1703-1712, 2016.","title":"Modelling Interaction of Sentence Pair with Coupled-LSTMs","authors":"Liu","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Maas et al., 2011] Andrew L. Maas, Raymond E. Daly, Pe- ter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning Word Vectors for Sentiment Analysis. In NAACL HLT, pages 142-150. Association for Computa- tional Linguistics, June 2011.","title":"Learning Word Vectors for Sentiment Analysis","authors":"Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, Christopher Potts","year":"2011","venue":"NAACL HLT","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mikolov et al., 2013a] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. CoRR, abs/1301.3781, 2013.","title":"Efficient Estimation of Word Representations in Vector Space","authors":"Mikolov","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mikolov et al., 2013b] Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed Representations of Words and Phrases and their Composi- tionality. In Advances in Neural Information Processing Systems, pages 3111-3119, 2013.","title":"Distributed Representations of Words and Phrases and their Compositionality","authors":"Mikolov","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Socher et al., 2013] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In EMNLP, pages 1631-1642, Stroudsburg, PA, October 2013. Association for Computational Linguistics.","title":"Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank","authors":"Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, Christopher Potts","year":"2013","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems, pages 3104-3112, 2014.","title":"Le. Sequence to Sequence Learning with Neural Networks","authors":"Sutskever","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}