{"type":"ScholarlyArticle","title":"Same Representation, Different Attentions: Shareable Sentence Representation Learning from Multiple Tasks","authors":[{"name":"Renjie Zheng","slug":"renjie-zheng","given":"Renjie","family":"Zheng","affiliation":"Shanghai Key Laboratory of Intelligent Information Processing , Fudan University School of Computer Science , Fudan University , 825 Zhangheng Road , Shanghai , China"},{"name":"Junkun Chen","slug":"junkun-chen","given":"Junkun","family":"Chen","affiliation":"Shanghai Key Laboratory of Intelligent Information Processing , Fudan University School of Computer Science , Fudan University , 825 Zhangheng Road , Shanghai , China"},{"name":"Xipeng Qiu","slug":"xipeng-qiu","given":"Xipeng","family":"Qiu","affiliation":"Shanghai Key Laboratory of Intelligent Information Processing , Fudan University School of Computer Science , Fudan University , 825 Zhangheng Road , Shanghai , China"}],"abstract":"Distributed representation plays an important role in deep learning based natural language processing.   However, the representation of a sentence often varies in different tasks, which is usually learned from scratch and suffers from the limited amounts of training data. In this paper, we claim that a good sentence representation should be invariant and can benefit the various subsequent tasks. To achieve this purpose, we propose a new scheme of information sharing for multi-task learning. More specifically, all tasks share the same sentence representation and each task can select the task-specific information from the shared sentence representation with attention mechanisms. The query vector of each task's attention could be either static parameters or generated dynamically. We conduct extensive experiments on 16 different text classification tasks, which demonstrate the benefits of our architecture. Source codes of this paper are available on Github.","doi":"10.24963/ijcai.2018/642","url":"https://ijcai.org/proceedings/2018/642","pdf":"https://ijcai.org/proceedings/2018/0642.pdf","markdown":"https://ijcai.org/proceedings/2018/642.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/642","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":642,"pages":{"first":4616,"last":4622},"keywords":["Machine Learning: Transfer, Adaptation, Multi-task Learning","Natural Language Processing: Natural Language Processing","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Bahdanau et al., 2014] D. Bahdanau, K. Cho, and Y. Ben- gio. Neural machine translation by jointly learning to align and translate. ArXiv e-prints, September 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"D Bahdanau, K Cho, Y Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Caruana, 1997] Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.","title":"Multitask learning","authors":"Caruana","year":"1997","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chung et al., 2014] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evalua- tion of gated recurrent neural networks on sequence mod- eling. arXiv preprint arXiv:1412.3555, 2014.","title":"Empirical evaluation of gated recurrent neural networks on sequence modeling","authors":"Chung","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Collobert and Weston, 2008] Ronan Collobert and Jason Weston. A unified architecture for natural language pro- cessing: Deep neural networks with multitask learning. In Proceedings of ICML, 2008.","title":"A unified architecture for natural language processing: Deep neural networks with multitask learning","authors":"Weston ; Collobert, Ronan Collobert, Jason Weston","year":"2008","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Collobert et al., 2011] Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493-2537, 2011.","title":"Natural language processing (almost) from scratch","authors":"Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, Pavel Kuksa","year":"2011","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Glorot et al., 2011] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Domain adaptation for large-scale sen- timent classification: A deep learning approach. In ICML, pages 513-520, 2011.","title":"Domain adaptation for large-scale sentiment classification: A deep learning approach","authors":"Xavier Glorot, Antoine Bordes, Yoshua Bengio","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882, 2014.","title":"Convolutional neural networks for sentence classification","authors":"Yoon Kim","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kingma and Ba, 2014] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2014.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lin et al., 2017] Zhouhan Lin, Minwei Feng, Cícero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self- attentive sentence embedding. Proceedings of ICLR, 2017.","title":"A structured selfattentive sentence embedding","authors":"Lin","year":"2017","venue":"Proceedings of ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Liu et al., 2016] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Recurrent neural network for text classification with multi-task learning. In Proceedings of IJCAI, pages 2873-2879, 2016.","title":"Recurrent neural network for text classification with multi-task learning","authors":"Pengfei Liu, Xipeng Qiu, Xuanjing Huang","year":"2016","venue":"Proceedings of IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/408"},{"position":11,"raw":"[Liu et al., 2017] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Adversarial multi-task learning for text classifi- cation. In ACL, 2017.","title":"Adversarial multi-task learning for text classification","authors":"Pengfei Liu, Xipeng Qiu, Xuanjing Huang","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Luong et al., 2015] Thang Luong, Hieu Pham, and Christo- pher D. Manning. Effective approaches to attention-based neural machine translation. In Proceedings of EMNLP, pages 1412-1421, 2015.","title":"Effective approaches to attention-based neural machine translation","authors":"Thang Luong, Hieu Pham, Christopher D Manning","year":"2015","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ma et al., 2015] Mingbo Ma, Liang Huang, Bing Xiang, and Bowen Zhou. Dependency-based convolutional neural networks for sentence embedding. ACL, 2015. [Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed rep- resentations of words and phrases and their composition- ality. In NIPS, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mingbo Ma, Liang Huang, Bing Xiang, Bowen Zhou, ; Mikolov","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mnih et al., 2014] Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In NIPS, 2014.","title":"Recurrent models of visual attention","authors":"Volodymyr Mnih, Nicolas Heess, Alex Graves","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Proceedings of EMNLP, pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ruder, 2017] Sebastian Ruder. An overview of multi- task learning in deep neural networks. arXiv preprint, arXiv:1706.05098, 2017.","title":"An overview of multitask learning in deep neural networks","authors":"Sebastian Ruder","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rush et al., 2015] Alexander M. Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. In EMNLP, pages 379-389, 2015.","title":"A neural attention model for abstractive sentence summarization","authors":"Alexander M Rush, Sumit Chopra, Jason Weston","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sang and Buchholz, 2000] Erik F. Tjong Kim Sang and Sabine Buchholz. Introduction to the conll-2000 shared task chunking. In CoNLL, 2000.","title":"Introduction to the conll-2000 shared task chunking","authors":"Erik F Tjong, Kim Sang, Sabine Buchholz","year":"2000","venue":"CoNLL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Socher et al., 2013] Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013.","title":"Recursive deep models for semantic compositionality over a sentiment treebank","authors":"Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, Christopher Potts","year":"2013","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. Sequence to sequence learning with neu- ral networks. In NIPS, 2014.","title":"Quoc VV Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tai et al., 2015] Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representa- tions from tree-structured long short-term memory net- works. arXiv preprint arXiv:1503.00075, 2015.","title":"Improved semantic representations from tree-structured long short-term memory networks","authors":"Kai Sheng, Tai, Richard Socher, Christopher D Manning","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NPIS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2017] Shaonan Wang, Jiajun Zhang, and Chengqing Zong. Learning sentence representation with guidance of human attention. In IJCAI, pages 4137-4143, 2017.","title":"Learning sentence representation with guidance of human attention","authors":"Wang","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/578"},{"position":24,"raw":"[Zhao et al., 2016] Kai Zhao, Liang Huang, and Mingbo Ma. Textual entailment with structured attentions and compo- sition. COLING, 2016.","title":"Textual entailment with structured attentions and composition","authors":"Kai Zhao, Liang Huang, Mingbo Ma","year":"2016","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhu et al., 2015] Xiaodan Zhu, Parinaz Sobihani, and Hongyu Guo. Long short-term memory over recursive structures. In ICML, 2015.","title":"Long short-term memory over recursive structures","authors":"Xiaodan Zhu, Parinaz Sobihani, Hongyu Guo","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}