{"type":"ScholarlyArticle","title":"Differentiated Attentive Representation Learning for Sentence Classification","authors":[{"name":"Qianrong Zhou","slug":"qianrong-zhou","given":"Qianrong","family":"Zhou","affiliation":"Center for Intelligence Science and Technology ,; School of Computer Science , Beijing University of Posts and Telecommunications"},{"name":"Xiaojie Wang","slug":"xiaojie-wang","given":"Xiaojie","family":"Wang","affiliation":"Center for Intelligence Science and Technology ,; School of Computer Science , Beijing University of Posts and Telecommunications"},{"name":"Xuan Dong","slug":"xuan-dong","given":"Xuan","family":"Dong","affiliation":"Center for Intelligence Science and Technology ,; School of Computer Science , Beijing University of Posts and Telecommunications"}],"abstract":"Attention-based models have shown to be effective in learning representations for sentence classification. They are typically equipped with multi-hop attention mechanism. However, existing multi-hop models still suffer from the problem of paying much attention to the most frequently noticed words, which might not be important to classify the current sentence. And there is a lack of explicitly effective way that helps the attention to be shifted out of a wrong part in the sentence. In this paper, we alleviate this problem by proposing a differentiated attentive learning model. It is composed of two branches of attention subnets and an example discriminator. An explicit signal with the loss information of the first attention subnet is passed on to the second one to drive them to learn different attentive preference. The example discriminator then selects the suitable attention subnet for sentence classification. Experimental results on real and synthetic datasets demonstrate the effectiveness of our model.","doi":"10.24963/ijcai.2018/644","url":"https://ijcai.org/proceedings/2018/644","pdf":"https://ijcai.org/proceedings/2018/0644.pdf","markdown":"https://ijcai.org/proceedings/2018/644.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/644","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":644,"pages":{"first":4630,"last":4636},"keywords":["Natural Language Processing: Sentiment Analysis and Text Mining","Natural Language Processing: Text Classification"],"videos":[],"references":[{"position":1,"raw":"[Chen et al., 2017] Peng Chen, Zhongqian Sun, Lidong Bing, and Wei Yang. Recurrent attention network on mem- ory for aspect sentiment analysis. In EMNLP, 2017.","title":"Recurrent attention network on memory for aspect sentiment analysis","authors":"Peng Chen, Zhongqian Sun, Lidong Bing, Wei Yang","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cheng et al., 2016] Jianpeng Cheng, Li Dong, and Mirella Lapata. Long short-term memory-networks for machine reading. In EMNLP, 2016.","title":"Long short-term memory-networks for machine reading","authors":"Cheng","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Choi et al., 2017] Jihun Choi, Kang Min Yoo, and Sang-goo Lee. Unsupervised learning of task-specific tree structures with tree-lstms. arXiv preprint arXiv:1707.02786, 2017.","title":"Unsupervised learning of task-specific tree structures with tree-lstms","authors":"Jihun Choi, Min Kang, Sang-Goo Yoo, Lee","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Collobert et al., 2011] Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. The JMLR, 12(Aug):2493-2537, 2011.","title":"Natural language processing (almost) from scratch","authors":"Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, Pavel Kuksa","year":"2011","venue":"The JMLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, 2014.","title":"Convolutional neural networks for sentence classification","authors":"Yoon Kim","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kiros et al., 2015] Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Ad- vances in NIPS, pages 3294-3302, 2015.","title":"Skip-thought vectors","authors":"Ryan Kiros, Yukun Zhu, R Ruslan, Richard Salakhutdinov, Raquel Zemel, Antonio Urtasun, Sanja Torralba, Fidler","year":"2015","venue":"Advances in NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Krishnamurthy and Mitchell, 2013] Jayant Krishnamurthy and Tom M Mitchell. Vector space semantic parsing: a framework for compositional vector space models. In ACL, page 1, 2013.","title":"Vector space semantic parsing: a framework for compositional vector space models","authors":"Jayant Krishnamurthy, Tom M Mitchell","year":"2013","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kumar et al., 2016] Ankit Kumar, Ozan Irsoy, Peter On- druska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. Ask me anything: dynamic memory networks for natural lan- guage processing. In ICML, pages 1378-1387, 2016.","title":"Ask me anything: dynamic memory networks for natural language processing","authors":"Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, Richard Socher","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Le and Mikolov, 2014] Quoc Le and Tomas Mikolov. Dis- tributed representations of sentences and documents. In ICML, pages 1188-1196, 2014.","title":"Quoc Le and Tomas Mikolov. Distributed representations of sentences and documents","authors":"Mikolov Le","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li and Roth, 2002] Xin Li and Dan Roth. Learning question classifiers. In Proceedings of the 19th International Con- ference on Computational Linguistics-Volume 1, 2002.","title":"Learning question classifiers","authors":"Roth ; Li, Xin Li, Dan Roth","year":"2002","venue":"Proceedings of the 19th International Conference on Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lin et al., 2017] Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. In ICLR, 2017.","title":"A structured self-attentive sentence embedding","authors":"Lin","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2016] Yang Liu, Chengjie Sun, Lei Lin, and Xi- aolong Wang. Learning natural language inference us- ing bidirectional lstm model and inner-attention. arXiv preprint arXiv:1605.09090, 2016.","title":"Learning natural language inference using bidirectional lstm model and inner-attention","authors":"Yang Liu, Chengjie Sun, Lei Lin, Xiaolong Wang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2017] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Dynamic compositional neural networks over tree structure. In IJCAI, 2017. [Looks et al., 2017] Moshe Looks, Marcello Herreshoff, De- Lesley Hutchins, and Peter Norvig. Deep learning with dynamic computation graphs. In ICLR, 2017.","title":"Dynamic compositional neural networks over tree structure","authors":"Pengfei Liu, Xipeng Qiu, Xuanjing Huang","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/566"},{"position":14,"raw":"[Munkhdalai and Yu, 2017] Tsendsuren Munkhdalai and Hong Yu. Neural semantic encoders. In EACL, 2017.","title":"Tsendsuren Munkhdalai and Hong Yu. Neural semantic encoders","authors":"Yu Munkhdalai","year":"2017","venue":"EACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pang and Lee, 2004] Bo Pang and Lillian Lee. A sentimen- tal education: Sentiment analysis using subjectivity sum- marization based on minimum cuts. In ACL, 2004.","title":"A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts","authors":"Lee ; Pang, Bo Pang, Lillian Lee","year":"2004","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pang and Lee, 2005] Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL, 2005.","title":"Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales","authors":"Lee ; Pang, Bo Pang, Lillian Lee","year":"2005","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: global vectors for word representation. In EMNLP, 2014.","title":"Glove: global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Qian et al., 2017] Qiao Qian, Minlie Huang, Jinhao Lei, and Xiaoyan Zhu. Linguistically regularized lstms for sen- timent classification. In ACL, 2017.","title":"Linguistically regularized lstms for sentiment classification","authors":"Qiao Qian, Minlie Huang, Jinhao Lei, Xiaoyan Zhu","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Socher et al., 2013] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for se- mantic compositionality over a sentiment treebank. In EMNLP, pages 1631-1642, 2013.","title":"Recursive deep models for semantic compositionality over a sentiment treebank","authors":"Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, Christopher Potts","year":"2013","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey E Hin- ton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The JMLR, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"The JMLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sukhbaatar et al., 2015] Sainbayar Sukhbaatar, Jason We- ston, Rob Fergus, et al. End-to-end memory networks. In Advances in NIPS, pages 2440-2448, 2015.","title":"End-to-end memory networks","authors":"Sukhbaatar","year":"2015","venue":"Advances in NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tai et al., 2015] Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representa- tions from tree-structured long short-term memory net- works. In ACL, 2015.","title":"Improved semantic representations from tree-structured long short-term memory networks","authors":"Kai Sheng, Tai, Richard Socher, Christopher D Manning","year":"2015","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tu et al., 2016] Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. Modeling coverage for neu- ral machine translation. In ACL, pages 76-85, 2016.","title":"Modeling coverage for neural machine translation","authors":"Tu","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Williams et al., 2017] Adina Williams, Andrew Drozdov, and Samuel R Bowman. Learning to parse from a se- mantic objective: It works. is it syntax? arXiv preprint arXiv:1709.01121, 2017.","title":"Learning to parse from a semantic objective: It works. is it syntax","authors":"Adina Williams, Andrew Drozdov, Samuel R Bowman","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2016] Zichao Yang, Diyi Yang, Chris Dyer, Xi- aodong He, Alexander J Smola, and Eduard H Hovy. Hi- erarchical attention networks for document classification. In HLT-NAACL, pages 1480-1489, 2016.","title":"Hierarchical attention networks for document classification","authors":"Yang","year":"2016","venue":"HLT-NAACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2016] Rui Zhang, Honglak Lee, and Dragomir Radev. Dependency sensitive convolutional neural net- works for modeling sentences and documents. In NAACL- HLT, pages 1512-1521, 2016.","title":"Dependency sensitive convolutional neural networks for modeling sentences and documents","authors":"Rui Zhang, Honglak Lee, Dragomir Radev","year":"2016","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhao et al., 2015] Han Zhao, Zhengdong Lu, and Pascal Poupart. Self-adaptive hierarchical sentence model. In IJCAI, pages 4069-4076, 2015.","title":"Self-adaptive hierarchical sentence model","authors":"Han Zhao, Zhengdong Lu, Pascal Poupart","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/571"},{"position":28,"raw":"[Zhou et al., 2016] Peng Zhou, Zhenyu Qi, Suncong Zheng, Jiaming Xu, Hongyun Bao, and Bo Xu. Text classifica- tion improved by integrating bidirectional lstm with two- dimensional max pooling. In COLING, 2016.","title":"Text classification improved by integrating bidirectional lstm with twodimensional max pooling","authors":"Peng Zhou, Zhenyu Qi, Suncong Zheng, Jiaming Xu, Hongyun Bao, Bo Xu","year":"2016","venue":"COLING","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}