{"type":"ScholarlyArticle","title":"TreeNet: Learning Sentence Representations with Unconstrained Tree Structure","authors":[{"name":"Zhou Cheng","slug":"zhou-cheng","given":"Zhou","family":"Cheng","affiliation":"Department of Computer Science and Technology , Tsinghua University; Graduate school at ShenZhen , Tsinghua University"},{"name":"Chun Yuan","slug":"chun-yuan","given":"Chun","family":"Yuan","affiliation":"Graduate school at ShenZhen , Tsinghua University"},{"name":"Jiancheng Li","slug":"jiancheng-li","given":"Jiancheng","family":"Li","affiliation":"Department of Computer Science and Technology , Tsinghua University; Graduate school at ShenZhen , Tsinghua University"},{"name":"Haiqin Yang","slug":"haiqin-yang","given":"Haiqin","family":"Yang","affiliation":"Department of Computing , Hang Seng Management College {z-cheng15 ,"}],"abstract":"Recursive neural network (RvNN) has been proved to be an effective and promising tool to learn sentence representations by explicitly exploiting the sentence structure. However, most existing work can only exploit simple tree structure, e.g., binary trees, or ignore the order of nodes, which yields suboptimal performance. In this paper, we proposed a novel neural network, namely TreeNet, to capture sentences structurally over the raw unconstrained constituency trees, where the number of child nodes can be arbitrary. In TreeNet, each node is learning from its left sibling and right child in a bottom-up left-to-right order, thus enabling the net to learn over any tree. Furthermore, multiple soft gates and a memory cell are employed in implementing the TreeNet to determine to what extent it should learn, remember and output, which proves to be a simple and efficient mechanism for semantic synthesis. Moreover, TreeNet significantly suppresses convolutional neural networks (CNN) and Long Short-Term Memory (LSTM) with fewer parameters. It improves the classification accuracy by 2%-5% with 42% of the best CNN’s parameters or 94% of standard LSTM’s. Extensive experiments demonstrate TreeNet achieves the state-of-the-art performance on all four typical text classification tasks.","doi":"10.24963/ijcai.2018/557","url":"https://ijcai.org/proceedings/2018/557","pdf":"https://ijcai.org/proceedings/2018/0557.pdf","markdown":"https://ijcai.org/proceedings/2018/557.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/557","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":557,"pages":{"first":4005,"last":4011},"keywords":["Natural Language Processing: Natural Language Semantics","Machine Learning: Neural Networks","Machine Learning: Deep Learning","Natural Language Processing: Natural Language Processing","Natural Language Processing: Text Classification"],"videos":[],"references":[{"position":1,"raw":"Hong Kong Special Administrative Region, China (Project No.UGC/IDS14/16).","title":"Special Administrative Region, China (Project","authors":"Hong Kong","year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 1994] Yoshua Bengio, Patrice Y. Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5 2:157-66, 1994.","title":"Learning long-term dependencies with gradient descent is difficult","authors":"Yoshua Bengio, Patrice Y Simard, Paolo Frasconi","year":"1994","venue":"IEEE transactions on neural networks","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dong et al., 2014] Li Dong, Furu Wei, Ming Zhou, and Ke Xu. Adaptive multi-compositionality for recursive neu- ral models with applications to sentiment analysis. In AAAI, 2014.","title":"Adaptive multi-compositionality for recursive neural models with applications to sentiment analysis","authors":"Li Dong, Furu Wei, Ming Zhou, Ke Xu","year":"2014","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Elman, 1990] Jeffrey L Elman. Finding structure in time. Cognitive science, 14(2):179-211, 1990.","title":"Finding structure in time","authors":"Jeffrey L Elman","year":"1990","venue":"Cognitive science","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Júrgen Schmidhuber. Long short-term memory. Neural computation, 9 8:1735-80, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Júrgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hochreiter, 1998] Sepp Hochreiter. The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 6:107-116, 1998.","title":"The vanishing gradient problem during learning recurrent neural nets and problem solutions","authors":"Sepp Hochreiter","year":"1998","venue":"International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu and Liu, 2004] Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In KDD, 2004.","title":"Mining and summarizing customer reviews","authors":"Liu Hu, Minqing Hu, Bing Liu","year":"2004","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, 2014.","title":"Convolutional neural networks for sentence classification","authors":"Yoon Kim","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kiros et al., 2015] Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Raquel Urtasun, Anto- nio Torralba, and Sanja Fidler. Skip-thought vectors. In NIPS, 2015.","title":"Skip-thought vectors","authors":"Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Raquel Urtasun, Antonio Torralba, Sanja Fidler","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li and Roth, 2002] Xin Li and Dan Roth. Learning question classifiers. In COLING, 2002.","title":"Learning question classifiers","authors":"Xin Li, Dan Roth","year":"2002","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2017a] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Adaptive semantic compositionality for sentence modelling. In IJCAI, 2017.","title":"Adaptive semantic compositionality for sentence modelling","authors":"Liu","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/567"},{"position":13,"raw":"[Liu et al., 2017b] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Dynamic compositional neural networks over tree structure. In IJCAI, 2017.","title":"Dynamic compositional neural networks over tree structure","authors":"Liu","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/566"},{"position":14,"raw":"[Pang and Lee, 2004] Bo Pang and Lillian Lee. A sentimen- tal education: Sentiment analysis using subjectivity sum- marization based on minimum cuts. In ACL, 2004.","title":"A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts","authors":"Lee ; Pang, Bo Pang, Lillian Lee","year":"2004","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pang and Lee, 2005] Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL, 2005.","title":"Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales","authors":"Lee ; Pang, Bo Pang, Lillian Lee","year":"2005","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In EMNLP, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Socher et al., 2011] Richard Socher, Cliff Chiung-Yu Lin, Andrew Y. Ng, and Christopher D. Manning. Parsing nat- ural scenes and natural language with recursive neural net- works. In ICML, 2011. Socher et al., 2013]","title":"Parsing natural scenes and natural language with recursive neural networks","authors":"Richard Socher, Cliff Chiung-Yu Lin, Andrew Y Ng, Christopher D Manning","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for se- mantic compositionality over a sentiment treebank. In EMNLP, 2013.","title":"Recursive deep models for semantic compositionality over a sentiment treebank","authors":"Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, Christopher Potts","year":"2013","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In NIPS, 2014.","title":"Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Tai et al., 2015] Kai Sheng Tai, Richard Socher, and Christopher D. Manning. Improved semantic represen- tations from tree-structured long short-term memory net- works. In ACL, 2015.","title":"Improved semantic representations from tree-structured long short-term memory networks","authors":"Kai Sheng, Tai, Richard Socher, Christopher D Manning","year":"2015","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Teng and Zhang, 2017] Zhiyang Teng and Yue Zhang. Head-lexicalized bidirectional tree lstms. TACL, 5:163- 177, 2017.","title":"Head-lexicalized bidirectional tree lstms","authors":"Zhiyang Teng, Yue Zhang","year":"2017","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tomas, 2012] Mikolov Tomas. Statistical language models based on neural networks. PhD thesis, Brno University of Technology, 2012.","title":"Mikolov Tomas. Statistical language models based on neural networks","authors":"Tomas","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang and Manning, 2012] Sida Wang and Christopher D Manning. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th An- nual Meeting of the Association for Computational Lin- guistics: Short Papers-Volume 2, pages 90-94. Associa- tion for Computational Linguistics, 2012.","title":"Baselines and bigrams: Simple, good sentiment and topic classification","authors":"Wang, Manning, Sida Wang, Christopher D Manning","year":"2012","venue":"Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang and Nyberg, 2015] Di Wang and Eric Nyberg. A long short-term memory model for answer sentence selection in question answering. In ACL, 2015.","title":"A long short-term memory model for answer sentence selection in question answering","authors":"Di Wang, Eric Nyberg","year":"2015","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2016] Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao. Attention-based lstm for aspect-level sentiment classification. In EMNLP, 2016.","title":"Attention-based lstm for aspect-level sentiment classification","authors":"Wang","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zaremba and Sutskever, 2014] Wojciech Zaremba and Ilya Sutskever. Learning to execute. arXiv preprint arXiv:1410.4615, 2014.","title":"Learning to execute","authors":"Wojciech Zaremba, Ilya Sutskever","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang and Wallace, 2015] Ye Zhang and Byron C. Wal- lace. A sensitivity analysis of (and practitioners' guide to) convolutional neural networks for sentence classification. CoRR, abs/1510.03820, 2015.","title":"A sensitivity analysis of (and practitioners' guide to) convolutional neural networks for sentence classification","authors":"Wallace ; ] Ye Zhang, Byron C Zhang, Wallace","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2016] Rui Zhang, Honglak Lee, and Dragomir R. Radev. Dependency sensitive convo- lutional neural networks for modeling sentences and documents. In HLT-NAACL, 2016.","title":"Dependency sensitive convolutional neural networks for modeling sentences and documents","authors":"Rui Zhang, Honglak Lee, Dragomir R Radev","year":"2016","venue":"HLT-NAACL","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang et al., 2017] Honglun Zhang, Liqiang Xiao, Yongkun Wang, and Yaohui Jin. A generalized re- current neural architecture for text classification with multi-task learning. In IJCAI, 2017.","title":"A generalized recurrent neural architecture for text classification with multi-task learning","authors":"Zhang","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/473"},{"position":30,"raw":"[Zhao et al., 2015] Han Zhao, Zhengdong Lu, and Pascal Poupart. Self-adaptive hierarchical sentence model. In IJCAI, pages 4069-4076, 2015.","title":"Self-adaptive hierarchical sentence model","authors":"Han Zhao, Zhengdong Lu, Pascal Poupart","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/571"},{"position":31,"raw":"[Zhou et al., 2016] Peng Zhou, Zhenyu Qi, Suncong Zheng, Jiaming Xu, Hongyun Bao, and Bo Xu. Text classifica- tion improved by integrating bidirectional lstm with two- dimensional max pooling. In COLING, 2016.","title":"Text classification improved by integrating bidirectional lstm with twodimensional max pooling","authors":"Peng Zhou, Zhenyu Qi, Suncong Zheng, Jiaming Xu, Hongyun Bao, Bo Xu","year":"2016","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhu et al., 2015] Xiao-Dan Zhu, Parinaz Sobhani, and Hongyu Guo. Long short-term memory over recursive structures. In ICML, 2015.","title":"Long short-term memory over recursive structures","authors":"Xiao-Dan Zhu, Parinaz Sobhani, Hongyu Guo","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}