{"type":"ScholarlyArticle","title":"On the Importance of Word and Sentence Representation Learning in Implicit Discourse Relation Classification","authors":[{"name":"Xin Liu","slug":"xin-liu","given":"Xin","family":"Liu","affiliation":"Department of CSE , the Hong Kong University of Science and Technology"},{"name":"Jiefu Ou","slug":"jiefu-ou","given":"Jiefu","family":"Ou","affiliation":"Department of CSE , the Hong Kong University of Science and Technology"},{"name":"Yangqiu Song","slug":"yangqiu-song","given":"Yangqiu","family":"Song","affiliation":"Department of CSE , the Hong Kong University of Science and Technology"},{"name":"Xin Jiang","slug":"xin-jiang","given":"Xin","family":"Jiang","affiliation":"Huawei Noah's Ark Lab"}],"abstract":"Implicit discourse relation classification is one of the most difficult parts in shallow discourse parsing as the relation prediction without explicit connectives requires the language understanding at both the text span level and the sentence level. Previous studies mainly focus on the interactions between two arguments.  We argue that a powerful contextualized representation module, a bilateral multi-perspective matching module, and a global information fusion module are all important to implicit discourse analysis. We propose a novel model to combine these modules together. Extensive experiments show that our proposed model outperforms BERT and other state-of-the-art systems on the PDTB dataset by around 8% and CoNLL 2016 datasets around 16%.  We also analyze the effectiveness of different modules in the implicit discourse relation classification task and demonstrate how different levels of representation learning can affect the results.","doi":"10.24963/ijcai.2020/530","url":"https://ijcai.org/proceedings/2020/530","pdf":"https://ijcai.org/proceedings/2020/0530.pdf","markdown":"https://ijcai.org/proceedings/2020/530.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/530","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":530,"pages":{"first":3830,"last":3836},"keywords":["Natural Language Processing: Discourse","Machine Learning: Deep Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25343/file_25343.m3u8","length_sec":289,"provider":"hls"}],"references":[{"position":1,"raw":"[Bai and Zhao, 2018] Hongxiao Bai and Hai Zhao. Deep enhanced representation for implicit discourse relation recognition. In COLING, pages 571-583, 2018.","title":"Deep enhanced representation for implicit discourse relation recognition","authors":"Zhao Bai, Hongxiao Bai, Hai Zhao","year":"2018","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bosselut et al., 2018] Antoine Bosselut, Asli C ¸elikyilmaz, Xi- aodong He, Jianfeng Gao, Po-Sen Huang, and Yejin Choi. Discourse-aware neural rewards for coherent text generation. In NAACL-HLT, pages 173-184, 2018.","title":"Discourse-aware neural rewards for coherent text generation","authors":"Antoine Bosselut, Xiaodong Asli C ¸elikyilmaz, Jianfeng He, Po-Sen Gao, Yejin Huang, Choi","year":"2018","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cohan et al., 2018] Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Go- harian. A discourse-aware attention model for abstractive sum- marization of long documents. In NAACL-HLT, pages 615-621, 2018.","title":"A discourse-aware attention model for abstractive summarization of long documents","authors":"Arman Cohan, Franck Dernoncourt, Soon Doo, Trung Kim, Seokhwan Bui, Walter Kim, Nazli Chang, Goharian","year":"2018","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dai and Huang, 2018] Zeyu Dai and Ruihong Huang. Improv- ing implicit discourse relation classification by modeling inter- dependencies of discourse units in a paragraph. In NAACL-HLT, pages 141-151, 2018.","title":"Improving implicit discourse relation classification by modeling interdependencies of discourse units in a paragraph","authors":"Huang ; ] Zeyu Dai, Ruihong Dai, Huang","year":"2018","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dai and Huang, 2019] Zeyu Dai and Ruihong Huang. A regular- ization approach for incorporating event knowledge and corefer- ence relations into neural discourse parsing. In EMNLP, pages 2974-2985, 2019.","title":"A regularization approach for incorporating event knowledge and coreference relations into neural discourse parsing","authors":"Huang ; ] Zeyu Dai, Ruihong Dai, Huang","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pages 4171-4186, 2019.","title":"BERT: pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jansen et al., 2014] Peter Jansen, Mihai Surdeanu, and Peter Clark. Discourse complements lexical semantics for non-factoid answer reranking. In ACL, pages 977-986, 2014.","title":"Discourse complements lexical semantics for non-factoid answer reranking","authors":"Jansen","year":"2014","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ji and Eisenstein, 2015] Yangfeng Ji and Jacob Eisenstein. One vector is not enough: Entity-augmented distributed semantics for discourse relations. TACL, 3:329-344, 2015.","title":"Yangfeng Ji and Jacob Eisenstein. One vector is not enough: Entity-augmented distributed semantics for discourse relations","authors":"Ji, Eisenstein","year":"2015","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kim et al., 2016] Yoon Kim, Yacine Jernite, David A. Sontag, and Alexander M. Rush. Character-aware neural language models. In AAAI, pages 2741-2749, 2016.","title":"Character-aware neural language models","authors":"Yoon Kim, Yacine Jernite, David A Sontag, Alexander M Rush","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lan et al., 2017] Man Lan, Jianxiang Wang, Yuanbin Wu, Zheng- Yu Niu, and Haifeng Wang. Multi-task attention-based neural networks for implicit discourse relationship representation and identification. In EMNLP, pages 1299-1308, 2017.","title":"Multi-task attention-based neural networks for implicit discourse relationship representation and identification","authors":"Lan","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2014] Junyi Jessy Li, Marine Carpuat, and Ani Nenkova. Assessing the discourse factors that influence the quality of ma- chine translation. In ACL, pages 283-288, 2014.","title":"Assessing the discourse factors that influence the quality of machine translation","authors":"Jessy Junyi, Marine Li, Ani Carpuat, Nenkova","year":"2014","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lin et al., 2009] Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. Recognizing implicit discourse relations in the penn discourse treebank. In EMNLP, pages 343-351, 2009.","title":"Recognizing implicit discourse relations in the penn discourse treebank","authors":"Lin","year":"2009","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu and Li, 2016] Yang Liu and Sujian Li. Recognizing implicit discourse relations via repeated reading: Neural networks with multi-level attention. In EMNLP, pages 1224-1233, 2016.","title":"Recognizing implicit discourse relations via repeated reading: Neural networks with multi-level attention","authors":"Li ; Liu, Yang Liu, Sujian Li","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2016] Yang Liu, Sujian Li, Xiaodong Zhang, and Zhi- fang Sui. Implicit discourse relation classification via multi-task neural networks. In AAAI, pages 2750-2756, 2016.","title":"Implicit discourse relation classification via multi-task neural networks","authors":"Yang Liu, Sujian Li, Xiaodong Zhang, Zhifang Sui","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A ro- bustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.","title":"Roberta: A robustly optimized bert pretraining approach","authors":"Liu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Peters et al., 2018] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL-HLT, pages 2227-2237, 2018.","title":"Deep contextualized word representations","authors":"Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer","year":"2018","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pitler et al., 2008] Emily Pitler, Mridhula Raghupathy, Hena Mehta, Ani Nenkova, Alan Lee, and Aravind K. Joshi. Easily identifiable discourse relations. In COLING, pages 87-90, 2008.","title":"Easily identifiable discourse relations","authors":"Emily Pitler, Mridhula Raghupathy, Hena Mehta, Ani Nenkova, Alan Lee, Aravind K Joshi","year":"2008","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pitler et al., 2009] Emily Pitler, Annie Louis, and Ani Nenkova. Automatic sense prediction for implicit discourse relations in text. In ACL, pages 683-691, 2009.","title":"Automatic sense prediction for implicit discourse relations in text","authors":"Pitler","year":"2009","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Prasad et al., 2008] Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind K. Joshi, and Bon- nie L. Webber. The penn discourse treebank 2.0. In LREC, 2008.","title":"The penn discourse treebank 2.0","authors":"Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind K Joshi, Bonnie L Webber","year":"2008","venue":"LREC","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Qin et al., 2016] Lianhui Qin, Zhisong Zhang, and Hai Zhao. A stacking gated neural architecture for implicit discourse relation classification. In EMNLP, pages 2263-2270, 2016.","title":"A stacking gated neural architecture for implicit discourse relation classification","authors":"Qin","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Qin et al., 2017] Lianhui Qin, Zhisong Zhang, Hai Zhao, Zhiting Hu, and Eric P. Xing. Adversarial connective-exploiting net- works for implicit discourse relation classification. In ACL, pages 1006-1017, 2017.","title":"Adversarial connective-exploiting networks for implicit discourse relation classification","authors":"Qin","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rutherford and Xue, 2014] Attapol Rutherford and Nianwen Xue. Discovering implicit discourse relations through brown cluster pair representation and coreference patterns. In EACL, pages 645-654, 2014.","title":"Attapol Rutherford and Nianwen Xue. Discovering implicit discourse relations through brown cluster pair representation and coreference patterns","authors":"Xue Rutherford","year":"2014","venue":"EACL","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rutherford and Xue, 2015] Attapol Rutherford and Nianwen Xue. Improving the inference of implicit discourse relations via classi- fying explicit discourse connectives. In NAACL-HLT, pages 799- 808, 2015.","title":"Attapol Rutherford and Nianwen Xue. Improving the inference of implicit discourse relations via classifying explicit discourse connectives","authors":"Xue Rutherford","year":"2015","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Rutherford and Xue, 2016] Attapol Rutherford and Nianwen Xue. Robust non-explicit neural discourse parser in english and chi- nese. In CoNLL, pages 55-59, 2016.","title":"Attapol Rutherford and Nianwen Xue. Robust non-explicit neural discourse parser in english and chinese","authors":"Xue Rutherford","year":"2016","venue":"CoNLL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Shi and Demberg, 2019] Wei Shi and Vera Demberg. Next sen- tence prediction helps implicit discourse relation classification within and across domains. In EMNLP-IJCNLP, pages 5789- 5795, 2019.","title":"Next sentence prediction helps implicit discourse relation classification within and across domains","authors":"Wei Shi, Vera Demberg","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Varia et al., 2019] Siddharth Varia, Christopher Hidey, and Tuhin Chakrabarty. Discourse relation prediction: Revisiting word pairs with convolutional networks. In SIGDIAL, pages 442-452, 2019.","title":"Discourse relation prediction: Revisiting word pairs with convolutional networks","authors":"Siddharth Varia, Christopher Hidey, Tuhin Chakrabarty","year":"2019","venue":"SIGDIAL","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Par- mar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wang and Lan, 2015] Jianxiang Wang and Man Lan. A refined end-to-end discourse parser. In CoNLL, pages 17-24, 2015.","title":"Jianxiang Wang and Man Lan. A refined end-to-end discourse parser","authors":"Lan Wang","year":"2015","venue":"CoNLL","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Wang et al., 2017] Zhiguo Wang, Wael Hamza, and Radu Florian. Bilateral multi-perspective matching for natural language sen- tences. In IJCAI, pages 4144-4150, 2017.","title":"Bilateral multi-perspective matching for natural language sentences","authors":"Zhiguo Wang, Wael Hamza, Radu Florian","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/579"},{"position":31,"raw":"[Xu et al., 2018] Yang Xu, Yu Hong, Huibin Ruan, Jianmin Yao, Min Zhang, and Guodong Zhou. Using active learning to ex- pand training data for implicit discourse relation recognition. In EMNLP, pages 725-731, 2018.","title":"Using active learning to expand training data for implicit discourse relation recognition","authors":"Yang Xu, Yu Hong, Huibin Ruan, Jianmin Yao, Min Zhang, Guodong Zhou","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Xue et al., 2016] Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Attapol Rutherford, Bonnie L. Webber, Chuan Wang, and Hong- min Wang. Conll 2016 shared task on multilingual shallow dis- course parsing. In CoNL, pages 1-19, 2016.","title":"Conll 2016 shared task on multilingual shallow discourse parsing","authors":"Nianwen Xue, Tou Hwee, Sameer Ng, Attapol Pradhan, Bonnie L Rutherford, Chuan Webber, Hongmin Wang, Wang","year":"2016","venue":"CoNL","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2015] Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, and Junfeng Yao. Shallow convolutional neural network for implicit discourse relation recognition. In EMNLP, pages 2230-2235, 2015.","title":"Shallow convolutional neural network for implicit discourse relation recognition","authors":"Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, Junfeng Yao","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}