{"type":"ScholarlyArticle","title":"Neural Abstractive Summarization with Structural Attention","authors":[{"name":"Tanya Chowdhury","slug":"tanya-chowdhury","given":"Tanya","family":"Chowdhury","affiliation":"IIIT-Delhi , India"},{"name":"Sachin Kumar","slug":"sachin-kumar","given":"Sachin","family":"Kumar","affiliation":"Carnegie Mellon University , USA"},{"name":"Tanmoy Chakraborty","slug":"tanmoy-chakraborty","given":"Tanmoy","family":"Chakraborty","affiliation":"IIIT-Delhi , India"}],"abstract":"Attentional, RNN-based encoder-decoder architectures have obtained impressive performance on abstractive summarization of news articles. However, these methods fail to account for long term dependencies within the sentences of a document. This problem is exacerbated in multi-document summarization tasks such as summarizing the popular opinion in threads present in community question answering (CQA) websites such as Yahoo! Answers and Quora. These threads contain answers which often overlap or contradict each other. In this work, we present  a hierarchical encoder based on structural attention to model such inter-sentence and inter-document dependencies. We set the popular pointer-generator architecture and some of the architectures derived from it as our baselines and show that they fail to generate good summaries in a multi-document setting. We further illustrate that our proposed model achieves significant improvement  over the baseline in both single and multi-document summarization settings -- in the former setting, it beats the  baseline by 1.31 and 7.8 ROUGE-1 points on CNN and CQA datasets, respectively; in the latter setting, the performance is further improved by   1.6 ROUGE-1 points on the CQA dataset.","doi":"10.24963/ijcai.2020/514","url":"https://ijcai.org/proceedings/2020/514","pdf":"https://ijcai.org/proceedings/2020/0514.pdf","markdown":"https://ijcai.org/proceedings/2020/514.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/514","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":514,"pages":{"first":3716,"last":3722},"keywords":["Natural Language Processing: Other","Natural Language Processing: Natural Language Summarization"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24967/file_24967.m3u8","length_sec":307,"provider":"hls"}],"references":[{"position":1,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Balachandran et al., 2020] Vidhisha Balachandran, Artidoro Pagnoni, Jay Yoon Lee, Dheeraj Rajagopal, Jaime Carbonell, and Yulia Tsvetkov. Structsum: Incorporating latent and explicit sentence dependencies for single document summarization. arXiv preprint arXiv:2003.00576, 2020.","title":"Structsum: Incorporating latent and explicit sentence dependencies for single document summarization","authors":"Balachandran","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cao et al., 2016] Ziqiang Cao, Chengyao Chen, Wenjie Li, Sujian Li, Furu Wei, and Ming Zhou. Tgsum: Build tweet guided multi- document summarization dataset. In AAAI, pages 1-8, 2016.","title":"Tgsum: Build tweet guided multidocument summarization dataset","authors":"Ziqiang Cao, Chengyao Chen, Wenjie Li, Sujian Li, Furu Wei, Ming Zhou","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chowdhury and Chakraborty, 2019] Tanya Chowdhury and Tan- moy Chakraborty. Cqasumm: Building references for com- munity question answering summarization corpora. In CODS- COMAD, pages 18-26. ACM, 2019.","title":"Cqasumm: Building references for community question answering summarization corpora","authors":"Tanya Chowdhury, Tanmoy Chakraborty","year":"2019","venue":"CODS-COMAD","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cohan et al., 2018] Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Go- harian. A discourse-aware attention model for abstractive sum- marization of long documents. In NAACL, pages 615-621, 2018.","title":"A discourse-aware attention model for abstractive summarization of long documents","authors":"Arman Cohan, Franck Dernoncourt, Soon Doo, Trung Kim, Seokhwan Bui, Walter Kim, Nazli Chang, Goharian","year":"2018","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Erkan and Radev, 2004] Günes Erkan and Dragomir R Radev. Lexrank: Graph-based lexical centrality as salience in text sum- marization. Journal of artificial intelligence research, 22:457- 479, 2004.","title":"Lexrank: Graph-based lexical centrality as salience in text summarization","authors":"Radev ; ] Günes Erkan, Erkan, Dragomir R Radev","year":"2004","venue":"Journal of artificial intelligence research","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fabbri et al., 2019] Alexander R Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R Radev. Multi-news: a large-scale multi-document summarization dataset and abstractive hierarchi- cal model. arXiv preprint arXiv:1906.01749, 2019.","title":"Multi-news: a large-scale multi-document summarization dataset and abstractive hierarchical model","authors":"Irene Alexander R Fabbri, Tianwei Li, Suyi She, Li, Dragomir R Radev","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Fernandes et al., 2018] Patrick Fernandes, Miltiadis Allamanis, and Marc Brockschmidt. Structured neural summarization. arXiv preprint arXiv:1811.01824, 2018.","title":"Structured neural summarization","authors":"Patrick Fernandes, Miltiadis Allamanis, Marc Brockschmidt","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ganesan et al., 2010] Kavita Ganesan, ChengXiang Zhai, and Ji- awei Han. Opinosis: A graph based approach to abstractive summarization of highly redundant opinions. In COLING, pages 340-348, 2010.","title":"Opinosis: A graph based approach to abstractive summarization of highly redundant opinions","authors":"Kavita Ganesan, Chengxiang Zhai, Jiawei Han","year":"2010","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Haghighi and Vanderwende, 2009] Aria Haghighi and Lucy Van- derwende. Exploring content models for multi-document sum- marization. In NAACL, pages 362-370. Association for Compu- tational Linguistics, 2009.","title":"Aria Haghighi and Lucy Vanderwende. Exploring content models for multi-document summarization","authors":"Vanderwende Haghighi","year":"2009","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hermann et al., 2015] Karl Moritz Hermann, Tomas Kocisky, Ed- ward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In NIPS, pages 1693-1701, 2015.","title":"Teaching machines to read and comprehend","authors":"Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, Phil Blunsom","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim et al., 2017] Yoon Kim, Carl Denton, Luong Hoang, and Alexander M Rush. Structured attention networks. arXiv preprint arXiv:1702.00887, 2017.","title":"Structured attention networks","authors":"Yoon Kim, Carl Denton, Luong Hoang, Alexander M Rush","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Koo et al., 2007] Terry Koo, Amir Globerson, Xavier Car- reras Pérez, and Michael Collins. Structured prediction models via the matrix-tree theorem. In EMNLP-CoNLL, pages 141-150, 2007.","title":"Structured prediction models via the matrix-tree theorem","authors":"Terry Koo, Amir Globerson, Xavier Carreras Pérez, Michael Collins","year":"2007","venue":"EMNLP-CoNLL","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lebanoff et al., 2018] Logan Lebanoff, Kaiqiang Song, and Fei Liu. Adapting the neural encoder-decoder framework from sin- gle to multi-document summarization. In EMNLP, pages 4131- 4141, 2018.","title":"Adapting the neural encoder-decoder framework from single to multi-document summarization","authors":"Lebanoff","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2015] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493, 2015.","title":"Gated graph sequence neural networks","authors":"Yujia Li, Daniel Tarlow, Marc Brockschmidt, Richard Zemel","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lin, 2004] Chin-Yew Lin. Rouge: A package for automatic evalu- ation of summaries. In Proc. ACL workshop on Text Summariza- tion Branches Out, pages 1-10, 2004.","title":"Rouge: A package for automatic evaluation of summaries","authors":"Chin-Yew Lin","year":"2004","venue":"Proc. ACL workshop on Text Summarization Branches Out","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu and Lapata, 2018] Yang Liu and Mirella Lapata. Learning structured text representations. TACL, 6:63-75, 2018.","title":"Learning structured text representations","authors":"Lapata Liu, Yang Liu, Mirella Lapata","year":"2018","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2019] Yang Liu, Ivan Titov, and Mirella Lapata. Single document summarization as tree induction. In NAACL, pages 1745-1755, Minneapolis, Minnesota, June 2019.","title":"Single document summarization as tree induction","authors":"Liu","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Luong et al., 2015] Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention-based neural ma- chine translation. In EMNLP, pages 1412-1421, 2015.","title":"Effective approaches to attention-based neural machine translation","authors":"Thang Luong, Hieu Pham, Christopher D Manning","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mihalcea and Tarau, 2004] Rada Mihalcea and Paul Tarau. Tex- trank: Bringing order into text. In EMNLP, pages 1-10, 2004.","title":"Textrank: Bringing order into text","authors":"Tarau Mihalcea, Rada Mihalcea, Paul Tarau","year":"2004","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Nallapati et al., 2016] Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, C ¸aglar Gulc ¸ehre, and Bing Xiang. Abstractive text sum- marization using sequence-to-sequence RNNs and beyond. In SIGNLL, pages 280-290, 2016.","title":"Abstractive text summarization using sequence-to-sequence RNNs and beyond","authors":"Ramesh Nallapati, Bowen Zhou, C Cicero Dos Santos, Bing ¸aglar Gulc ¸ehre, Xiang","year":"2016","venue":"SIGNLL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nema et al., 2017] Preksha Nema, Mitesh M. Khapra, Anirban Laha, and Balaraman Ravindran. Diversity driven attention model for query-based abstractive summarization. In ACL, pages 1063-1072, 2017.","title":"Diversity driven attention model for query-based abstractive summarization","authors":"Preksha Nema, M Mitesh, Anirban Khapra, Balaraman Laha, Ravindran","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Paulus et al., 2018] Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. In ICLR, 2018.","title":"A deep reinforced model for abstractive summarization","authors":"Romain Paulus, Caiming Xiong, Richard Socher","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Rush et al., 2015] Alexander M. Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence sum- marization. In EMNLP, page 379-389, 2015.","title":"A neural attention model for abstractive sentence summarization","authors":"Alexander M Rush, Sumit Chopra, Jason Weston","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[See et al., 2017] Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer- generator networks. In ACL, pages 1073-1083, 2017.","title":"Get to the point: Summarization with pointergenerator networks","authors":"Abigail See, Peter J Liu, Christopher D Manning","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Song et al., 2018] Kaiqiang Song, Lin Zhao, and Fei Liu. Structure-infused copy mechanisms for abstractive summariza- tion. In COLING, pages 1717-1729, 2018.","title":"Structure-infused copy mechanisms for abstractive summarization","authors":"Song","year":"2018","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NIPS, pages 3104-3112, 2014.","title":"Quoc V Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Tang et al., 2015a] Duyu Tang, Bing Qin, and Ting Liu. Document modeling with gated recurrent neural network for sentiment clas- sification. In EMNLP, pages 1422-1432, 2015.","title":"Document modeling with gated recurrent neural network for sentiment classification","authors":"Tang","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tang et al., 2015b] Duyu Tang, Bing Qin, and Ting Liu. Learn- ing semantic representations of users and products for document level sentiment classification. In ACL-IJCNLP, pages 1014- 1023, 2015.","title":"Learning semantic representations of users and products for document level sentiment classification","authors":"Tang","year":"2015","venue":"ACL-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tutte, 1984] William Thomas Tutte. Graph theory, vol. 21. Ency- clopedia of Mathematics and its Applications, 1984.","title":"Graph theory","authors":"William Thomas, Tutte","year":"1984","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Vinyals and Le, 2015] Oriol Vinyals and Quoc V. Le. A neural conversational model. CoRR, abs/1506.05869, 2015.","title":"A neural conversational model","authors":"Oriol Vinyals, V Quoc, Le","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In NIPS, pages 2692-2700, 2015.","title":"Pointer networks","authors":"Oriol Vinyals, Meire Fortunato, Navdeep Jaitly","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Yang et al., 2016] Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. Hierarchical attention net- works for document classification. In NAACL, pages 1480-1489, 2016.","title":"Hierarchical attention networks for document classification","authors":"Yang","year":"2016","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Yasunaga et al., 2017] Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Pareek, Krishnan Srinivasan, and Dragomir Radev. Graph-based neural multi-document summarization. In CoNLL, pages 452-462, 2017.","title":"Graph-based neural multi-document summarization","authors":"Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Pareek, Krishnan Srinivasan, Dragomir Radev","year":"2017","venue":"CoNLL","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Yin et al., 2016] Jun Yin, Xin Jiang, Zhengdong Lu, Lifeng Shang, Hang Li, and Xiaoming Li. Neural generative question answer- ing. In Proceedings of the Workshop on Human-Computer Ques- tion Answering, ACL, pages 36-42, San Diego, California, June 2016.","title":"Neural generative question answering","authors":"Jun Yin, Xin Jiang, Zhengdong Lu, Lifeng Shang, Hang Li, Xiaoming Li","year":"2016","venue":"Proceedings of the Workshop on Human-Computer Question Answering, ACL","doi":null,"url":null,"internal_url":"/proceedings/2016/422"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}