Neural Abstractive Summarization with Structural Attention

1 Introduction

Sequence-to-sequence (seq2seq) architectures with attention have led to tremendous success in many conditional language generation tasks such as machine translation [Sutskever et al., 2014; Bahdanau et al., 2015; Luong et al., 2015], dialog generation [Vinyals and Le, 2015], abstractive summarization [Rush et al., 2015; See et al., 2017; Paulus et al., 2018] and question answering [Yin et al., 2016]. Attention models that compute a context vector based on the entire input text at every decoder step have especially benefited abstractive summarization. This is because - (1) LSTM encoders struggle to encode long documents into just one vector from which one can decode the summary, and (2) generating summary involves a lot of copying from the source text which attention models easily facilitate that can also be made part of the training process [See et al., 2017; Nallapati et al., 2016].

Studies in abstractive summarization majorly focus on generating news summaries, and the CNN/Dailymail dataset [Hermann et al., 2015; Nallapati et al., 2016] is largely used as a benchmark. This is a very clean dataset with fluent and properly structured text collected from a restricted domain, and it hardly features any repetitive information. Therefore, the current models which have been built on this dataset do not need to account too much for repeated or contradicting information. Additionally, these models are trained on truncated documents (up to 400 words), and increasing the word length leads to a drop in performance of these models [See et al., 2017]. This is because either LSTM is unable to encode longer documents accurately or in this dataset, the first few sentences contain most of the summary information. However, real-world documents are usually much longer, and their summary information is spread across the entire document rather than in the first few sentences. In this paper, we explore one such dataset dealing with community question answering (CQA) summarization.

Multi-document summarization (MDS) is a well-studied problem with possible applications in summarizing tweets [Cao et al., 2016], news from varying sources [Yasunaga et al., 2017], reviews in e-commerce services [Ganesan et al., 2010], and so on. CQA services such as Quora, Yahoo! Answers, Stack Overflow help in curating information which may often be difficult to obtain directly from the existing resources on the web. However, a majority of question threads on these services harbor unstructured, often repeating or contradictory answers. Additionally, the answers vary in length from being a few words to a thousand words long. A large number of users with very different writing styles contribute to form these knowledge bases, which results in a diverse and challenging corpora. We define the task of CQA summarization as follows - given a list of such question threads, summarize the popular opinion reflected in them. This can be viewed as a multi-document summarization task [Chowdhury and Chakraborty, 2019].

We envision a solution to this problem by incorporating structural information present in language in the summarization models. Recent studies have shown promise in this direction. For example, [Fernandes et al., 2018] add graph neural networks (GNNs) [Li et al., 2015] to the seq2se encoder model for neural abstractive summarization. [Liu et al., 2019] model documents as multi-root dependency trees, each node representing a sentence; and pick the induced tree roots to be summary sentences in an extractive setting. Recently, [Song et al., 2018] attempt to preserve important structural dependency relations obtained from syntactic parse trees of the input text in the summary by using a structure infused copy mechanism. However, obtaining labeled data to explicitly model such dependency relations is expensive. Here, we attempt to mitigate this issue by proposing a structural encoder for summarization based on the prior work on structure attention networks [Liu and Lapata, 2018] which implicitly incorporates structural information within end-to-end training.

Our major contributions in this work are three-fold:

1. We enhance the pointer-generator architecture by adding a structural attention based encoder to implicitly capture long term dependency relations in summarization of lengthy documents.
2. We further propose a hierarchical encoder with multilevel structural attention to capture document-level discourse information in the multi-document summarization task.
3. We introduce multi-level contextual attention in the structural attention setting to enable word level copying and to generate more abstractive summaries, compared to similar architectures.

We compare our proposed solution against the popular pointer-generator model [See et al., 2017] and a few other summarization models derived from it in both single and multi-document settings. We show that our structural encoder architecture beats the strong pointer-generator baseline by 1.31 ROUGE-1 points on the CNN/Dailymail dataset and by 7.8 ROUGE-1 points on the concatenated CQA dataset for single document summarization (SDS). Our hierarchical structural encoder architecture further beats the concatenated approach by another 1.6 ROUGE-1 points on the CQA dataset. A qualitative analysis of the generated summaries observe considerable qualitative gains after inclusion of structural attention. Our structural attention based summarinzation model is one of the few abstractive approaches to beat extractive baselines in MDS. The code is public at https: //bit.ly/35i7q93.

2 Models

In this section, we first describe the seq2seq based pointergenerator architecture proposed by [See et al., 2017; Nallapati et al., 2016]. We then elaborate on how we incorporate structural attention into the mechanism and generate nonprojective dependency trees to capture long term dependencies within documents. Following this, we describe our hierarchical encoder architecture and propose multi-level contextual attention to better model discourse for MDS.

2.1 Pointer-Generator Model

The pointer-generator architecture (PG) [See et al., 2017] serves as the starting point for our approach. PG is inspired from the LSTM encoder-decoder architecture [Nallapati et al., 2016] for the task of abstractive summarization. Tokens are fed to a bidirectional encoder to generate hidden representations h, and a unidirectional decoder is used to generate the summary. At the decoding step t, the decoder uses an attention mechanism [Bahdanau et al., 2015] to compute an attention vector at for every encoder hidden state. These attention vectors are pooled to compute a context vector ct for every decoder step. The context vector ct at that step along with the hidden representations h are passed through a feedforward network followed by softmax to give a multinomial distribution over the vocabulary pvocab(w).

Additionally, [See et al., 2017] introduce a copy mechanism [Vinyals et al., 2015] in this model. At each decoding step t, it predicts whether to generate a new word or copy the word based on the attention probabilities at using a probability value pgen. This probability is a function on the decoder input xt, context vector ct, and decoder state st . pgen is used as a soft switch to choose between copying and generation as shown below:

This model is trained by minimizing the negative loglikelihood of the predicted sequence normalized by the sequence length: L = - 1 T ∑Tt=0 log p(w∗ t ), where T is the length of the generated summary. Once L gets converged, an additional coverage loss is used to train the model further, which aims to minimize repetitions in the generated summaries. We refer the readers to [See et al., 2017] for more details.

2.2 Structural Attention

We observe that while the pointer-generator architecture performs well on the news corpora, it fails to generate meaningful summaries on complex datasets. We hypothesize that taking into account intra-document discourse information will help in generating better summaries. We propose an architecture that implicitly incorporates such information using document structural representation [Liu and Lapata, 2018; Kim et al., 2017] to include richer structural dependencies in the end-to-end training.

Proposed Model

We model our document as a non-projective dependency parse tree by constraining inter-token attention as weights of the dependency tree. We use the Matrix tree theorem [Koo et al., 2007; Tutte, 1984] to carry out the same.

As shown in Figure 1, we feed our input tokens (wi) to a bi-LSTM encoder to obtain hidden state representations hi. We decompose the hidden state vector into two parts: di and ei, which we call the structural part and the semantic part, respectively:

For every pair of two input tokens, we transform their structural parts d and try to compute the probability of a parent-child relationship edge between them in the dependency tree. For tokens j and k, this is done as:

where Wp and Wc are learned. Next, we compute an intertoken attention function fjk as follows:

where Wa is also learned. For a document with K tokens, f is a K ×K matrix representing inter-token attention. We model each token as a node in the dependency tree and define the probability of an edge between tokens at positions j and k, P(zjk = 1), which is given as,

where δ(x, y) = 1 when x = y . We denote P(zjk = 1) by ajk (structural attention). Let arj be the probability of the j th

We use this (soft) dependency tree formulation to compute a structural representation r for each encoder token as,

Thus, for encoder step i, we now obtain a structure infused hidden representation ri. We then compute the contextual attention for each decoding time step t as,

Now, using atstruct , we can compute a context vector similar to standard attentional model by weighted sum of the hidden state vectors as ctstruct = ∑ni=1 atstructi hi. At every decoder time step, we also compute the basic contextual attention vector at (without structure incorporation), as discussed previously. We use ctstruct to compute Pvocab and pgen. We, however, use the initial attention distribution at to compute P(w) in order to facilitate token level pointing.

2.3 Hierarchical Model

The structural attention model, while efficient, requires O(K2 ) memory to compute attention, where K is the length of the input sequence making it memory intensive for long documents. Moreover, in CQA, answers reflect different opinions of individuals. In such a case, concatenating answers results in user specific information loss. To model discourse structure better in case of conflicting (where one answer contradicts other), overlapping or varying opinion, we introduce a hierarchical encoder model based on structural attention (Figure 2). We feed each answer independently to a bi-LSTM encoder and obtain token level hidden representations hidx,tidx where idx is the document (or answer) index and tidx is the token index. We then transform these representations into structure infused vectors ridx,tidx as described previously. For each answer in the CQA question thread, we pool the token representations to obtain a composite answer vector ridx. We consider three types of pooling: average, max and sum, out of which sum pooling performs the best after initial experiments. We feed these structure infused answer embeddings to an answer-level bi-LSTM encoder and obtain higher-level encoder hidden states hidx from which we calculate structure infused embeddings gidx. At the decoder time step t, we calculate contextual attention at answer as well as token level as follows:

We use the answer-level attention distribution atans to compute the context vector at every decoder time step which we use to calculate pvocab and pgen as described before.

To enable copying, we use atoken. The final probability of predicting word w is given by,

3 Datasets

We primarily use two datasets to evaluate the performance:

(i) The CNN/Dailymail 1 dataset [Hermann et al., 2015; Nallapati et al., 2016] is a news corpora containing document-summary pairs. Bulleted extracts from the CNN and Dailymail news pages online are projected as summaries to the remaining documents. The scripts released by [Nallapati et al., 2016] are used to extract approximately 250k training pairs, 13k validation pairs and 11.5k test pairs from the corpora. We use the non-anonymized form of the data to provide fair comparison to the experiments, conducted by [See et al., 2017]. It is a factual, English corpora with an average document length being 781 tokens and an average summary length being 56 tokens. We use two versions of this dataset - one with 400 word articles, and the other with 800 word articles. Most research reporting results on this dataset are on CNN/Dailymail-400. We also consider a 800 token version of this dataset as longer articles harbor more intra-document structural dependencies which would allow us to better demonstrate the benefits of structure incorporation. Moreover, longer documents resemble real-world datasets. (ii) We also use the CQA dataset 2 [Chowdhury and Chakraborty, 2019] which is generated by filtering the Yahoo! Answers L6 corpora to find question threads where the best answer can serve as a summary for the remaining answers. The authors use a series of heuristics to arrive at a set of 100k question thread-summary pairs. The summaries are generated by modifying best answers and selecting most question-relevant sentences from them. The remaining answers serve as candidate documents for summarization making up a large-scale, diverse and highly abstract dataset. On an average, the corpus has 12 answers per question thread, with 65 words per answer. All summaries are truncated at hundred words. We split the 100k dataset into 80k training instances, 10k validation and 10k test instances. We additionally extract the upvote information corresponding to every answer from the L6 dataset and assume that upvotes have a high correlation with the relative importance and relevance of an answer. We then rank answers in decreasing order of upvotes before concatenating them as required for several baselines. Since Yahoo! Answers is an unstructured and unmoderated question-answer repository, this has turned out to be a challenging summarization dataset [Chowdhury and Chakraborty, 2019]. Additionally, we include analysis on MultiNews [Fabbri et al., 2019], a news based MSD corpora to aid similar studies. It is the first large scale MSD news dataset consisting of 56,216 article summary pairs, crowd-sourced from various different news websites.

1 https://github.com/abisee/cnn-dailymail

2 https://bitbucket.org/tanya14109/cqasumm/src/master/

4 CompetingMethods

We compare the performance of the following SDS and MDS models.

• Lead3: It is an extractive baseline where the first 100 tokens of the document (in case of SDS datasets) and concatenated ranked documents (in case of MDS datasets) are picked to form the summary.
• KL-Summ: It is an extractive summarization method introduced by [Haghighi and Vanderwende, 2009] that attempts to minimize KL-Divergence between candidate documents and generated summary.
• LexRank: It is an unsupervised extractive summarization method [Erkan and Radev, 2004]. A graph is built with sentences as vertices, and edge weights are assigned based on sentence similarity.
• TextRank: It is an unsupervised extractive summarization method which selects sentences such that the information being disseminated by the summary is as close as possible to the original documents [Mihalcea and Tarau, 2004].
• Pointer-Generator (PG): It is a supervised abstractive summarization model [See et al., 2017], as discussed earlier. It is a strong and popularly used baseline for summarization.
• Pointer-Generator + Structure Infused Copy (PG+SC): Our implementation is similar to one of the methods proposed by [Song et al., 2018]. We explicitly compute the dependency tree of sentences and encode a structure vector based on features like POS tag, number of incoming edges, depth of the tree, etc. We then concatenate this structural vector for every token to its hidden state representation in pointer-generator networks.
• Pointer-Generator+MMR (PG+MMR): It is an abstractive MDS model [Lebanoff et al., 2018] trained on the CNN/Dailymail dataset. It combines Maximal Marginal Relevance (MMR) method with pointer-

generator networks, and shows significant performance on DUC-04 and TAC-11 datasets.

• Hi-MAP: It is an abstractive MDS model by [Fabbri et al., 2019] extending PG and MMR.
• Pointer-Generator + Structural Attention (PG+SA): It is the model proposed in this work for SDS and MDS tasks. We incorporate structural attention with pointer generator networks and use multi-level contextual attention to generate summaries.
• Pointer-Generator + Hierarchical Structural Attention (PG+HSA): We use multi-level structural attention to additionally induce a document-level non-projective dependency tree to generate more insightful summaries.

5 Experimental Results

5.1 Quantitative Analysis

We compare the performance of the models on the basis of ROUGE-1, 2 and L F1-scores [Lin, 2004] on the CNN/Dailymail (Table 2), CQA and Multinews (Table 3) datasets. We observe that infusion of structural information leads to considerable gains over the basic PG architecture on both the datasets. Our approach fares better than explicitly incorporating structural information [Song et al., 2018]. The effects of incorporating structure is more significant in the CQA dataset (+7.8 ROUGE-1 over PG) as compared to the CNN/Dailymail dataset (+0.49 ROUGE-1 over PG). The benefit is comparatively prominent in CNN/Dailymail-800 (+1.31 ROUGE-1 over PG) as compared to CNN-Dailymail400 (+0.49 ROUGE-1 over PG).

5.2 Qualitative Analysis

We ask human evaluators 3 to compare CQASUMM summaries generated by competing models on the ground of content and readability. Here we observe a significant gain with structural attention. PG has difficulty in summarizing articles with repetitive information and tends to assign lower priority to facts repeating across answers. Methods like LexRank, on the other hand, tend to mark these facts the most important. Incorporating structure solves this problem to some extent (due to the pooling operations). We find that PG sometimes picks sentences containing opposing opinions for the same summary. We find this occurrence to be less frequent in the structural attention models. This phenomenon is illustrated with an instance from the CQA dataset in Table 4.

5.3 Model Diagnosis and Discussion

Tree depth. The structural attention mechanism has a tendency to induce shallow parse trees with high tree width. This leads to highly spread out trees especially at the root.

MDS baselines. While supervised abstractive methods like PG significantly outperform unsupervised non-neural methods in SDS, we find them to be inferior or similar in performance in MDS. This has also been reported by recent MDS related studies such as [Lebanoff et al., 2018] where LexRank is shown to significantly beat basic PG on the DUC04 and TAC-11 datasets. This justifies the choice of largely unsupervised baselines in recent MDS related studies such as [Yasunaga et al., 2017; Lebanoff et al., 2018].

Dataset layout. We observe that while increasing the length of the input documents lowers ROUGE score for the CNN/Dailymail dataset, it boosts the same in the CQA dataset. This might be attributed to the difference in information layout in the two datasets. The CNN/Dailymail has a very high Lead-3 score signifying summary information concentration within the first few lines of the document.

Coverage with structural attention. Training a few thousand iterations with coverage loss is known to significantly reduce repetition. However, in the structural attention models, while training with the CQA dataset we observe that soon after coverage loss has converged, on further training the repetition in generated summaries starts increasing again. Future work would include finding a more appropriate coverage function for datasets with repetition.

3 The evaluators were experts in NLP, and their age ranged between 20-30 years.

6 RelatedWork

Neural Abstractive Summarization

PG-based models. Since the studies conducted by [Nallapati et al., 2016; See et al., 2017], many approaches to neural abstractive summarization have extended them in different ways. [Cohan et al., 2018] extend the approach proposed by [See et al., 2017] with a hierarchical encoder to model discourse between various sections of a scientific paper using PubMed dataset.

MDS. Most studies on MDS were performed during DUC (2001-2007) and TAC (2008-2011) workshops. Opinosis [Ganesan et al., 2010] uses word-level opinion graphs to find cliques and builds them into summaries in highly redundant opinion sentences. Recently, [Lebanoff et al., 2018] adapt the single document architecture to a multi-document setting by using maximal marginal relevance method to select representative sentences from the input and feed to the encoderdecoder architecture. [Nema et al., 2017] propose a query based abstractive summarization approach where they first encode both the query and the document, and at each decoder time step, calculate attention based on both.

Structure Infused Document Modeling

There have been numerous studies to incorporate structure in document representations by adding syntactic parsing to pipeline like architectures. However, external parser computation is costly and does not scale well. [Tang et al., 2015a; Tang et al., 2015b] are one of the first to obtain document representations by first computing sentence representations and hierarchically aggregating them. [Yang et al., 2016] propose a model to implicitly add structural information within endto-end training by assigning attention scores to each sentence according to the context at that state. [Kim et al., 2017] encoded structural information as graphical models within deep neural networks to facilitate end-to-end training. They use two graphical attention structures - linear chain conditional random fields and graph-based parsing models, and show how both of them can be implemented as layers of a neural network. Recent work by [Balachandran et al., 2020] proposes a similar idea by incorporating both latent and explicit sentence dependencies into single document summarization architecture. They further explicitly induce structure by injecting a coreferring mention graph across sentences.

7 Conclusion

In this work, we proposed an approach to incorporate structural attention within end-to-end training in summarization networks. We achieved a considerable improvement in terms of ROUGE scores compared to our primary baseline model on both the CNN/Dailymail and CQA datasets. We also introduced multi-level contextual attention which helped in generating more abstract summaries. The analysis hinted that incorporation of some form of structural attention might be the key to achieve significant improvement compared to the extractive counterpart in the complex multi-document summarization task.