Dialogue Discourse-Aware Graph Model and Data Augmentation for Meeting Summarization

1 Introduction

Meeting summarization aims to distill the most important in-

formation from a recorded meeting into a short textual passage, which could be of great value to people by providing quick access to the essential content of past meetings [Gurevych and Strube, 2004]. An example is shown in Figure 1, which includes 3 speakers (A, B and C) and their corresponding utterances, as well as a human-written summary.

Recently, neural-based sequence-to-sequence methods have become the mainstream mode for generating meeting summaries via accurately modeling the refined semantic representation of the original meeting text. Goo and Chen [2018] incorporate utterance-level dialogue acts to enhance the representation for each utterance. Li et al. [2019] consider topics as a structure information to enrich the meeting representation. However, we claim that the current research suffers from two problems. One is sequential text modeling. A meeting is a dynamic information exchange flow, which is informal, verbose, and less structured than traditional documents [Sacks et al., 1978]. But all previous works adopt a sequential modeling strategy for encoding the meeting, hindering the exploration of inherently rich interactive relations between utterances, which makes the meeting modeling inadequate. The other one is the lack of sufficient training data. As we know, the scale of corpus is significant to the performance of neural models, the data size of meeting summarization is only onethousandth of the traditional news summarization (e.g. 137 meetings in AMI v.s. 312K articles in CNN/DM).

To address the above concerns, we draw support from dialogue discourse, a dialogue-specific structure, which can provide pre-defined relations between utterances [Stone et al., 2013]. As shown in Figure 1, QA, Contrast and Continuation are three dialogue discourse relations, which explicitly show the information flow and interactions between utterances. After knowing these relations, we can generate better summaries. Furthermore, we find that a question often sparks a discussion in a meeting. As shown in Figure 1, the discussion about "design" and "cost" surrounds the salient term "battery charger" in the question. Therefore, we assume that a question tends to contain core topics or key concepts that can be used as a pseudo summary for subsequent discussion.

In this paper, we propose a Dialogue Discourse-Aware Meeting Summarizer (DDAMS, see §3) for modeling the dialogue discourse relations among utterances. Specifically, we first convert meeting utterances with discourse relations into a meeting graph where utterance vertices interact with relation vertices. Then we design a graph-to-sequence framework to generate meeting summaries. Besides, to tackle the insufficient training problem, we devise a Dialogue DiscourseAware Data Augmentation (DDADA, see §4) strategy, which constructs a pseudo-summarization corpus from existing input meetings. In detail, we use the QA discourse relation to identify the question as pseudo-summary, then we view subsequent utterances with associated discourse relations as a pseudo-meeting. Finally, we can construct a pseudosummarization corpus, which is 20 times larger than the original dataset and can be used to pretrain DDAMS.

We conduct various experiments on the widely used AMI [Carletta et al., 2005] and ICSI [Janin et al., 2003] datasets. The results show the effectiveness of DDAMS method and DDADA strategy. In summary, (1) We make the first attempt to successfully explore dialogue discourse to model the utterances interactions for meeting summarization; (2) We devise a dialogue discourse-aware data augmentation strategy to alleviate the data insufficiency problem; (3) Extensive experiments show that our model achieves SOTA performance.

2 Preliminaries

In this section, we describe the task definition and give a brief introduction to dialogue discourse.

2.1 Task Definition

Meeting summarization aims at producing a summary Y for the input meeting U, where U consists of |U| utterances [u1, u2, ...u|U|] and Y consists of |Y| words [y1, y2, ...y|Y|]. The i-th utterance of the meeting can be represented as a sequence of words ui = [ui,1, ui,2, ...ui,|ui|], where ui,j denotes the j-th word of i-th utterance. Each utterance ui associates with a speaker pi ∈ P , P being a set of participants.

2.2 Dialogue Discourse

Dialogue discourse indicates relations between discourse units in a conversation (i.e. utterances in a meeting). This dependency-based structure allows relations between nonadjacent utterances that is applicable for multi-party conversations [Li et al., 2021]. There are 16 discourse relations [Asher et al., 2016] in total: comment, clarificationquestion, elaboration, acknowledgment, continuation, explanation, conditional, QA, alternation, question-elaboration, result, background, narration, correction, parallel, contrast.

3 Dialogue Discourse-Aware Meeting Summarizer

In this section, we present our Dialogue Discourse-Aware Meeting Summarizer (DDAMS) that consists of four components: a meeting graph, a vertex representation module, a graph encoder, and a pointer decoder, as shown in Figure 3.

3.1 Meeting Graph

Given meeting utterances, we first use a SOTA dialogue discourse parser [Shi and Huang, 2019] to get discourse relations, one relation connects from one utterance to another one with a relation type, as shown in Figure 2(a).

Then, we perform the Levi graph transformation, which turns labeled edges into additional vertices [Gross et al., 2013]. Through this transformation, we can model discourse relations explicitly and update the utterance and discourse vertices simultaneously. There are two types of edges in the Levi graph: default and reverse. For example, an edge (U3, Contrast, U1) in the original graph becomes (U3, default, Contrast) and (Contrast, default, U1) in the Levi graph, as shown in Figure 2(b).

Furthermore, to aggregate non-local information, a global vertex is added that connects all vertices by global edges and will be used to initialize the decoder. We also add self edges to aggregate self-information, as shown in Figure 2(c).

Note that different types of vertices may have different features that fall in different space [Beck et al., 2018]. Specifically, previous works ignore the type of source and target vertices and use the same type of edge to pass information, which may reduce the effectiveness of discourse information. Thus, we propose our meeting graph which transforms the default edge into default-in-discourse and default-out-discourse edges and the reverse edges into reverse-in-discourse and reverse-out-discourse edges, as shown in Figure 2(d).

Let GM = (VM, EM,RM) denote a meeting graph, with vertices vi ∈ VM and labeled edges (vi, r, vj) ∈ EM, where r ∈ RM is the relation type of the edge. For our meeting graph, there are six types of relations, where RM becomes default-in-discourse, default-out-discourse, reversein-discourse, reverse-out-discourse, global, self.

3.2 Vertex Representation

Vertex representation module aims to obtain initial representations for three types of vertices: global vertex, relation vertex and utterance vertex. For global vertex and relation vertices, we get the initial representation h 0i by looking up from an embedding table. For utterance vertices, we employ a BiLSTM 1 as the utterance encoder that encodes the utterance forwardly and backwardly [Qin et al., 2020]. As --→ hi,j = LSTMf (---→ h i,j-1, e i,j ), ←-- hi,j = LSTMb (←--- h i,j+1, e i,j ), hi,j = [--→ h i,j ; ←-- hi,j], where hi,j and ei,j denote the hidden state and embedding of word ui,j. ; denotes concatenation. To involve speaker information, we encode speaker pi using a one-hot vector and get ei,j by concatenating word embedding and one-hot speaker embedding [Wang et al., 2020]. The representation h 0i =[---→ h i,|ui | ; ←-- hi,1] is used as input to the graph encoder.

3.3 Graph Encoder

After getting the initial feature h 0i of each vertex vi ∈ VM, we feed them into the graph encoder to digest the structural information. We use relational graph convolutional networks [Schlichtkrull et al., 2018] to capture high-level hidden features considering different types of edges. The convolutional computation for vi at the (l +1)-th layer takes the representation h (l) at the l-th layer as input can be defined as:

where Nri denotes the set of neighbors of vertex vi under relation r andW (l) r denotes relation-specific learnable parameters at the l-th layer.

However, uniformly accepting information from different discourse relations is not suitable for identifying important discourse. Thus, we use the gate mechanism [Marcheggiani and Titov, 2017] to control the information passing:

where W (l) g,r denotes a learnable parameter under relation type r at the l-th layer. Equipped with the gate mechanism, the convolutional computation can be defined as:

1 We have also tried Transformer [Vaswani et al., 2017] as our backbone model. However, it has too many parameters which can easily over-fit on the small meeting dataset.

3.4 Pointer Decoder

We use a standard LSTM decoder with attention and copy mechanism to generate the summary [Bahdanau et al., 2015; See et al., 2017]. The global representation is used to initialize the decoder. At each step t, the decoder receives the word embedding of previous word and has decoder state st. The attention distribution is calculated as in [Luong et al., 2015]. We consider both word-level and utterance-level attention. The word-level context vector h wl t is computed as:

whereWa is a learnable parameter and h 0i,j is obtained from utterance encoder for ui,j . The utterance-level context vector h ul t is calculated similarly to the word-level context vector, except that we use the final outputs of the graph encoder h (l) i which represent utterances to calculate the attention distribution. The final context vector is the concatenation of wordlevel and utterance-level context vector h ∗t = [h wl t ; h ul t ], which is then used to calculate generation probability and the final probability distribution [See et al., 2017].

3.5 Training Objective

We use maximum likelihood estimation to train our model. Given the ground truth summary Y ∗ = [y∗ 1 , y∗ 2 , ..., y∗ |Y ∗ | ] for an input meeting U. We minimize the negative log-likelihood of the target words sequence:

4 Dialogue Discourse-Aware Data Augmentation

In this section, we introduce Dialogue Discourse-Aware Data Augmentation strategy (DDADA), which constructs a pseudo-summarization corpus from the original input meetings based on the QA discourse relation.

4.1 Pseudo-summarization Corpus Construction

Given a meeting and its corresponding discourse relations, we find a question often sparks a discussion and contains salient terms or concepts expressed in the discussion. As shown in Figure 4, speaker A asked "What's the standard colour?", other participants start to discuss this small topic. Thus, we can view the discussion as a pseudo meeting and the question as a pseudo summary of this pseudo meeting 2 .

According to the above observation, we collect pseudosummarization data from the original meeting summarization dataset where the question identified by QA discourse serves as the pseudo summary and N utterances after the question with their associated discourse relations serve as the pseudo meeting 3 . Note that there are some uninformative and normal questions such as "what is this here", which are not suitable for pseudo-summarization corpus construction. Thus, we filter out questions that contain no noun and adjective to make the pseudo data more realistic. Table 1 shows the statistics for the pseudo-summarization corpus.

4.2 Pretraining

Given the pseudo corpus, we first pretrain DDAMS through a question generation pretraining objective, which pretrains DDAMS to generate questions conditioning on subsequent discussions. Afterward, we fine-tune DDAMS on the meeting summarization dataset. Narayan et al. [2020] also reveal that treating question generation as the pretraining objective can help downstream tasks in identifying important contents in the open domain. Our motivations are two-fold: (1) we can potentially augment the training data; (2) since the pseudo data is constructed from the meeting summarization dataset (in-domain), the pre-training can give the model a warm start.

2 In our preliminary experiments, we randomly sample 100 pseudo-summarization instances and observe that 68% of questions can cover important information, such as salient terms and concepts.

3 We conduct experiments with N ∈ [6 : 12]. Finally, we choose N=10 which performs the best.

5 Experiments

Datasets We experiment on AMI [Carletta et al., 2005] and ICSI [Janin et al., 2003] datasets. We preprocess the dataset into train, valid and test sets for AMI (97/20/20) and ICSI (53/25/6) separately following Shang et al. [2018]. We get discourse relations for one meeting based on the SOTA dialogue discourse parser, namely Deep Sequential [Shi and Huang, 2019], which is trained on the STAC corpus [Asher et al., 2016].

Evaluation Metrics We adopt ROUGE [Lin, 2004] for evaluation and obtain the F1 scores for ROUGE-1, ROUGE-2, and ROUGE-L between the ground-truth and the generated summary respectively.

Baseline Models TextRank [Mihalcea and Tarau, 2004] is a graph-based extractive method. SummaRunner [Nallapati et al., 2017] is an extractive method based on hierarchical RNN network. UNS [Shang et al., 2018] is an unsupervised abstractive method that combines several graph-based approaches. Pointer-Generator [See et al., 2017] is an abstractive method equips with copy mechanism. HRED [Serban et al., 2016] is a hierarchical Seq2Seq model. SentenceGated [Goo and Chen, 2018] is an abstractive method that incorporates dialogue acts by the sentence-gated mechanism. TopicSeg [Li et al., 2019] is an abstractive method using a hierarchical attention mechanism at three levels (topic, utterance, word). 4 HMNet [Zhu et al., 2020] is an abstractive method that employs a hierarchical model and incorporates part-of-speech and entity information, which is also pretrained on large-scale news summarization dataset 5 .

5.1 Automatic Evaluation

As shown in Table 2, our model DDAMS outperforms various baselines, which shows the effectiveness of dialogue discourse. By pre-training on pseudo-summarization data, our model DDAMS +DDADA can further boost the performance by a large margin, which shows the need for DDADA. Specially, DDAMS +DDADA (w/o fine-tune) still achieves a basic effect, which appears DDAMS +DDADA (w/o fine-tune) can simulate as a summarization model in terms of selecting important information.

4 [Li et al., 2019] also proposed TopicSeg+VFOA by incorporating vision features in a multi-modal setting. In this paper, we compare our model with baselines using only textual features.

5 Zhu et al. [2020] directly use news summarization dataset to pre-train their model, which can not be applied to our model, since our DDAMS needs dialogue discourse information.

5.2 Human Evaluation

To further assess the quality of the generated summaries, we conduct a human evaluation study. We choose two metrics: relevance (consistent with the original input) and informativeness (preserves the meaning expressed in the groundtruth). We hired five graduates to perform the human evaluation. They were asked to rate each summary on a scale of 1 (worst) to 5 (best) for each metric. The results are shown in Table 3.

We can see that our DDAMS + DDADA achieves higher scores in both relevance and informativeness than other baselines. Ground truth obtains the highest scores compare with generated summaries indicating the challenge of this task.

6 ROUGE-2 of TopicSeg is significantly lower than other baselines, which contains some anomalies. we reproduce the results of UNS and HMNet by the official codes and evaluate them by PyRouge package, the results are consistent with the original paper, which shows the increase of our ROUGE-2 is reasonable.

5.3 Analysis

Effect of the number of dialogue discourse. We conduct

experiments by randomly providing parts of discourse rela-

tions to our model DDAMS in the test process. The results

are shown in Figure 5. We can see that the more discourse

relations, the higher the ROUGE score, which indicates dis-

course can do good to summary generation. When given no

discourse information, our model gets similar scores com-

pared with HRED, which models utterances sequentially.

Effect of the quality of dialogue discourse. We train our

DDAMS by using discourse relations that are provided by

parsers of different qualities. We can see that the higher

the quality of discourse parser, the higher the ROUGE score,

which potentially indicates high-quality discourse relations

can further improve our model.

Effect of the type of dialogue discourse. To verify the importance of each discourse relation type, we then test our model by giving different discourse relations. The results are shown in Figure 7. On AMI dataset, the participants take a remote control design project. We can see that Conditional and Background provide more accurate and adequate information, facilitating summary generation. On ICSI dataset, meetings focus on academic discussion. We can see that the relation Result is more important than other relations, we attribute this to the fact that this relation can strongly indicate the outcome of a discussion that covers salient information.

Effect of meeting graph. To verify the effectiveness of designing different types of edges for different types of vertices, we replace our meeting graph with Levi graph (shown in Figure 2(c)), namely DDAMS (w/ Levi graph). The results are shown in Table 4. We can see that taking the type of vertices into consideration, our model DDAMS can get better results.

Effect of pseudo-summarization data. To further study the effectiveness of pseudo-summarization data, instead of using questions identified by dialogue discourse as pseudo summaries, we extract questions following two rules: (1) utterances that begin with WH-words (e.g. "what"), (2) utterances that end with a question mark. We call this Rule-based Data Augmentation (RBDA). We first pre-train our DDAMS on the two types of pseudo data, which are constructed based on DDADA and RBDA separately. And then fine-tune it on the meeting dataset. The results are shown in Table 5. We can see that DDADA is better than RBDA, demonstrating the effectiveness of dialogue discourse. Besides, pretraining on pseudo-summarization data constructed based on RBDA still achieves a better result, which indicates the rationality of our pretraining strategy.

6 RelatedWork

Meeting Summarization With the abundance of automatic meeting transcripts, meeting summarization attracts more and more attention [Murray et al., 2010; Shang et al., 2018;

Zhao et al., 2019]. There are two main paradigms in meeting summarization. In the first paradigm, some works try to incorporate auxiliary information for better modeling meetings. Goo and Chen [2018] incorporated dialogue acts. Li et al. [2019] incorporated topics and vision features. Ganesh and Dingliwal [2019] reconstructed the dialogue into a sequence of words based on discourse labels. In the second paradigm, some works used news summarization data to alleviate the insufficient training problem. Zhu et al. [2020] used news data to pretrain their model. Ganesh and Dingliwal [2019] used a news summarizer to generate meeting summaries in a zero-shot manner. In this paper, we propose to use dialogue discourse relations as the auxiliary information to model the interaction between utterances and construct a pseudo-summarization corpus, addressing the data insufficiency problem.

Discourse-Aware News Summarization Some works applied a constituency-based discourse structure, namely Rhetorical Structure Theory [Mann and Thompson, 1988] to news summarization [Liu and Chen, 2019; Xu et al., 2020]. These works aimed to analyze the core part of a given sentence and extracted sub-sentential discourse units to form the summary. In this paper, we introduce the dependency-based structure, namely dialogue discourse, for exploring the diverse interactive relations between utterances.

7 Conclusion

In this paper, we apply the dialogue discourse to model the diverse interactions between utterances and present a Dialogue Discourse-Aware Meeting Summarizer (DDAMS). We design a meeting graph to facilitate the information flow and demonstrate that incorporating dialogue discourse is effective for this task. Moreover, we propose a Dialogue Discourse-Aware Data Augmentation (DDADA) strategy to alleviate the insufficient training problem. We build a pseudosummarization corpus by utilizing the QA discourse relation. Experiments on AMI and ICSI datasets show that our model achieves new SOTA performances.

Acknowledgements

This work is supported by the National Key R&D Program of China via grant 2018YFB1005103 and National Natural Science Foundation of China (NSFC) via grant 61906053 and 61976073. We thank all the anonymous reviewers for their insightful comments. We also thank Libo Qin, Yibo Sun and Jiaqi Li for helpful discussion.