On the Importance of Word and Sentence Representation Learning in Implicit Discourse Relation Classification

1 Introduction

Discourse parsing is a fundamental task in natural language processing, which is considered to be a crucial step for many downstream tasks, such as machine translation [Li et al., 2014], question answering [Jansen et al., 2014], text generation [Bosselut et al., 2018], and text summarization [Cohan et al., 2018]. Shallow discourse parsing, such as Penn Discourse Treebank 2.0 (PDTB 2.0) [Prasad et al., 2008] and CoNLL shared task [Xue et al., 2016], focuses on identifying relations between two text spans (i.e., clauses or sentences) that are named discourse units. Among different sub-tasks in PDTB and CoNLL shared task, implicit relation classification between two sentences, with the absence of discourse markers or connectives in between (e.g., however, because), is considered as the most difficult problem.

Machine learning has been widely used in implicit discourse relation classification. Traditional research mainly focuses on linguistic features generation and selection at the text span level and sentence level, and the chosen classifiers are often linear classifiers [Pitler et al., 2009; Lin et al., 2009]. Recently, more flexible deep neural architectures are proposed to learn better representations. A considerable amount of effort has also been put for finding a good neural network architecture for implicit relation classification [Zhang et al., 2015; Liu et al., 2016; Qin et al., 2017; Bai and Zhao, 2018; Dai and Huang, 2018] or some better learning settings such as multi-task learning [Liu et al., 2016; Lan et al., 2017; Varia et al., 2019]. More recently, contextualized word representation learning, such as ELMo [Peters , 2018 ], BERT [Devlin et al., 2019], and RoBERTa [Liu et al., 2019], has been proven to significantly improve many downstream tasks. It has been shown that BERT, especially the next sentence prediction (NSP) task, can significantly improve the implicit discourse relation classification task [Shi and Demberg, 2019]. However, in RoBERTa, while using a larger amount of data training and achieving better performance on many different tasks, it is claimed that NSP is not used. Nonetheless, we find RoBERTa cannot surpass BERT in this task without great modifications. Thus, it is unclear how to choose a contextualized encoder and what leads to improvement.

∗This work was done when Xin Liu was an intern at Huawei Noah's Ark Lab.

In this work, we show that different levels of representation learning are all important to this difficult implicit relation classification problem: deep contextualized word representations associated with an additional sentence separator can improve argument representations by better integrating context information; a bilateral multi-perspective matching can learn the interaction between two arguments without the heavy burden of feature engineering; a fusion module combining the multi-head attention and the gate mechanism can directly collect important clues to understand the text in depth. To this end, we propose a novel model, BMGF-RoBERTa (Bilateral Matching and Gated Fusion with RoBERTa), to combine these modules together to address the discourse relation recognition with the absence of explicit connectives. Extensive experiments show that our proposed model outperforms BERT and other state-of-the-art systems on the standard PDTB dataset around 8% and CoNLL datasets around 16%. We also systematically analyze the effectiveness of different modules and demonstrate how different levels of semantic representation learning contribute to the discourse relation recognition problem. The source code is available at:

https://github.com/HKUST-KnowComp/BMGF-RoBERTa.

2 RelatedWork

After the release of Penn Discourse Treebank 2.0 (PDTB 2.0) [Prasad et al., 2008], the task of shallow discourse parsing has received much attention. Despite the great progress in classifying explicit discourse relations with F1 scores higher than 93% on four-class classification [Pitler et al., 2008] and 90% on 15-class classification [Wang and Lan, 2015], implicit discourse relation recognition still remains a challenge. Most approaches regard the recognition problem as a classification problem of pairs of arguments. Previous research mainly focuses on feature engineering for linear classifiers to classify implicit discourse relations. The first work [Pitler et al., 2009] selected several surface features to train four binary classifiers for the four top-level classes on PDTB. Results showed word pairs were most important among all features developed. [Lin et al., 2009] was the first work to consider the classification on second level types. They further employed four different feature types to represent the context, the constituent parse trees, the dependency parse trees, and the raw text. [Rutherford and Xue, 2014] improved the baseline performance by as much as 25% with the help of brown cluster pairs and coreference patterns.

Recently, with the development of deep learning, more and more researchers choose to design neural architectures for this task as well as other NLP tasks. [Ji and Eisenstein, 2015] adopted two recursive neural networks to exploit the representation of arguments and entity spans. [Liu and Li, 2016] introduced the multi-level attention to mimic the repeated reading strategy and extract efficient features dynamically. [Qin et al., 2016] found stacking gated convolutional neural networks can further improve the performance, compared with MLP, LSTM, and highway networks. [Bai and Zhao, 2018] proposed a model with different grained text representations, including character-, subword-, word-, sentence-, and sentence pair- levels, to enhance the representation power. [Dai and Huang, 2018] took the paragraph as a whole to and modeled inter-dependencies of discourse units by a conditional random filed layer. [Varia et al., 2019] introduced word/n-gram pair convolutions with a gating mechanism to capture the interaction between the arguments for either explicit or implicit relation classification. [Shi and Demberg, 2019] found that the next sentence prediction played a role in finding relationships between sentences and benefited to the implicit discourse relation classification task. Giant contextualized representation models with pretraining show their power in many natural language understanding tasks even those low-resource tasks [Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019]. We can also easily find the trend to work with them from the latest research work in implicit relation classification, e.g., ELMo [Dai and Huang, 2019] and BERT [Shi and Demberg, 2019].

Some researchers chose different routines by expanding training data or integrating human-selected knowledge. [Liu et al., 2016; Lan et al., 2017] shared word embeddings with explicit relation classification. [Xu et al., 2018] used active learning to sample explicitly-related arguments and expanded informative and reliable instances. External event knowledge and coreference relations in the paragraph-level were incorporated into discourse parsing by [Dai and Huang, 2019].

3 Model

In this section, we introduce the details of our model.

3.1 Model Overview

Figure 1 shows the architecture of our model. Our model consists of six parts, including (1) a hybrid representation layer that maps each word to a hybrid between characterand word-level embedding, (2) a contextualized representation layer that enhances the representation power of embeddings, (3) a matching layer that compares each token of one argument against all tokens of the other one and vice versa, (4) a fusion layer that assigns different importance to each word in arguments based on arguments themselves as well as matching results, (5) an aggregation layer that aggregates fusion results and encodes each argument into a vector, (6) a prediction layer that evaluates the probability distribution P(y|Arg1,Arg2). Finally, we name our model as BMGF-RoBERTa (Bilateral Matching and Gated Fusion with RoBERTa).

3.2 Contextualized Argument-aware Representation

Contextualized word representations have helped many models improve their performance in different natural language processing tasks. Some encoders, e.g., ELMO [Peters et al., 2018], BERT [Devlin et al., 2019], and RoBERTa [Liu et al.,

2019], have been widely applied. Here, we use RoBERTa since it is reported to have the best performance on various tasks.

We first use a byte-level Byte-Pair Encoding (BPE) in RoBERTa to encode arguments and map encoded results into hybrid representations:

where CLS and EOS are two special token embeddings introduced in BERT and RoBERTa for attention and downstream tasks. M is the length of Arg1 and N is the length of Arg2, and eji is the hybrid between character- and word-level embedding of the j th word in Argi. Then the embedding lengths of Arg1 and Arg2 respectively are M +2 and N +2.

Then we concatenate the token representations as follows:

where SEP is a special token embedding to indicate boundaries of the sentence concatenation. We use two SEPs here since we will further split these embeddings into two arguments and perform matching and fusion shown in the following sections.

Position embeddings are necessary for transformer-based models as shown in BERT and RoBERTa because the order and the distance between two words affect their dependence. Moreover, RoBERTa does not use segment embeddings to show the order of two sentences after removing the next sentence prediction (NSP) loss. However, as shown in [Shi and Demberg, 2019], the NSP task is very helpful. Therefore, we add back the trainable segment embeddings as what NSP does in addition to the hybrid embeddings and the position embeddings.

After multiple transformer layers, we get the contextualized representations [h0, h1, · · · , hM+N+2, hM+N+3]. Then we split the representations for two arguments:

3.3 Bilateral Multi-Perspective Matching

The resulting contextualized representations are sent to the matching module. Those complex discourse relations usually cannot be easily derived from surface features from argument pairs so that a multiple-perspective matching is required. We follow [Wang et al., 2017] to utilize fullmatching, maxpooling-matching, attentive-matching, and max-attentive-matching but reduce parameters by weight sharing.

We first define a multi-perspective cosine similarity:

where c ∈ Rl is the similarity vector, W c ∈ Rl×d is a trainable matrix, v1 , v2 ∈ Rd , and d is the dimension of v1 , v2 and l is the number of perspectives.

And then, we compute four matching results based on the above multi-perspective cosine similarity.

Full-Matching

We compute the full matching vector for each token in both arguments as follows:

where ˆW f , ˆW l ∈ Rl×d are trainable matrices for the first token and the last token respectively, ˆm1i ∈ R2l is the full-matched vector for ith word in Arg1 and ˆm2j ∈ R2l is for j th word in Arg2. As we add special tokens for each argument (CLS and SEP for Arg1, SEP and EOS for Arg2) and these special tokens contain much semantic information of the whole argument, we use these special tokens to consider the full matching in additional to the individual tokens.

Maxpooling-Matching

The maxpololing-matching is similar to the full-matching but more focused on the most important part. We need to compute all matching results and select the maximum elements.

where ˇW ∈ Rl×d is a trainable matrix for the maxpoolingmatching and max is the element-wise maximum.

Attentive-Matching

The attentive-matching uses the idea of the weighted mean. We need to compute weights based on the cosine similarity and compute the weighted average vectors:

where ci,j = cos(h1i , h2j ) is the cosine similarity of h1i and h2j . Based on the weighted average representations of Arg1 and Arg2, we can compute the corresponding attentive vectors:

where ¯W ∈ Rl×d is a trainable matrix for the attentivematching.

Max-Attentive-Matching

Instead of taking the weighted average of all the contextual embeddings as the attentive vector, the max-attentivematching picks the contextual embedding with the highest cosine similarity as the attentive vector:

where ˜W ∈ Rl×d is a trainable matrix for the max-attentivematching, ˜h1 corresponds to the contextualized representation h1i with argmax0≤i≤M+1 ci,j, and ˜h2 corresponds to that of h2j with argmax0≤j≤N+1 ci,j .

Finally, the bilateral multi-perspective matching results can be obtained by concatenating the four matching results:

where m1i ,m2j ∈ R5l .

3.4 Gated Information Fusion

The matching results contain the interaction between two arguments but miss the interaction inside each argument. The most direct method is to apply self-attention on the concatenation of h and m. Multi-head attention with layer normalization has been used in the Transformer architecture [Vaswani et al., 2017]. We modify it as a gated multihead attention without layer normalization to remain the scale of each dimension and retain the non-linear capability:

where Q ∈ Rdq ,K ∈ Rdk , V ∈ Rdv ,Q′ ∈ Rdq are the query, key, value, and output respectively, W a ∈ R2dq×1 is trainable matrix to transform the concatenation of Q and Q′ to a scalar value a. The final attention result is controlled by the gate a.

To fuse the contextualized representations [h1i , · · · , h1M+1 ] and matching vectors [m1i , · · · ,m1M+1 ], we apply the gated multi-head attention over the concatenation results:

We can also get [f 2 1 , · · · , f 2N+1 ] similarly.

3.5 Aggregation and Prediction

The aggregation layer aggregates the two sequences of fused vectors into a fixed-length matching vector. Inspired by [Kim et al., 2016], we use a convolutional neural network followed by a highway network to capture unigram, bigram, ..., n-gram information. Assuming we apply z convolutional operations Conv1, Conv2, ..., Convz followed by max pooling operations to [f 1 i , · · · , f 1M+1 ], then the output of each conv-pool operation applied on arguments is

where the kernel size of Convc is c, the stride is 1, and the dimension of u1c depends on the number of filters (assume we use s filters).

The flatten concatenation of all conv-pool results are sent as the input of a highway network to extract more features:

where u1 ∈ Rzs is a flatten vector, W h ∈ Rzs×zs ,W g ∈ Rzs×1 are trainable. As before, we can get o2 similarly.

Finally, the prediction layer evaluates the probability distribution Pr(y|Arg1, Arg2). We use a two layer feed-forward neural network to consume o1 and o2 , and apply the softmax function in the last layer. To approximate the data distribution, we optimize cross-entropy loss between the outputs of the prediction layer and the ground-truth labels. 1

4 Experiments

In this section, we show our experimental results and detailed analysis of different representation learning modules.

4.1 Datasets

The Penn Discourse Treebank 2.0 (PDTB 2.0) PDTB 2.0 is a large scale corpus annotated with information related to discourse structure and discourse semantics, containing 2,312 Wall Street Journal (WSJ) articles. PDTB 2.0 has three levels of senses, i.e., classes, types, and sub-types. Most work focuses on the top level, including Comparison (Comp.), Contingency (Cont.), Expansion (Exp.), and Temporal (Temp.). We follow [Ji and Eisenstein, 2015] to split section 2-20, 0-1, and 21-22 as training, validation, and test sets respectively. We not only evaluate our model on the four top-level implicit senses but the 11 major second level implicit types. Table 1 shows the statistics of the top-level senses.

The CoNLL-2016 Shared Task (CoNLL) The CoNLL2016 shared task [Xue et al., 2016] provides more abundant annotation for shadow discourse parsing. The PDTB section 23 (CoNLL-Test) and newswire texts (CoNLL-Blind) following the PDTB annotation guidelines were organized as the test sets. CoNLL merges several labels. For example, Contingency.Pragmatic cause is merged into Contingency.Cause.Reason to remove the former type with very few samples. Finally, there is a flat list of 15 sense categories to be classified.

1 Some instances in PDTB and CoNLL were annotated with more than one label. Following previous work and the scorer at https://github.com/attapol/conll16st, a prediction is regarded as correct once it matches one of the ground-truth labels.

4.2 Parameter Settings

We use the RoBERTa-base as our context representation layer. Because the data size is not large enough so that we freeze all parameters but add trainable segment embeddings like BERT. The number of perspectives of the matching layer l, the number of heads of the fusion layer h, the number of convolutional operations z, the number of filters s are set as 16, 16, 2, 64 respectively, and all hidden dimensions are set as 128. We apply dropout to every layer with the dropout rate 0.2, clip the gradient L2-norm with a threshold 2.0, and add the L2 regularization with coefficient 0.0005 to alleviate overfitting. We adopt Adam [Kingma and Ba, 2015] with an initial learning rate of 0.001 and a batch size of 32 to train models up to 50 epochs. All experiments are performed three times with 2 × 32 GB NVIDIA V100 GPUs and all reported results are averaged performance. Our model and variants are all implemented by PyTorch.

4.3 Experimental Results

We compare BMGF-RoBERTa with other recent state-ofthe-art baselines on multi-class classification and multiple binary classification to demonstrate the importance of our representation, matching, and fusion modules. A siamese network (denoted as BMGF-RoBERTa-Siamese) whose encoder does not concatenate embeddings in two arguments but uses embeddings from RoBERTa separately obtained from each argument is also implemented for analysis.

2 We select the best results from their different settings.

3 We use the average results of their best single model.

Macro-averaged F1 is the main metric for the four-class classification, and accuracy is considered as the main metric for remaining multi-class classification problems. For binary classification, F1 is adopted to evaluate the performance on each class.

Multi-class Classification

We first evaluate our model on multi-class classification which is the final part of a shallow discourse parser as usual. Two different settings are set for PDTB in view of the hierarchical categories. One is the four-class classification (denoted as PDTB-4) at the top-level and the other is the 11-class classification (denoted as PDTB-11) at the second level. We conduct 15-class classification for CoNLL dataset, denoted as CoNLL-Test and CoNLL-Blind for the test set and blind set.

Results of our model and baselines are shown in Table 2, and BMGF-RoBERTa achieves the new state-of-the-art performance with substantial improvements on all implicit discourse classification settings. Most previous systems considered siamese architectures except [Shi and Demberg, 2019] which was the first to consider two arguments as a whole and fine-tune BERT based on the next sentence prediction loss for discourse relation classification. Our variant model BMGFRoBERTa-Siamese can also beat other siamese models and narrow 40% of the gap with [Shi and Demberg, 2019]. Furthermore, BMGF-RoBERTa is the first model that exceeds the 60% macro-averaged F1 and approximates 70% accuracy in the four-class classification on PDTB.

Binary Classification

To compare previous results more directly and analyze model behaviour more comprehensively, we conduct experiments based on one-vs-rest binary classifiers on the four top-level classes in PDTB. Table 3 shows the results of binary classification on the top-level classes in PDTB. It is as expected that BMGF-RoBERTa has the best performance, but it is still surprising that the improvement on Comp. exceeds the previous best more than 11% and the improvement on Temp. exceeds the previous best around 5%. In contrast, the performance of the siamese variant looks a bit ordinary. Results of binary classification indicate that RoBERTa encoder can further capture the deeper discourse relations after giving the "previousnext" context relationship. Such hard discourse relations can not be handled well when two arguments are encoded separately. However, it is still unclear whether the self-attention during encoding or segment embeddings provides the most useful information. We discuss more in Section 4.4.

4.4 Ablation Study

We first analyze the effectiveness of the encoding module, the matching module, and the fusion module. To this end, we remove the segment embeddings (SE), the bilateral multiperspective matching (BM), and the gated information fusion (GF) from BMGF-RoBERTa one by one. To compare the common matching and fusion modules, we implement another two ablation models based on the original RoBERTa: RoBERTa w/ SourceAttn is a model that uses the RoBERTa to encode the concatenation of two arguments and then do the source attention between two separated contextualized embeddings with the residual connection; RoBERTa w/ SourceAttn, SelfAttn adds the self-attention and the residual connection after the source attention.

Table 4 shows the results. We can see that eliminating any of the three modules would hurt the performance significantly. The original RoBERTa model cannot surpass the previous best siamese network on PDTB-4 too much or outperform BERT on PDTB-11, but the SE, BM, and GF can help it improve around 9.3% on average. At the same time, the segment embeddings improve mostly. When comparing source attention and self-attention with our bilateral multiperspective matching and gated information fusion, we can also conclude the combination of BM and GF is much better than those vanilla multi-head attention mechanisms when BMGF-RoBERTa w/o SE as the reference. Moreover, the self-attention hurts the F1 score in PDTB-4.

BMGF-RoBERTa-Siamese shown in Table 2 is not comparable with BMGF-RoBERTa w/o SE and show in Table 4. Separately encoding cannot benefit from the longer selfattention after concatenation. And we can also find SE can improve much more in Table 4, indicating that the longer selfattention is not as straightforward as segment embeddings.

We further design the encoder ablation study for the encoder module and results are listed in Table 5. We can find that the main gaps between encoding without concatenating (BMGF-RoBERTa-Siamese) and encoding without segment embeddings (BMGF-RoBERTa w/o SE) come from the Comp. and Temp. classes. And adding SE brings a qualitative leap in all four classes because BMGF-RoBERTa finally captures clues of two arguments by multiple attention and relative segment order embeddings. That is, a more powerful representation layer can bring unexpected improvement and understand implicit relations better. But how to inject useful information to the representation module looks more important. In our paper, we inject the "previous-next" context relationship by the concatenation and segment embeddings for implicit discourse relation classification.

5 Conclusion

We present a novel model, BMGF-RoBERTa, that combines representation, matching, and fusion modules for implicit discourse relation classification. Experimental results show BMGF-RoBERTa outperforms all previous state-of-the-art systems with substantial margins on the PDTB dataset and CoNLL datasets. We find the "previous-next" context relationship provides much information for this difficult task. Furthermore, the effectiveness of these well-designed modules and what is important in the representation layer are thoroughly analyzed in the ablation study. One important future work would be to find more clues for better representations within and even beyond the word- and sentence-levels.

Acknowledgements

This paper was supported by the Early Career Scheme (ECS, No. 26206717) from the Research Grants Council in Hong Kong and the Gift Fund from Huawei Noah's Ark Lab.