SWIM: A Simple Word Interaction Model for Implicit Discourse Relation Recognition

1 Introduction

Sentences alone do not serve to form coherent discourse. Logical relations, both inter- and intra-sententially, are needed for a coherent text. Such relations are termed discourse relations. Automatically recognizing discourse relations is useful for downstream applications such as machine translation and summarization.

Discourse relations can be overtly signaled by occurrences of explicit discourse connectives such as Indeed and After that (cf Ex. (3) & (4)). In contrast when the context is clear, such overt signals can be omitted, leading to discourse relations that it is said to be implicitly signaled (not marked by a lexical connective in the text; cf Ex. (1) & (2)). The lack of any overt signal makes implicit discourse relations much more challenging to recognize.

This explicit/implicit distinction is adopted by the Penn Discourse Treebank (PDTB, version 2.0) [Prasad et al., 2008]. We adopted the PDTB for this study due to its large size when compared against other discourse corpora such as the Rhetorical Structure Theory Treebank (RST) [Carlson et al., 2002]. While the PDTB has a hierarchical annotation scheme, currently most studies (e.g. [Chandrasekaran et al., Networks (GRN) [Chen et al., 2016] and Neural Networks with Multi-Level Attention (NNMA) [Liu and Li, 2016], apply a bi-directional Long Short-Term Memory network (BiLSTM) [Hochreiter and Schmidhuber, 1997; Schuster and Paliwal, 1997] to represent each argument. Both models achieve the-state-of-the-art F1 score without employing handcrafted features. However, BiLSTMs inevitably introduce many parameters. The large number of parameters slows the training process and is prone to overfitting. Especially in the context of the PDTB, a relatively small dataset, a lightweight representation is a viable method to simplify the model.

∗Wenqiang Lei and Xuancong Wang are the contact authors.

Motivated by this observation, we propose a new model that integrates the modeling of both word pair interaction and argument representation, without the use of BiLSTMs. Our model - termed the Simple Word Interaction Model (SWIM) - achieves a comparable F1 score to the state-of-the-art and runs seven times faster by eschewing the use of BiLSTMs.

2 RelatedWork

Supervised learning approaches for the implicit discourse relations recognition is the common paradigm in prior work. Surface features for the task include word pairs [Marcu and Echihabi, 2002b] sentiment polarity scores, General Inquirer tags [Pitler et al., 2009], and parser production rules [Lin et al., 2009]. Among all these models, the na¨ıve feature of word pairs [Marcu and Echihabi, 2002a; Rutherford and Xue, 2014] - which is the co-occurrence frequency of a pair of words, one drawn from each of the two arguments - has proven to be extremely efficient.

Another trend is to employ semi-supervised methods. This is due to the limited amount of annotated data and the relative abundance of weakly-labeled data (i.e., explicitlysignaled instances). For example, Lan et al. [2013] explored multitask learning; Hernault et al. [2010] applied feature vector extension; and Rutherford and Xue [2015] selectively added some explicit instances as implicit training data, according to their connectives.

The recent wave of deep learning approaches to NLP problems leverage word embeddings pre-trained on large corpora to achieve significant gains on tasks involving complex semantics. While many architectural designs have been explored (e.g. [Zhang et al., 2016]), the state-of-the-art deep learning methods on our task - the GRN (discussed in detail later) and NNMA - are fundamentally BiLSTM based, involving many parameters, which leads to inefficiency in training and testing.

3 SimpleWord InteractionModel (SWIM)

SWIM models both the fine-grained word pair interaction and coarse-grained argument representation. We first introduce the model here and detail its implementation in the following sections. To capture word pair interaction, we calculate an interaction score for each word pair that measures the importance of the interaction between its component words. For argument representation, we apply a weighted average of the component word pair representations. The argument representation thus encapsulates word pair interaction, and is MLP

WA

interaction

embedding

(a) SWIM

MLP

interaction

BiLSTMs

embedding

(b) GRN

passed to a final a multilayer perceptron layer (MLP) to determine the final discourse relation (cf Figure 2(a)).

3.1 Word Interaction Score

Following current best practices, SWIM's word interaction score captures both linear and quadratic relations between the two word's embeddings. Formally, let M and N denote the lengths of Argument 1 (hereafter, Arg1) and Argument 2 (Arg2). Further, let xi, yj denote the pre-trained (row) word embeddings of ith word of Arg1 and j th word of Arg2, separately. Then for each pair of words xi and yj , SWIM calculates an interaction score as in Eq. (1):

where A ∈ Rd×d , B ∈ R1×2d and cij ∈ R are trainable parameters, and [xi, yj] is a concatenation of the two word embeddings. xiAyj T models the quadratic relation between the two word embeddings, while B[xi, yj] captures linear relationships. cij is the bias term for the final interaction. At training, these parameters -A, B and cij -model and encode word pair semantics, assigning a high interaction score when well correlated with a particular discourse relation class. Similar approaches have been applied to many fields, e.g. recommendation system [He et al., 2017].

SWIM calculates the interaction matrix S, computing sij for each possible (ith ) word of Arg1 and (j th ) word of Arg2.

3.2 Argument Representation

Both (Bi)LSTMs and embedding averaging are valid methods for representing text sequences, inclusive of discourse arguments and sentences. Recent studies suggest that LSTMs performwell when sequential order is important [Iyyer et al., 2015]. However, this is less the case in argument representation, where content plays a larger role than ordering - e.g., in Exs. (1) & (2), the discourse relations do not change even when the words are reordered. Simpler methods, such word averaging may be sufficient and effective as suggested by Wieting et al. [2016] who concluded that "word averaging models perform well for [the related tasks of] sentence similarity and entailment, outperforming LSTMs." Hence, SWIM adopts embedding averaging.

However, embedding averaging alone is insufficient - each argument's words are actually understood in the context of the opposing argument. For example, on its own down in Arg1 of Ex. (2) is less likely considered to signal any discourse relation. However, once combined with the word up in Arg2, it becomes the most important signal in Arg1 for the Comparison relation. In light of this, SWIM represents

(a) LSTM Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17)

each argument as an average of its component words' word embeddings, weighted for its interaction with the opposing argument.

We denote SWIM's argument representation for Arg1 (Arg2) as x′ (y′ ), as calculated in Eq. (2).

In Eq. (2), we start with the concatenations of word pair embeddings [xi, yj], weighting them with the interaction score sij, to account for its importance.

Let us walk through Eq. (2), taking the computation of x′ as an example. For each word xi in Arg1, we enumerate all words in Arg2 (yj for j ∈ {1, 2...N}) to form word pairs ([xi, yj]). We weight these word pair representation according to its normalized interaction score, obtaining an interaction weighted word representation, the term in the parentheses. All M interaction-enhanced word representations are averaged to arrive at the final form for x′ .

SWIM computes a single representation of both Arg1 and Arg2 by concatenating x′ and y′ for input to the final MLP classification to obtain the output discourse relation:

where Wh ∈ R4d×k and Wo ∈ Rk×n . In Wo and Wh, n is the number of class labels, d is the embedding size, k is the size of the hidden layer. fh and fo are sigmoid activation functions. The final classification layer described in Eq. (3) is a two-layer MLP whose hidden layer is designed to obtain a more abstract representation.

3.3 Model Discussion On BiLSTMs

Both state-of-the-art prior models for implicit discourse relation classification - GRN and NNMA - adopt BiLSTMs for argument representation. A discussion on the role of BiLSTMs in both models is relevant. We use GRN as a sample for this discussion as i) both GRN and our proposed SWIM is designed to model word pair interaction and apply similar word interaction calculation approach; ii) the necessity of BiLSTMs in GRN has been previously studied in [Chen et al., 2016].

The GRN workflow is illustrated in Figure 2(b). GRN feeds one argument into an individual BiLSTMs to get an intermediate representation for each word, denoted as hxi and hyj for each xi and yj, separately. As illustrated in Figure 3, the intermediate representation incorporates information from the argument as a whole.

For each pair of hxi and hyj , an interaction score sij is calculated as in Eqs. (4) & (5). As we focus on the role of BiLSTMs, we do not discuss interaction scores further.

In the above, ⊙ denotes element-wise multiplication. M[1:r] ∈ Rr×2d×2d , V ∈ Rr×4d , Wg ∈ Rr×4d , u, b ∈ Rr are trainable parameters where d is the embedding size and r = 2 according to the original paper [Chen et al., 2016]. Both f and σ are sigmoid activation functions. Eq. (4) & (5) of GRN degrades to Eq. (1) in SWIM if we set r = 1, g = 0.5 and remove aggregating vector u. Similar to SWIM, a word pair interaction matrix S ∈ RM×N is obtained. Finally, the matrix S is max pooled and then reshaped into a vector to feed a final multi-layer perceptron (MLP) for classification.

The matrix S, which is the representation for each instance, actually contains two aspects of information: i.) word content information (whether significant word pair interactions exist); and ii) positional information (where the strong interacting words are). Thus, BiLSTMs in GRN contribute both position information and content information. However, we can eschew BiLSTM use by only handling content information via embedding average, since embedding average and BiLSTM are know to have similar content representations. We can test this hypothesis by inserting BiLSTMs into SWIM to evaluate its effect: we plug a BiLSTM layer in between the embedding layer and word interaction layer, as illustrated in Figure 2. We refer to this model as the BiLSTM + SWIM model in the following discussion 1 .

To evaluate BiLSTMs' effect on model complexity, we further calculate the number of parameters of various model. As shown in Tabel 1, the use of BiLSTMs, which encode superfluous position information, increase the model complexity by an order of magnitude (Row 1, 2, 5, 6). As for NNMA, it contains a complex multi-level attention layer unlike BiLSTMs which is less different from our design. Therefore, we just list its number of parameters for reference without discussing it in more details.

4 Experiments

We now evaluate SWIM, with the aim to show its competitiveness in terms of prediction performance. Importantly, we note that the evaluation of a computational model not only lies in the best prediction performance, but also its stability (robustness), and efficiency. This is especially important for neural network models, which are computationally intensive and sensitive to random factors. We first describe the systems we compare against, then describe the experimental setup before describing the main results.

1 We use a dense layer between BiLSTM and SWIM to reduce the output size from 2d to d, to lessen the number of parameters in BiLSTM + SWIM.

4.1 Comparison Systems

In the following list of architectures, numberings follow those used in Table 2 for convenience. First, we must validate performance against the current state-of-the-art baselines in terms of raw prediction performance, in the numbered systems below. The cited papers, our SWIM and replicated experiments follows the same training, development, and testing splits, so are comparable.

• 1-2. Gated Relevance Networks (GRN; [Chen et al., 2016]). We replicate their architecture with the assistance of the original author, tuning parameters according to the original paper.
• 3-4. Neural Networks with Multi-Level Attention (NNMA; [Liu and Li, 2016]). We cite results from the paper.

We study a number of variants of SWIM to test our design decisions. In the lettered systems below, we examine variations on argument representation, using embedding averaging and BiLSTMs:

1. Embedding Average (EA). We perform na¨ıve embedding averaging to get a representation vector for each argument. Then, we concatenate the two vectors for input to a MLP as in Eq. (3).
2. BiLSTM. We use the word embeddings as input to BiLSTMs (hfT , hb1 ), concatenate their output to form the argument representation and pass it to the MLP layer for classification 2 . This setting benchmarks the vanilla use of BiLSTM for argument representation.
3. Word Interaction Score (WIS). We use scores in the interaction matrix S for classification without modification. Following the GRN approach (cf Section 3.3), we perform max-pooling on S before inputting the scores to the MLP layer. This tests whether our weighted argument representation helps.
4. BiLSTMs + WIS. Following GRN, we feed word embeddings to BiLSTMs to get an intermediate representation. Then we calculate word interaction scores based on

2 Another common approach for BiLSTM argument representation is to get the intermediate representation for every time step hi, i ∈ {1...T} and average these intermediate representation as EA does. We find that both approaches yield similar results, hence only the approach in the body text is reported.

these intermediate representation using Eq. (1), obtaining the interaction matrix S which is max-pooled and fed into a MLP layer. This aims to test the effect of BiLSTMs on WIS. BiLSTM + WIS is similar to GRN except for the word interaction computation part. However, we here only focus on the study of BiLSTMs without discussion its difference.

1. BiLSTM+ SWIM. As described in Section 3.3, this tests whether BiLSTMs further aid SWIM, since content information has already been encoded by SWIM's word embedding averaging.

We also investigate whether both linear and quadratic terms are necessary for SWIM's word interaction computation:

1. Quadratic Term (QT only). Only the quadratic term and bias term of Eq. (1) are retained.
2. Linear Term (LT only). Only the linear and bias terms are retained.

4.2 Experimental Setting

We adopt the standard settings for the PDTB v2.0 dataset use in our experimentation (Sections 2-20, Sections 0-1 & Sections 23-24, and Sections 21-22 for training, development and testing, respectively.

We follow the practice of [Zhou et al., 2010; Liu and Li, 2016] which follow the standard definition of PDTB v2.0, which admits a separate categorization outside of discourse relations for EntRel (hence distinct from Expansion), unlike [Pitler et al., 2009; Chen et al., 2016] that deemed entity relations (EntRel) as a form of Expansion relations. Aside from this, our other settings are standard: we use Stanford CoreNLP [Manning et al., 2014] for tokenization, pad all sentences to length 50, and use Stanford's GloVe [Pennington et al., 2014] 100 dimensional pre-trained word embeddings for SWIM and 50 dimensional pre-trained embedding for BiLSTMs + SWIM. The embedding layer is fixed during training, and dropout is performed on the input and MLP layers (dropout percentage = 20%). For training, we adopt multiclass cross-entropy loss, using AdaGrad for the stochastic optimization [Duchi et al., 2011]. The initial learning rate is set at 0.01, with a batch size of 32. Following [Liu and Li, 2016; Rutherford and Xue, 2014], we use instance re-weighting.

4.3 Experimental Results

Table 2 presents our results. Prediction performance is evaluated in both standard schemes of n binary classifications and a single n-way classification (where n = 4), using macro F1.

The upper portion of Table 2 shows that our SWIM architecture (Row 5) performs on par with GRN and NNMA, with the exception of Temporal relations. Introspecting these results at an instance level, we find that GRN, NNMA and SWIM exhibit little agreement on Temporal relations. As Temporal relations only constitute 5% of the dataset, it is a minority class where its performance may be largely affected by random factors. In our efficiency analysis, we calculate the time cost per instance in one training epoch (measured in milliseconds). Our proposed SWIM runs 7 times (3.46/0.49) faster than GRN.

This result is significant as the training times listed in Table 2 are for individual epochs and for a single architecture. The time savings benefits two aspects: first, neural models are typically trained over hundreds of epochs and with hundreds of individual configurations for hyperparameter tuning, leading to significant savings in development and tuning times in creating the final model. Second, semi-supervised learning is an important direction for implicit discourse relation recognition, due to the abundance of weakly labeled data. In semisupervised learning, usually a much large number of training instances are involved. For example, Liu et al. [2016] report training with over 400k instances. With 400k instances, we estimate that a GRN architecture will take about 38 hours for a run of ∼100 epochs; in comparison, SWIM would complete in less than 6 hours.

Model stability is another concern. In our analysis we run each model 10 times by initializing with different random seeds for both 4 binary classifications and the single, fourway classification, calculating the standard deviation over the ten runs. We see that SWIM's predictive performance is more stable (last column, comparing 0.008 with 0.012, F-test p = 0.001984 < 0.005) than GRN.

The middle portion of the table provides empirical validation of SWIM's choice of argument and word pair representations. Rows a and b validate our hypothesis that embedding averaging achieves prediction performance comparable to the more complex BiLSTM model. Simplifying SWIM's model further by employing only the raw word interaction score (Row c) also underperforms.

Could the simplified treatment of the interaction scores coupled with the standard BiLSTM approach close the performance gap? Row d explores this option. The answer is 'no', as it underperforms; SWIM (Row 5) adds the use of embedding averaging to sensitize the argument representation to the interaction scores. Augmenting SWIM with BiLSTMs does not improve the results further (Row e), plausibly supporting our hypothesis that positional information encoded by BiLSTMs may contribute noise over the use of the content cues already encoded by embedding averaged argument representation. Additionally, all BiLSTMs models take significantly longer training times and produce less stable results (cf Row 5 vs. Rows 2, b, d, and e).

Finally, comparing SWIM against the selective use of only the quadratic or linear word interaction terms in the bottom portion of the table suggest that both are needed to properly model word interaction (Row 5 vs. Rows A and B).

Overall, we conclude that the current SWIM configuration strikes a good balance among effectiveness, stability, efficiency and model complexity.

Discussion

There are loose ends to validate in our claims that have not yet been supported by evidence. Here, we detail three issues where we present auxiliary data to buttress our arguments.

First, we have argued that positional information encoded by BiLSTM (in the guise of GRNs) is not needed for our classification task. We validate this proposition via supplemental experimentation. To give credence that GRN models positional information, we randomize the order of the words in each argument in the test set to obtain a new test set (test′ ). We then test the pre-trained GRN, SWIM against test′ . Table 3 gives the 4-way classification performance figures on test′ . The results indicate that GRN is sensitive to the position of words, while SWIM-based models retain their performance, validating that SWIM does not model positional information.

Second, we need to assess the sensitivity of the neural architectures to dataset scale, as we have argued that the SWIM architecture is optimal for the task, independent of the scale of data currently available. We train and test GRN and SWIM with different model sizes by changing the hidden layer and embedding sizes 3 of each model to have their total number of parameters be at parity: set to 10k, 30k, 100k and 300k (Table 4). This result suggests that GRN intrinsically requires much more parameters to achieve better prediction results; in contrast, SWIMmakes better use of its limited number of parameters.

3 For GRN, 50d is the smallest pre-trained GloVe word embed-

Finally, we also wish to assess whether SWIM actually does assigns high interaction scores to important word pairs in the matrix S through per instance analysis. In Figure 4, we introspect the interaction matrices of WIS and SWIM on a PDTB example to illustrate how they differently they behave. We note that both WIS and SWIM calculate word interaction scores in the same way, but that SWIM's contextualization of its argument representation modeling assigns high scores more intuitively than WIS does. WIS assigns word pairs featuring are ('re) high scores, possibly due to its corpus frequency. The arithmetic post-processing of the argument context by SWIM redistributes scores, correctly assigning the fortunate-hurricane pair the highest interaction score. Separately, Figure 5 shows SWIM's interaction scores on a Comparison instance, where the words (row) down and (column) up are assigned high weights, which is in accord with our intuition that the word pair is critical in determining the discourse relation. Interestingly, the word (row) about also yields a high score. What was the cause for this? Drilling down, we find that the Wall Street Journal uses similar textual expressions "[numbers] ... increase/decrease/up/down ... about" when reporting many instances of financial news. The high weight for about lends evidence that SWIM can discover such important contextual co-occurrence corpus patterns.

ding. Therefore, we have to reduce the hidden size of GRN. For SWIM, we use larger pre-trained embeddings and enlarge the hidden size to increase model size.

5 Conclusion

We proposed a simple neural model for implicit discourse relation recognition, named SWIM, which accounts for both word pair interaction and argument representation. In contrast to previous works, we utilize word embedding average, instead of BiLSTMs, for argument representation. Experiment results show our model is more stable and runs faster while still achieving state-of-the-art F1 scores.

In the wider context of neural network research, our work finds additional evidence that BiLSTMs spend much effort to model positional information, which we have shown to be less helpful for our task. In tasks where word content is valued over word ordering information, our work suggests that simpler models such as an embedding averaging, can replace BiLSTMs while achieving similar performance.

Acknowledgments

We gratefully acknowledge Qian Zhong of the Department of Linguistics and Translation, City University of Hong Kong for fruitful discussions on this research, and also acknowledge the support of NVIDIA Corporation for their donation of the Titan X GPU that facilitated this research.