Boosting Causal Embeddings via Potential Verb-Mediated Causal Patterns

1 Introduction

Causation or causality is a special semantic relationship between one cause process (or state) and another process (or state), where the cause is partially responsible for the effect, and the effect is partially dependent on the cause. Causation plays an important role in human thinking and reasoning. Whenever something serious happens, human usually tries to determine its causes and its effects.

Human is able to figure out causalities in text for at least two reasons. On the one hand, human can recognize lexical units that explicitly indicate causalities, such as because, cause, result in and as a consequence, which makes it possible to detect causalities even when human has no knowledge about the meaning of the cause or the effect. On the other hand, human can understand the meaning of language and has both commonsense and domain-specific knowledge, and thus can detect causalities even when there are no explicit causal lexical units. These two cognitive characteristics get mixed together and assign human the ability of causality understanding and causal reasoning. The results of causality understanding and causal reasoning can enrich human's knowledge, the other way round.

∗Contact Author

It is challenging for computer to perform such a task. For many years, NLP community has made a lot of efforts to causality-related tasks, resulting in a lot of proposed methods. Ealiest approaches were based on hand-coded domainspecific knowledge bases to extract explicit causal knowledge from text [Kaplan and Berry-Rogghe, 1991] and to rank the extracted possible causalities [Girju and Moldovan, 2002]. These works achieved satisfactory precision but low recall. Newer approaches employed machine learning algorithms to extract explicit and implicit causalities from text [Girju, 2003; Chang and Choi, 2006; Hashimoto et al., 2015a], which relies heavily on feature engineering. Automatic detection of causalities has played a fundamental role in various downstream applications such as question answering [Oh et al., 2013], event prediction [Radinsky et al., 2012] and future scenario generation [Hashimoto et al., 2014].

In recent years there has been an increasingly interest in using word embeddings [Collobert et al., 2011; Mikolov et al., 2013] as an alternative to traditional hand-crafted features. More recent studies have focused on how to build and use task-specific word embeddings [Tang et al., 2014; Boros et al., 2014; Nguyen and Grishman, 2014; Hashimoto et al., 2015b; Li et al., 2017]. As for the cause-effect relationship, several methods have been proposed in [Sharp et al., 2016; Xie and Mu, 2019] to construct causality-specific word embeddings (or causal embeddings in short) from a set of causal phrase pairs extracted via a handful of high-precision patterns.

The construction of causal embeddings is a valuable causality-related task in several aspects. Firstly, causal word pairs are important lexical knowledge resource which is helpful to causality extraction [Chang and Choi, 2006]. Causal embeddings can provide a simple way to measure the causal affinity of word pairs, or equivalently, provide a large scale collection of causal word pairs. Secondly, with causal embeddings, the causal affinity between two phrases can be measured as the maximal causal affinity of all the word pairs between them, which can facilitate detecting implicit causality detection and measuring causal strengths of potential patterns. Last but not the least, causal embeddings are complementary to vanilla word embeddings in downstream applications such as why-question answering [Sharp et al., 2016; Xie and Mu, 2019].

As for the causal embedding methods, existing works

[Sharp et al., 2016; Xie and Mu, 2019] usually have to first extract a large number of causal phrase pairs by a handcrated set of strong causal patterns, and then construct causal embeddings from them. The resulting causal embeddings have shown high precision, but relatively low coverage, because a large number of textual causalities are connected only by weak patterns. To solve this problem, we propose a novel method (called BoostedMaxMatching, or BMM in short) to boost the causal embeddings by exploring potential weak causal patterns. This paper makes the following contributions:

• Firstly, it proposes an extraction method for verbmediated patterns, defines an inversion operation of patterns to handle the causal directionality problem, and designs an algorithm to learn pattern weights that indicate their causality-indicating strengths.
• Secondly, it proposes a method to boost causal embeddings by fusing the existing strong causal phrase pairs and the extactions from (possibly weak) verb-mediated patterns. Experimental results have shown that the resulting causal embeddings have achieved high precision and recall on both Chinese and English test data.
• Finally, our method also ouputs verb-mediated patterns together with their causal strengths, as byproducts. Examination of them has shown that the top-ranked patterns coincide with human intuition about causality.

2 RelatedWork

To the best of our knowledge, there are two existing works on causality-specific embeddings [Sharp et al., 2016; Xie and Mu, 2019] which are closely related to this paper.

Sharp et al. [2016] did the first work on causal embeddings and proposed a series of cEmbed-family methods. They generated a causality-specific embeddings and demonstrated that these dedicated embeddings are helpful in a downstream causal QA task. Their cEmbed-family methods for embedding construction are based on the Skip-Gram algorithm [Mikolov et al., 2013], by treating an effect phrase as the context of its cause and a cause phrase as the context of its effect. However, such treatment is based on the assumption that each word pair between a cause-effect phrase pair is causally related. Due to the fact that the assumption is far from reality and thus introduces too much noise, the performance of cEmbed-family models is undistinguished.

Xie and Mu [2019] proposed three methods (Pairwise Matching, Max Matching, and Attentive Matching) for building causal embeddings from a corpus of cause-effect phrase pairs. In order to transfer causal relationship from phrase-pair level to word-pair level, Max-Matching method assumes that there is at least one word pair carrying the causality information of the positive phrase pair, which can be thought of as a special case of multi-instance learning. Attentive-Matching takes the assumption that for a causal phrase pair, there must exist at least one close interaction between a cause word and the effect phrase and another between an effect word and the cause phrase. Both Max-Matching and Attentive-Matching have achieved satisfactory performance on both English and Chinese corpora.

The research work on vanilla and task-specific word embeddings is also related to our work. In recent years, there is an upsurge of deep learning in natural language processing [Collobert et al., 2011], where distributed representation of words serves as the basis. The neural methods that learn vanilla distributed representation of words (called word embeddings) usually capture only co-occurrence relationships between words [Mikolov et al., 2013]. Although such a general-purpose word embeddings are helpful for various NLP tasks, the acquisition of generality is often at the cost of losing specificity to a certain degree. Tang et al. [2014] proposed learning sentiment-specific word embedding for sentiment analysis, where sentiment information is encoded into the continuous representation of words such that it can separate good and bad to opposite ends of the spectrum. Hashimoto et al. [2015b] proposed a novel method to train word embeddings for semantic relation classification, by predicting words between noun pairs using lexical relationspecific features. Li et al. [2017] developed a tailored neural network to learn contradiction-specific word embedding.

3 Method

Existing works on causal embedding (inclusive of Sharp et al. [2016], Xie and Mu [2019]) extract a set of causal phrase pairs by a handful of hand-crafted causal patterns, and then build causal embeddings from the phrase pairs, which may suffer from limited coverage. To remedy this deficiency, we propose a novel method to take weak-causality patterns into consideration, in order to boost the coverage of the learned causal embeddings. The architecture of our method is illustrated in Figure 1, which consists of four main components:

• Triple extraction via verb-mediated patterns: This paper focuses on the verb-mediated patterns and treats their extractions;
• Causal word pair seeding: A high-quality seed set of causal word pairs is generated, which will provide supervision information to the phase of pattern weighting;
• Pattern weighting and scoring: The candidate patterns are weighted and scored according to their ability to extract causal pairs;
• Boosted causal embeddings via weak-causality patterns: The phrase pairs extracted by the verb-mediated patterns are weighted and filtered according to the pattern scores and then get fused with the previously extracted high-precision causal phrase pairs, to construct the boosted causal embeddings.

3.1 Triple Extraction via Verb-Mediated Patterns

Verb-mediated patterns that indicate causation are a most frequent kind of explicit intra-sentential causal pattern [Girju, 2003]. In this paper, we focus on three simplest easy-toextract forms, which work on English corpus as follows, after parsing the corpus with the SpaCy 1 dependency parser:

• for each transitive verb with active voice, locate its nsubj or csubj as the subject phrase, its dobj or

1 http://spacy.io

ccomp extended by its oprd, xcomp, and acomp as the object phrase, and use the verb's lemma as the pattern;

• for each transitive verb with passive voice, locate its agent as the subject phrase, its nsubjpass or csubjpass as the object phrase, and use the verb's lemma as the pattern;
• for each intransitive verb (with active voice), locate its nsubj or csubj as the subject phrase, the pobj or pcomp of its prep as the object phrase, and use the verb's lemma plus the preposition as the pattern.

A phrase is valid if it contains at least a content word (i.e., noun, verb, adjective or adverb). Only the extractions whose subject and object phrases are both valid will be kept. Figure 2 illustrates several extraction examples. On Chinese corpus, the dependency parser in LTP toolkit 2 is used instead and some minor modifications are made to the rules accordingly.

Applying the above rules on a raw text corpus, we can obtain a dataset E of triples 〈S, p,O〉, where S is the subject phrase, O is the object phrase, and p is the pattern that is either simply a transitive verb or an intransitive verb plus a following preposition.

For these causality-indicating patterns, the causal directionality deserves our attention. Some of them such as prompt have their subject phrases as cause, for example in the first sentence "The powerful typhoon prompts a huge flood", while the others such as result from have their object phrases as cause, for example in the second sentence "A huge flood resulted from the powerful typhoon", where (typhoon, flood) is known to be a causal word pair. Therefore, it is desirable to create a new pattern by inverting an existing one. Specifically, for a given pattern p, we can make its inverted version pT by inverting the subject phrase and the object phrase of p. That is, for each triple 〈S, p,O〉 ∈ E, we will add its inverted triple 〈O, pT , S〉 into E. For example, the triple 〈A huge flood, result from, the powerful typhoon〉 extracted from the second sentence will be inverted to 〈the powerful typhoon, result from inv, A huge flood〉, where result from inv is the inverted pattern of result from. Clearly, inverting the patterns will double the size of dataset E. Let P = {p1, p2, . . . , pK} denote the set of patterns in E, and γ(p) denote the set of subject-object phrase pairs extracted with a pattern p ∈ P .

2 http://github.com/HIT-SCIR/pyltp

In addition, these verb patterns vary significantly in the strength of indicating causality, due to the ambiguity of verbs. Some verb patterns such as cause and result from always convey strong causal signal, some patterns such as contain and consist of have weak or even no signal of causality.

3.2 Seed Set of Causal Word Pairs

At the starting point of our method, it is assumed that there

• are· a collection Eh of high-precision causal phrase pairs,
• a cause embedding function c which maps a word w to its cause embedding c(w) in a d-dimensional Euclidean space S c (called the cause space), i.e. c(w) ∈ Rd , and
• an effect embedding function e which maps a word w to its effect embedding in another d-dimensional Euclidean space S e (called the effect space), i.e. c(w) ∈ Rd ,

where the embeddings c and e are built up from Eh by one of the cEmbed-family methods [Sharp et al., 2016] or the MaxMatching and Attentive-Matching algorithms [Xie and Mu, 2019]. To measure the causality of word pair (w1,w2), the causal interaction score cs(w1,w2) is defined as the inner product between the cause embedding of w1 and the effect embedding of w2:

Identifying Candidate Causal Word Pairs

Given a high-precision causal phrase pair (C,E) ∈ Eh, the word pair (c, e) with the highest causal interaction score betweeen C and E is thought of as the dominant word pair, that is:

A word pair (w1,w2) is a candidate causal word pair if it serves as the dominant word pair of at least λ1 phrase pairs in Eh and its causal interaction score is not less than λ2 (λ1 and λ2 are set to 30 and 0.55 by default).

Generating Seed Causal Word Pairs by Pruning

The candidate causal word pairs are then pruned according to the irreflexive and antisymmetric properties of causal relationship:

• We remove all the word pairs (w,w) whose cause and effect words are identical, because of the irreflexity of causal relation;
• For two different words, w1 ̸= w2, if (w1,w2) and its reverse (w2, w1) belong to the seed set at the same time, we remove both of them, because the causal relation is antisymmetric.

The resulting set of seed causal word pairs is denoted as S, which will provide supervision information for learning pattern weights.

3.3 PatternWeighting and Scoring

Since different patterns usually have different strengths of causality, we would like to assign a weight to each pattern. The problem is how to learn these weights and where the supervision information comes from. In Section 3.1, we have built up the connections between patterns and phrase pairs in E, which is equivalent to a co-occurrence matrix of patterns and phrase pairs (or PPP matrix in short), denoted as Mppp. The dataset Eh can serve as the gold standard, but it is not helpful in practice to tell us which phrase pairs in Mppp are positive (or causally-related), because of the sparsity of phrase pairs (i.e., a phrase pair seldom co-occurs more than one time in the text). Therefore, we cannot infer the pattern weights directly on the level of phrase pairs.

Pattern Weighting

To solve this problem, let us move to the level of word pairs and seek for possible solution. We first build connections between patterns and word pairs in a so-called co-occurrence matrix of patterns and word pairs (PWP matrix in short, denoted as Mpwp), where the element fk(w1,w2) at the row of word pair (w1, w2) and the column of pattern pk indicates how many times w1 and w2 are linked by pk:

fk(w1,w2) = |{(S,O) ∈ γ(pk) : w1 ∈ S,w2 ∈ O}|. (3) For the i-th word pair in Mpwp, it is labeled as positive (yi = 1) if it belongs to the seed set S; otherwise, it is negative (yi = 0). Since the positives and negatives are ready and the examples (or word pairs) are described by a feature set {f1, . . . , fK}, we propose a pattern-weighting method as following, similar to logistic regression.

The causal affinity ca of a word pair (w1, w2) is measured by the average weight of all its features:

where σ(·) = exp(·) 1+exp(·) is the sigmoid function that makes normalization.

The weight vector a = (a1, . . . , aK) is trainable to minimize the cross-entropy loss with l2 regularization:

where wpi is the i-th word pair in Mpwp, N = |Mpwp| is the total number of word pairs in Mpwp, and λ is the regularization coefficient that is set to 10-5 by default. It is expected that causality-indicating patterns get positive weights and word pairs connected by many causality-indicating patterns will receive high causal affinities.

Pattern Scoring

Given a pattern pi, it is of interest to know the average causal affinity of the phrase pairs it connects, measured by the score s(pi):

where the causality affinity CA of phrase pair (C,E) is calculated as the maximum causal affinity among the word pairs between C and E:

Patterns whose scores are less than a threshold (0.55 by default), together with the phrase pairs they connect, are removed from E. For each phrase pair PP ∈ E, the score of its pattern p is assigned as its weight (v(PP) = s(p)); while for each phrase pair PP ∈ Eh, the weight v(PP) is set to 1.0. As a result, we can merge E and Eh into a set of weighted phrase pairs, denoted as Ew. Section 3.4 will build the boosted causal embeddings on it.

3.4 WeightedMax-Matching for Boosted Causal Embeddings

Now that we have obtained the weighted phrase pair set: Ew = {(Si,Oi)|1 ≤ i ≤ |Ew|}, the task is to construct boosted causal embeddings, ˜c and ˜e from it. This task looks similar to the one solved by [Sharp et al., 2016] and [Xie and Mu, 2019], except that all the phrase pairs have their associated weights. Therefore, we propose a variant of MaxMatching algorithm, called Weighted Max-Matching, to take the weight information into consideration.

For a given word pair (w1,w2), the boosted causal score ˜ cs(w1,w2) is defined as the inner product between the boosted cause embedding of w1 and the boosted effect embedding of w2:

It is different from the causal interaction score defined by Equation (1) in that it uses different embeddings ˜c and ˜e.

Similar to the Max-Matching model, its weighted variant also treats the task of learning causal embeddings as a multiinstance learning problem [Maron and Lozano-P´erez, 1998]. A phrase pair (C,E) can be mapped to a bag of word pairs {(c, e)|c ∈ C, e ∈ E}. It assumes that the bag contains at least one positive word pair if the phrase pair is positive, and all the word pairs are negative if the phrase pair is negative. That is, for a given positive phrase pair (C,E), it is expected that we can find a word pair (c, e) between C and E such that the cause word c ∈ C has a large boosted causal score with the effect word e ∈ E.

The boosted causal score of a phrase pair (C, E) is defined as the maximum causal interaction score among all the word pairs between C and E:

In other words, only the word pair with the highest causal interaction score is selected as the positive word pair, which serves as the representative of the positive phrase pair.

The boosted variant has adopted the same negative sampling strategy as Max-Matching algorithm. Let N denote the set of all the sampled negative phrase pairs. Then, the loss J to minimize is calculated by summing all the cross-entropy losses of the training examples:

where v((C,E)) is the weight of the phrase pair (C,E).

4 Evaluation on English Corpus

To make an evaluation on English, we make use of a highprecision corpus Eh of 815,233 cause-effect phrase pairs which was extracted with a set of 13 hand-crafted rules from Gigaword and Simple English Wikipedia. Both the rules and the corpus are taken from [Sharp et al., 2016] 3 . The MaxMatching algoirthm [Xie and Mu, 2019] is used to build the initial causal embeddings 4 c and e because of its superior performance.

We compare our causal embedding method against several state-of-the-arts:

• vEmbed: The vanilla word embeddings trained on raw text corpus by the skip-gram algorithm [Mikolov et al., 2013] with a sliding window of 5;
• cEmbed: The cEmbed method [Sharp et al., 2016] treats the effect phrase as the context of the cause, and uses a variant of Skip-Gram to train the causal embeddings;
• cEmbedBi: The bidrectional model [Sharp et al., 2016] trains a second model treating causes as context, and ranks word pairs by averaging it and cEmbed;
• cEmbedBiNoise: The noise-aware bidirectional model [Sharp et al., 2016] improves cEmbedBi by weighting word pairs with their likelihoods of being truly causal;
• Attentive-Mathing: It assumes that there are close interactions between a cause word and the effect phrase, and also between an effect word and the cause phrase [Xie and Mu, 2019];
• Max-Matching: It is built upon the assumption that there is at least one word pair carrying the causality information of positive phrase pair [Xie and Mu, 2019];

3 http://clulab.cs.arizona.edu/data/emnlp2016-causal/

4 http://www.ke.fudan.edu.cn/data/causal/en causal embeds.zip

• BoostedMaxMatching (BMM in short): The proposed method in this paper, running on the corpus of English Wikipedia 5 .

4.1 Causal Word Pair Ranking

Our method together with several state-of-the-arts are evaluated on an external set of word pairs drawn from the SemEval 2010 Task 8 6 [Hendrickx et al., 2010], which is originally a multi-way classification of semantic relations between nominals. This external set contains 1730 nominal pairs in total, 865 of which are from the Cause-Effect relation and an equal number of which are from the other eight relations.

Figure 3 shows the precision-recall (PR) curves for these embedding methods. Two facts can be observed from it:

• First, our BMM method has achieved much higher coverage for it has covered almost all the test word pairs. However, the curves of all the other causal embedding models become straight at tail, because about 30% of test word pairs have at least one word missing from their training examples, which means that they have only 70% coverage of the test word pairs;
• Second, our BMM method is much more precise than cEmbed-family methods (inclusive of cEmbed, cEmbedBi and cEmbedBiNoise). As for Attentive-Matching and Max-Matching methods, their precisions both have a sharp drop after the recalls exceed 0.5, while the curve of BMM runs more smoothly, even on the range of high recall values.

As the result of the combination of the two observations, the PR curve of BMM runs above all the others, at almost all the positions, resulting in the highest Area-Under-PRCurve (AUC) value as shown in Table 1.

4.2 Top-Ranked English Patterns

Besides the boosted causal embeddings, our method outputs potential patterns with the calculated causal strengths by Equation (6), which can help detect and extract explicit (potential) causalities from text. The top-21 verb-mediated patterns are listed in Table 2. It can be observed that:

5 http://dumps.wikimedia.org/enwiki/20181120/ enwiki-20181120-pages-articles-multistream.xml.bz2

6 http://www.kozareva.com/downloads.html

• Based on human examination, all the listed patterns can indicate causality in some degree, and coincide with human intuition about causality;
• Among the top-21 verbs, 18 of them are excitatory, and only 3 are inhibitory. We also find that the excitatory and inhibitory verbs ususally have high ranks, which is related to the work of [Hashimoto et al., 2012];
• Interestingly, an inverted pattern result from inv is top-ranked, indicating that result from is a effect-tocause pattern. Such information is conducive to improving the accuracy of causality extraction.

5 Evaluation on Chinese Corpus

To make evaluation on Chinese, we use the Sogou 7 set of 517,746 high-precision causal phrase pairs which were extracted by Xie and Mu [2019] from the SogouCS 8 news corpus [Wang et al., 2008]. The causal embeddings 9 constructed by the Max-Matching algorithm [Xie and Mu, 2019] are used to generate the seed set of causal word pairs.

5.1 Causal Word Pair Identification

To evaluate the boosted causal embeddings on the task of identifying causal word pairs from causal phrase pairs, we use an external set of 355 phrase pairs 10 extracted from a Chinese encyclopedia corpus, where each phrase pair was annotated with the causal word pair that indicates the causality between the phrases[Xie and Mu, 2019].

Given a cause-effect phrase pair pp = (C, E) where C = c1c2 . . . cm and E = e1e2 . . . en, all the word pairs between C and E get ranked according to their interaction scores cs(·, ·) in Equation 1. For a word pair wp = (c, e) where c ∈ C and e ∈ E, let r(wp, pp) denote the rank of word pair wp with respect to the phrase pair pp. If the 1st ranked word pair is the same as the annotated causal word pair, then we say that the causal embedding has correctly identified the causal word pair. The accuracy of a causal embedding model on the test dataset is defined as the percentage of the correctly identified causal word pairs. The mean reciprocal rank (MRR) is calculated as:

7 http://www.ke.fudan.edu.cn/data/causal/sg hp extractions.txt

8 http://www.sogou.com/labs/resource/cs.php

9 http://www.ke.fudan.edu.cn/data/causal/sg causal embeds.zip

10 http://www.ke.fudan.edu.cn/data/causal/bk eval.txt

where ann(pp) denotes the annotated word pair for the phrase pair pp ∈ D.

The accuracies and MRRs are reported in Table 3. It can be seen that our Boosted Max-Matching model has achieved the highest performance on both accuracy and MRR, among all the competitors. In addition, the top-21 patterns ranked by their causal strengths are listed in Table 4, where observations can be made similar to Section 4.2.

6 Conclusion

In this paper, we propose a novel method to boost causal embeddings by exploring potential verb-mediated causal patterns. It first learns the pattern weights that indicate their causal strengths, and then boosts causal embeddings by fusing the existing strong causal phrase pairs and the extactions from (possibly weak) verb-mediated patterns. Experimental results have shown that the resulting causal embeddings have achieved high precision and recall on both Chinese and English. In addition, the top-ranked patterns, as byproducts, coincide with human intuition about causality.

Acknowledgements

This work is supported by National Key Research and Development Program of China (No.2018YFB1005100). We are grateful to anonymous reviewers for their valuable comments.