Event Factuality Identification via Generative Adversarial Networks with Auxiliary Classification

1 Introduction

Event factuality expresses the commitment of relevant sources towards the factual nature of events, conveying whether an event is characterized as a fact, a possibility, or an impossible situation. Event factuality identification is useful for deep NLP applications, such as opinion detection, temporal ordering of events, textual entailment, and rumor identification. In principle, event factuality is related to various factors, including predicates, speculative and negative cues. Two examples are given below:

• (S1) McCulley, a famous economist, doubts that the tax rate will increase soon.
• (S2) He knows they are not able to go to the village due to the flood.

In this paper, events are marked in bold and sources are underlined in example sentences. In S1, the event increase is possible (PS+) according to the predicate doubts, while in S2, the event go is impossible (CT-) due to the predicate knows and the negation word not. Table 1 shows that factuality can be characterized by the combination of epistemic modality and polarity. Modality conveys the certainty degree of events, such as certain (CT), probable (PR), and possible (PS), while polarity expresses whether the event has happened, including positive (+) and negative (-). In addition, U/u means underspecified. Some combined values are not applicable (NA) grammatically (e.g., PRu, PSu, and U+/- [Saur´ı, 2008; Saur´ı and Pustejovsky, 2012]), and are not considered.

Previous methods employed rules [Saur´ı, 2008; Saur´ı and Pustejovsky, 2012], machine learning models [de Marneffe et al., 2012; Lee et al., 2015], or a combination of the two [Qian et al., 2015; Stanovsky et al., 2017]. These approaches rely on annotated information, such as predicates, sources, speculative and negative cues, which are limited and can be costly to obtain. In addition, the performance of previous work is imbalanced on different values of event factuality. On one hand, the performance of speculative values is low due to their scarcity (4.36%) (e.g., [Saur´ı and Pustejovsky, 2012] achieved much lower performance of PR+ and PS+ compared to that of CT+ and Uu (F1: 45.71, 59.46 vs 84.85, 74.61) on Aquaint TimeML in FactBank using annotated data). On the other hand, events embedded in other predicates and sources (i.e., embedded events (31.04%)), which can have complicated syntactic structures (e.g., the event increase is embedded in the predicate doubts in S1), gave lower macroaveraged F1 than that of those events only with AUTHOR as sources (F1 67 vs 73 [Saur´ı and Pustejovsky, 2012]).

This paper proposes a two-step supervised framework to identify event factuality in raw texts, in which we first extract basic factors related with factuality (i.e., events, predicates, sources, and cues), and then utilize an Generative Adversarial Network with Auxiliary Classification for Event Factuality identification (EF-AC-GAN). To automatically produce more syntactic paths and improve the performance of embedded events with complicated syntactic structures, we utilize the generator in EF-AC-GAN to generate syntactic paths that are close to the distribution of real ones. In addition, to address the imbalance among factuality values and improve the performance of factuality values that are in minority (i.e., CT-, PR+, and PS+), we design two auxiliary classification tasks in EF-AC-GAN, one output deciding whether the event is Uu or Non-Uu, and the other indicating whether the event is modified by a cue and determining the modality and polarity of the event. Shortest Dependency Paths (SDP) are the main syntactic features for EF-AC-GAN.

Experimental results on a standard benchmark show that EF-AC-GAN outperforms the baselines significantly, especially on embedded events, speculative and negative factuality values. The code of this paper is released at https: //github.com/qz011/ef_ac_gan.

2 Basic Factor Extraction

This section introduces the basic factors related with factuality, namely events, SIPs, sources and cues, and presents the methods to identify them.

Events in FactBank are defined by TimeML [Pustejovsky et al., 2003]. For event detection we utilize the maximum entropy classification model [Chambers, 2013].

Source Introducing Predicates (SIPs) are events that can not only introduce additional sources but influence event factuality. For example, in S1 above, the SIP doubts introduces McCulley as a new source, and McCulley evaluates the event increase as PS+ according to doubts. We consider both lexical level features and sentence level features to detect SIPs. Similar to event detection, we consider the token, part-ofspeech (POS) and hypernym in WordNet 1 of the event as lexical level features and concatenate them into vector l.

A SIP has at least one embedded event [Saur´ı, 2008]. Hence, we propose Pruned Sentence (PSen) as a sentence level feature (i.e., a clause of an event is replaced by 〈event〉 if containing other embedded events; Nouns, pronouns and the current event are unchanged, while other tokens are replaced by 〈O〉). S4 is the PSen of the event says in S3.

(S3) Tom, who is the secretary of the manager, says the manager will attend a meeting later .

(S4) Tom 〈O〉 who 〈O〉 〈O〉 secretary 〈O〉 〈O〉 manager 〈O〉 says 〈event〉 〈O〉

Because PSen has a simplified structure, we extract sentence level features c from PSen X0 ∈ Rd0×n through an attention-based CNN instead of an RNN. The CNN and its objective function is defined as follows, where ⊕ is the concatenation operator, and Wc, bc, vc,Ws0 and bs0 are model parameters:

1 http://wordnet.princeton.edu

A Relevant Source is the participant of an event holding a specific stance with regard to the factuality. AUTHOR is always the source by default, and further sources (e.g. McCulley in S1) are represented in chain form [Saur´ı and Pustejovsky, 2012]: McCulley AUTHOR, which means that we know about McCulley's perspective only according to AUTHOR and McCulley is an Embedded Source in AUTHOR. The grammatical subjects of SIPs are chosen as the introduced new sources. After we have identified events and SIPs, we employ the recursive algorithm of [Saur´ı, 2008] to identify relevant sources.

Cues include speculative and negative words. PR/PS events are modified by PR/PS cues, while events can be negated by negative (NEG) cues (e.g., the factuality of event go in S2 is CT- due to the NEG cue not). [Velldal et al., 2012] concluded that lexical sequence-oriented n-gram features can achieve excellent results on cue detection. Hence, we employ the lexical features developed by [Velldal et al., 2012] to classify each token as PR/PS/NEG cue, or not cue.

3 AC-GAN for Event Factuality Identification 3.1 Overall Structure

GAN [Goodfellow et al., 2014] involves a generator G and a discriminator D, which are trained in opposition to one another. Due to the game-theoretic formulation, G produces samples that looks real, and D discriminates between generated samples and real ones. On top of GAN, AC-GAN consider auxiliary classification for class labels. In AC-GAN, G synthesizes samples conditioned on class labels, and D discriminates not only real and generated samples but assigns class labels for them. The objective function has two parts, i.e., the log-likelihood of the real samples LS and the correct

where D is trained to maximize LC +LS, and G is trained to maximize LC - LS .

We develop lexical and syntactic features according to the basic factors defined above, and consider Shortest Dependency Paths (SDP) from basic factors to events as syntactic features. As mentioned above, embedded and speculative/negative events are in the minority. In particular, events only with AUTHOR as sources are nearer to the root of the dependency tree, and their SDPs are simpler than those of embedded events. Hence, we design EF-AC-GAN for event factuality identification shown in Figure 1, where G generates SDPs conditioned on class labels, and D discriminates whether SDPs are generated and the auxiliary classification in D determines the class labels of events. We assign two class labels for each event: labelu, which represents whether the event is Uu, Non-Uu or other, and labelcue, which indicates whether the event is modified by a cue and further classifies Non-Uu events as CT+/-, PR+/-, PS+/-. Event factuality is determined directly by these two labels, which are demonstrated in detail below.

3.2 Features

Due to the success of dependency parse trees in previous work [Saur´ı, 2008; Saur´ı and Pustejovsky, 2012], we develop the following SDPs as syntactic features:

SIP Path is extracted from the ancestor SIP that introduces the sources to the event.

Relevant Source Path (RS Path) is extracted from the root of the dependency tree to the relevant sources of the event, and contains all the sources in the chain form.

Cue Path is extracted from the cue to the event.

SIP Path and RS Path are used to judge whether the event is Un, Non-Uu or other. In addition to Cue Path, we also consider the following cue-related lexical features to decide whether the event is governed by the cue:

Relative Position is the surface distance from the cue to the event, and is mapped into lp with dimensions dp.

Type of Cue includes PR, PS, and NEG, and is mapped into lt with dimensions dt.

If there is more than one cue in the sentence, we consider whether the current event is modified by each cue separately. An example sentence and its features for our EF-AC-GAN are shown in Figure 2.

3.3 The Discriminator

We utilize LSTM [Hochreiter and Schmidhuber, 1997] with hidden units nlstm in D to model the sequences. Bidirectional LSTM (BiLSTM) is used to access the future as well as past context of SDPs, producing forward/backward hidden sequences -→ H/←- H and the output sequence Hp = -→ H + ←- H. To capture the most important sources of information from the syntactic path, we adopt the attention model and obtain the output hp:

where p=SIP Path(sp), RS Path(rp), Cue Path(cp). We concatenate hsp and hrp into fu to judge whether the event is Uu, Non-Uu or other:

where ⊕ is the concatenation operator. To determine whether the event is governed by a cue, we consider not only hcp but the lexical features lp (Relative Position) and lt (Type of Cue) described above:

For the auxiliary classification of class labels, fu and fcue are fed into a softmax layer:

where Ws1, Ws2 are model parameters. ou represents whether the event is Uu, Non-Uu or other (labelu), and ocue is used to determine whether the event is governed by the cue (labelcue), and classify Non-Uu events as CT+/-, PR+/-, or PS+/-. We have two main reasons for the design of the two class labels. First, we can identify speculative and negative values (e.g., CT-, PR+/-, PS+/-) more precisely with the cues. Second, we can address imbalance among instances because speculative and negative values are typically in the minority.

In GAN it is essential to consider whether SIP Path (sp), RS Path (rp), and Cue Path (cp) are generated:

where p = sp, rp, and cp. The objective function of each output above is defined as:

where j = labelu, labelcue, fake sp, fake rp, fake cp, and y(i) j is the golden label of the corresponding output. The final objective function of D is :

3.4 The Generator

To produce more syntactic information and improve the performance of embedded and speculative/negative events, we generate the SDP S = x0, · · · , xt by feeding labelu and labelcue of events to the noise vector z, i.e., S = G(z, labelu, labelcue). LSTM is employed as the generator and generates a sequence of hidden states h0, . . . , ht:

where x0 is the input vector related to the noise vector z and the class labels:

where ⊙ is the element-wise multiplication, and vu and vcue are the embeddings of labelu and labelcue, respectively. SIP Path, RS Path, and Cue Path have their respective noise vectors, which follow the normal distribution and are initialized randomly. Finally, a softmax layer maps the hidden states into the output token distribution:

4.2 Results on Basic Factor Extraction

Table 3 presents the performance of basic factor extraction. It is worth noting that a correctly identified SIP means that both the SIP and the new source introduced by it are correctly detected. For the SIP detection task, we also employ the model of [Chambers, 2013], obtaining F1=72.56, while our CNN achieves a higher F1=73.66 (p < 0.05 on two-sample twotailed t-test). We argue that one SIP can determine ALL the sources of events embedded in it. Therefore, our CNN based on the PSen structures is effective.

4.3 Results on Event Factuality Identification

We employ the following baselines, whose features are developed according to the outputs of basic factor extraction, for fair comparison with our model:

Rules are developed by [Saur´ı, 2008] and [Saur´ı and Pustejovsky, 2012]. Instead of using annotated data directly, we obtain the performance of Rules using identified information according to basic factor extraction.

SVM is developed by [Saur´ı and Pustejovsky, 2012]. Besides, [de Marneffe et al., 2012] and [Lee et al., 2015] only considered AUTHOR as the source and employed traditional machine learning models. We re-implement them and obtain lower results than the SVMmodel on AUTHOR (macroaveraged F1 are 46.29 and 48.42, respectively).

ME+Rules: A two-step model combining a maximum entropy classification model and a simple rule-based model [Qian et al., 2015].

3.5 Training

When updating the discriminator D in each batch, we generate samples as the same number as the real samples, and the generated samples share the same Relative Position and Type of Cue with the corresponding real samples. D(or G) can become too strong and result in a gradient that cannot be used to improve G(or D). Therefore, we stop updating D when its training loss is less than 80% that of G, and also take the corresponding strategy when G is too strong.

To prevent G over-training on D, we utilize Feature Matching [Salimans et al., 2016] to encourage greater variance in G. Instead of maximizing the error of the output of D, the new objective requires G to generate data that match the real data. More specifically, we train G to match the feature representations on the penultimate layer of D. For each SDP, the objective is redefined as:

where R is the output of the BiLSTM layer before the final softmax layer, xp are SDPs, and p=SIP Path(sp), RS Path(rp), Cue Path(cp). The objective of G is defined as:

4 Experiments

4.1 Experimental Settings

We evaluate our models on FactBank [Saur´ı and Pustejovsky, 2009], which contains 3864 sentences and 13506 event factuality values. Table 2 presents the distribution of the values in FactBank. Following previous studies [Saur´ı and Pustejovsky, 2012; de Marneffe et al., 2012], we only consider the five main categories of values, i.e., CT+, CT-, PR+, PS+, and Uu, which make up 99.05% of all the instances.

For fair comparison, we perform 10-fold cross-validation on FactBank. In addition to Precision, Recall, and F1measure, macro- and micro-averaging are also applied to obtain the performance of all the factuality values. For SIP detection we set the dimensions of the POS and hypernym embeddings as 50 and λ = 10-4 . For event factuality identification, we set nlstm = 50 and dp = dt = 10. We initialize word embeddings via Word2Vec [Mikolov et al., 2013], setting the dimensions as d0 = 100, and fine-tuning them during model training. SGD with momentum is applied to optimize our models.

CNN-D is a variant of the EF-AC-GAN whose BiLSTM in the discriminator D is replaced by the CNN with the hidden units nc = 50.

L1 is a variant of the EF-AC-GAN, which concatenates fu and fcue in Equations (11) and (12) into ONE vector and consider only ONE class label of factuality.

BiLSTM utilizes BiLSTM in D of the EF-AC-GAN and does NOT consider generative model.

Table 4 shows the performance of various models on event factuality identification. Rules have low performance on Uu due to the error propagation from the basic factor extraction. SVM gets low results on CT-, PR+, and PS+ with minority events. For the model combining rules and machine learning classifiers, we employed the method of [Qian et al., 2015] and achieved the performance between Rules and SVM (microand macro-averaged F1 are 56.89 and 43.56, respectively). Compared to the traditional models that utilized complicated algorithms or features (19 features in SVM and 15 features by [Qian et al., 2015]), the features of our EF-AC-GAN are much fewer and easier to access.

Among all the models, EF-AC-GAN achieves the best results not only on CT+ and Uu but on CT-, PR+ and PS+ with All sources. The performance gaps among different factuality values illustrate that it is challenging to identify CT-, PR+ and PS+, which only cover 7.57% of all the instances. Compared to SVM, EF-AC-GAN improves the F1 of CT-, PR+ and PS+ by 10.01, 17.83 and 20.27, respectively. All the improvements are significant with p < 0.05 applying twosample two-tailed t-test (the same below). The improved performance on CT+ and Uu means that EF-AC-GAN can effectively discriminate Non-Uu from Uu events, which can contribute to the outstanding results of CT+/-, PR+/- and PS+/-.

For both AUTHOR and Embedded Sources, EF-AC-GAN significantly outperforms the other methods on the microaveraged and macro-averaged F1. The performance of CT-, PR+, PS+, and embedded sources shows that the generator G in EF-AC-GAN can produce meaningful dependency paths and EF-AC-GAN can benefit from G effectively. Besides, the results of events with embedded sources are lower than those with AUTHOR in EF-AC-GAN, mainly due to the complex syntactic structures of the events embedded in other sources. Similar to other models, the F1 of Uu for embedded sources is quite low (24.97) indicating that it is difficult to discriminate Uu from Non-Uu for embedded events.

To further verify the advantages of our model, we implement two variants of EF-AC-GAN, namely CNN-D and L1, as baselines. CNN ignores the context, while EF-AC-GAN considers both future and past context in syntactic paths and is superior to CNN-D. Compared with L1, both BiLSTM and EF-AC-GAN can obtain much better performance on CT, PR+, and PS+, which can demonstrate that the design of the two class labels in auxiliary classification can address the problem of data imbalance. EF-AC-GAN also outperforms BiLSTM that does not consider the generative model, especially on PR+, PS+, and embedded sources, which illustrates that EF-AC-GAN can benefit from those generated samples. Moreover, if we only consider whether the syntactic paths are generated and neglect the auxiliary classification for the class labels of event factuality, we obtain poor performance (i.e., micro- and macro-averaged F1 are 17.85 and 20.79, respectively), which proves that class labels of event factuality are important supervised information for the training of EF-ACGAN, and the auxiliary classification tasks are effective.

In conclusion, the experimental results of the neural network models above show that:

1. The design of the two class labels in the auxiliary classification task can improve the performance of speculative and negative factuality values (i.e., CT-, PR+, PS+) compared to the use of one class label.
2. The generator in EF-AC-GAN can produce useful and

meaningful dependency paths to offer more syntactic information, and can improve the performance of the speculative/negative and embedded events in minority.

We investigate the relationship between the data size of FactBank and the performance of EF-AC-GAN, showing the results in Figure 3. Both the micro- and macro-averaged F1 of EF-AC-GAN become steady when we use more than 10000 samples. Particularly, in term of the performance of EF-AC-GAN on the whole FactBank, standard deviations of the micro- and macro-averaged F1 in 10 folds are less than 3, indicating that we obtain steady results.

To explore the upper bound of the performance of EF-ACGAN, Table 5 shows the performance of Rules, BiLSTM and EF-AC-GAN with annotated information. Comparing Table 4 and 5, the micro- and macro-averaged F1 of Rules are improved by 30.81 and 29.38, respectively, and those of EF-AC-GAN by 18.34 and 18.57, respectively, which illustrates that the performance of Rules relies much more heavily on annotated information compared to EF-AC-GAN. Compared to Rules and BiLSTM, EF-AC-GAN achieves higher F1 of CT-, PR+, and PS+, indicating that our model is better at identifying speculative and negative factuality values. For embedded events, EF-AC-GAN is superior to BiLSTM on both micro- (73.79>72.67) and macro-averaged F1 (68.11>65.33). Therefore, the EF-AC-GAN model is more effective than BiLSTM, which does not consider GAN.

4.4 Error Analysis

We analyze the errors produced by the EF-AC-GAN model, which can be classified into the following main categories:

Incorrect relevant sources. Our model fails to identify correct relevant sources for events due to the error propagation from the basic factor extraction tasks (72.64%). These errors prove the significance of the SIP and relevant source detection task.

(S5) Parliamentary president Rita Suessmuth said the People's Union correctly listed Jacques de Mathan as having donated about 274,500 marks in 1995, but gave a false address for him.

(Event: donated, Source: Union AUTHOR, Uu)

For example, event donated is Uu according to Union in S8, but we fail to identify Union as the new source introduced by the SIP listed and miss the source Union AUTHOR for donated.

Incorrect Non-Uu/Uu. Whether an event is Non-Uu or Uu is mistakenly identified (EF-AC-GAN: 22.57%, BiLSTM: 25.08%). In S6, the BiLSTM model evaluates the event change as Uu incorrectly, and cannot assign PR+ to change even if determines that change is governed by the PR cue appears. In S7, BiLSTM classified the event take as Non-Uu and assign CT- to it according to the negative cue not incorrectly. While EF-AC-GAN identifies change as PR+ and take as Uu correctly, proving the usefulness of the auxiliary classification and the generated syntactic paths that can offer more syntactic information.

• (S6) Everyone appears to believe that somehow Cuba is going to change.

(Event: change, Source: Everyone AUTHOR, PR+)

• (S7) He added, "This has nothing to do with Marty Ackerman and it is not designed, particularly, to take the company private."

(Event: take, Source: He AUTHOR, Uu)

Incorrect Modality/Polarity. The modality or polarity of an event cannot be identified correctly because the model fails to determine whether the event is governed by a cue (EF-ACGAN: 4.14%, BiLSTM: 4.39%).

(S7) He indicated that some assets might be sold off to service the debt.

(Event: service, Source: He AUTHOR, PS+)

• (S8) There was no hint of trouble in the last conversation between controllers and TWA pilot Steven Snyder.

(Event: conversation, Source: AUTHOR, CT+)

In S7, the event service is PS+ according to He due to the PS cue might. The BiLSTMmodel did not identify might for the event service, while EF-AC-GAN gave the correct result. In S8, the event conversation is not governed by the negative cue no and is annotated as CT+. However, EF-AC-GAN regards no as the NEG cue of conversation and evaluates it as CT- mistakenly, indicating that EF-AC-GAN may overfit to the generated paths.

5 RelatedWork

Event factuality identification is a challenging task. Many studies limited the sources to the reader or AUTHOR. [Diab et al., 2009] and [Prabhakaran et al., 2010] presented a study of belief annotation and classified predicates into Committed Belief (CB), Non-CB or NA under a supervised framework. Recently, researchers presented scalable annotation schemes [Lee et al., 2015; Stanovsky et al., 2017], and developed new corpus [Soni et al., 2014; Prabhakaran et al., 2015]. To predict the factuality, [Lee et al., 2015] employed lexical and dependency features, and [Stanovsky et al., 2017] developed deep linguistic features.

FactBank [Saur´ı and Pustejovsky, 2009] considers both AUTHOR and embedded sources. Previous studies [Saur´ı, 2008; Saur´ı and Pustejovsky, 2012] proposed a complicated rule-based method to identify factuality of events on FactBank. [de Marneffe et al., 2012] proposed a new annotation framework and identified the factuality of events in some sentences of FactBank. [Qian et al., 2015] utilized a twostep framework combining machine learning and simple rulebased approaches.

Previous work also employed neural networks for factuality identification, but only considering the coarse-grained sentence-level factuality, e.g., uncertainty detection [Adel and Sch¨utze, 2017] and modal sense classification [Marasovi´c and Frank, 2016] for sentences, while this paper focuses on event factuality identification via AC-GAN instead of sentence factuality.

Syntactic paths [Roth and Lapata, 2016] and attention [Chen et al., 2016; Zhou et al., 2016] are helpful for many neural network-based NLP applications. Hence, we consider BiLSTM with attention to learn features from dependency paths in the discriminator of AC-GAN.

Generative Adversarial Networks (GAN) [Goodfellow et al., 2014] aim at fitting generative models to the distribution of realistic data, and have been proven successful in various AI and NLP applications, such as speech recognition [Chang and Scherer, 2017], image synthesis [Ghosh et al., 2017], and text generation [Yu et al., 2017]. Compared with GAN, AC-GAN [Odena et al., 2017] considers both class labels of samples and the synthesis of sequences, and sets them as different outputs. AC-GAN is mainly applied on image synthesis [Dash et al., 2017; Zhang et al., 2017] instead of factuality-related NLP tasks. This paper applies AC-GAN to event factuality identification.

6 Conclusion

We presented a two-step framework for event factuality identification, which first extracts various basic factors, such as events, source introducing predicates, relevant sources and cues from the texts, and then employs EF-AC-GAN to identify event factuality. The design of auxiliary classification tasks in the discriminator can address the data imbalance among factuality values, and the generator can produce more syntactic information to improve the performance of speculative/negative and embedded events in minority. Experimental results show that EF-AC-GAN is superior to several state-ofthe-art methods, especially on embedded events, and speculative and negative factuality values. To our best knowledge, this is the first work to apply AC-GAN to event factuality identification.

Acknowledgments

The authors would like to thank the four anonymous reviewers for their comments on this paper. This research was supported by the National Natural Science Foundation of China under Grant Nos.61751206, 61772354 and 61773276, and was also supported by the Strategic Pioneer Research Projects of Defense Science and Technology under Grant No. 17ZLXD-XX-02-06-02-04.