Evidence-Aware Hierarchical Interactive Attention Networks for Explainable Claim Verification

1 Introduction

There are a large number of unverified claims on social media, such as hoaxes and fake news, which are a widespread menace that has resulted in protests and violence around the globe. Research indicates that fake news accounts for nearly 6% of all news consumption during the US presidential election (2016) [Grinberg et al., 2019], and even some institutions utilize Facebook to selectively expose false claims that affect voters' attitudes [Guess et al., 2018]. Vosoughi et al. [2018] stated that effects were more pronounced for false political news than for false news about terrorism, natural disaster, science, urban legends or financial information. These have driven social media organizations and government institutions to scramble together to tackle this problem.

1 Corresponding Author.

Currently, the research for claim verification could be divided into three stages: 1) Hoax-debunkers 12 are employed to manually assess whether specific claims are true or false, which guarantees high accuracy, but it is difficult to cope with the increasing number of unverified claims; 2) Various methods for automatic claim verification are proposed based on deep neural networks, which learn credibility indicators from the perspectives of semantics [Ma et al., 2018; Wu et al., 2020], emotions [Ajao et al., 2019], write styles [Gr¨ondahl and Asokan, 2019], and stances [Wu et al., 2019] from claims. Although being effective, they cannot explain one claim's correctness in practice; and 3) A recent trend is to discover evidence from relevant articles for explainable claim verification, which designs interactive models to explore the relationships between claims and the articles through semantic conflicts [Popat et al., 2018; Shu et al., 2019; Wu and Rao, 2020], semantic matching [Nie et al., 2019], and semantic entailments [Ma et al., 2019].

However, there are several general drawbacks of the evidencebasedmethods. First, they capture the top-k check-worthy articles or salient words as evidence, lacking focus on the questionable parts of unverified claims, which makes the evidence unable to provide an accurate and effective explanation for the verification results. Second, they exploit relevant articles indiscriminately, neglecting the source credibility of these articles, which may lead to the untrustworthy features in unreliable articles interfering with the final results.

To address the above problems, we propose Evidenceaware Hierarchical Interactive Attention Networks (henceforth, EHIAN) to explore more credible evidence semantics for explainable claim verification. Specifically, in EHIAN, to focus on fragments of evidence from relevant articles, we design an internal interaction layer (IIL) that adopts the combination of self-attention networks and symmetrical attention networks to facilitate full interaction of the claim and each relevant article. In order to earn the more credible evidence semantics discussing the questionable parts of claims, we develop global inference layer (GIL) that devises gated affine transformation to coherently integrate the source features of articles and evidence fragments, and explores global interaction inference to promote the interaction between claims and relevant articles again by combining the average semantics of all articles for mitigating the impact of extreme semantics on result discrimination. Experimental results reveal that EHIAN not only achieves the state-of-the-art performance but also discovers effective evidence for explaining the results. Our contributions are summarized as follows:

1 https://www.snopes.com

2 https://www.politifact.com

• We propose a novel explainable claim verification framework based on hierarchical interactive networks, which not only captures the questionable parts of claims but also wins the more credible evidence semantics aiming at the parts.
• Developed global inference layer focuses on more credible evidence from key fragments through proposed gated affine transformation, and alleviates the impact of extreme voices on the results with the help of global interaction inference.
• We report state-of-the-art performance in two datasets.

2 RelatedWork

The existing work endeavors to understand the differences between false and true claims from different perspectives, especially in terms of claim content and sources.

Claim Content. In order to reveal linguistic differences between true and false claims, shallow features [Kakol et al., 2017; Potthast et al., 2018], and deep features, like semantic [Ma et al., 2018], stance-based [Dungs et al., 2018], emotional [Giachanou et al., 2019], and stylistic [Gr¨ondahl and Asokan, 2019] have been exploited. As an example, Rashkin et al. [2017] compared the language of true claims with that of satire, hoaxes, and propaganda to find linguistic characteristics of untrustworthy text. Giachanou et al. [2019] incorporated emotional signals extracted from the text of claims to differentiate between credible and noncredible ones. More recently, the methods that explore evidence to enrich the semantics of claims to improve performance are recognized, which develop effective interaction models to discover the relationship between evidence articles and claim content by considering semantic matching [Nie et al., 2019], semantic conflicts [Popat et al., 2018; Shu et al., 2019], and textual entailment [Ma et al., 2019]. Popat et al. [2018] proposed an evidence-aware attention model that aggregates salient words from source news articles as the main evidence for finding false claims. Ma et al. [2019] employed representation learning to embed sentence-level evidence based on textual entailment and natural language inference. In this work, different from the existing evidencebased models only capturing check-worthy sentences or salient words as evidence, the hierarchical interactive model we developed discovers more credible and accurate evidence semantics for catching the questionable parts of claims.

Source Credibility. The source credibility of claims is crucial auxiliary information for claim verification. As false claims are usually published by unbelievable individuals or automatic bots, source credibility plays a crucial role in message communication [Shu et al., 2017]. With the aid of information sources, Popat et al. [2016] found that tweets from highly credible institutions and individuals are mostly correct and then made use of source reliability of articles reporting the claim to assess its credibility and obtain satisfied assessment results. Thus, when information comes from different websites, the credibility of websites represents the credibility of information to a certain extent [Li et al., 2016]. Considering that most evidence-based methods usually ignore the articles' source credibility, we measure and integrate multiple metadata features of article sources for obtaining more credible evidence semantics.

3 Evidence-aware Hierarchical Interactive Attention Networks

As shown in Figure 1, EHIAN consists of a 4-level hierarchical structure: input embedding layer, internal interaction layer, global inference layer, and task learning layer.

3.1 Input Embedding Layer

The inputs of EHIAN include a claim sequence and n relevant article sequences. For any sequence containing l tokens, it can be expressed as X = {x1, x2, ..., xl}, X ∈ Rd×l , where each token xi ∈ Rd is a d-dimensional vector obtained by pre-trained BERT model [Devlin et al., 2019]. Here, the embeddings of one claim and n relevant articles are respectively represented asXc,Xa 1, ..., andXa n . Particularly, we use the average embeddings of all relevant articles as a newly-synthesized relevant article Xa avg (the right part in Figure 1), which serves as the semantic benchmark of relevant articles to attend to Global Interaction Inference (Section 3.3) for alleviating the interference of extreme semantics in individual articles.

3.2 Internal Interaction Layer (IIL)

In order to capture key evidence fragments, we design internal interaction layer (IIL) consisting of two self-attention modules and a symmetrical interaction module to enable deep interaction and matching between claims and relevant articles.

Self-attention Module. We adopt multi-head self-attention mechanism [Vaswani et al., 2017] to explicitly learn the dependencies between any two characters in sequence and capture the inner structure information of sequence. Given query matrix Q ∈ Rl×d , key matrix K ∈ Rl×d , and value matrix V ∈Rl×d , the scaled dot-product attention is described as

where we set Q=K=V =Xc in the claim, and Q=K= V =Xa i in the relevant article.

Multi-head attention first linearly projects the queries, keys, and values h times by utilizing different linear projections. Then h results perform the scaled dot-product attention in parallel. Finally, the results of attention are concatenated and once again projected to get the new representation. Formally, the multi-head attention could be formulated as:

where W Q i , W K i , W V i (all ∈ Rd×H ), and W o ∈Rd×d are trainable parameters and H is d/h. O=Oc and O=Oa i are the outputs aiming at the claim and the relevant article i respectively.

Symmetrical Interaction Module. We design a symmetrical attention mechanism to boost in-depth interaction and matching between the claim Oc and each relevant article Oa i . The process can be formalized as follows:

where S(·) is the symmetric function [Huang et al., 2018], f is an RELU activation function. D is a diagonal matrix, where D and W (both∈Rd×d ) are parameters. Intuitively, each element of Oa i is weighted by an importance score defined by the similarity of an element of the claim Oc and that of the relevant article Oa i . The result Ii is the interactive semantics.

Integration. Finally, we integrate the long-term dependencies of the claim and the article, and interactive semantics between the claim and the relevant article to gain evidence fragments.

where ; denotes the concatenation operation.

3.3 Global Inference Layer (GIL)

In order to discover the more credible evidence semantics discussing the questionable parts of claims, we design GIL including two components, i.e., gated affine transformation for fusing the source features of relevant articles to infer more credible evidence semantics, and global interaction inference for mitigating the impact of extreme semantics on results.

Gated Affine Transformation. Following "the more credible the information source is, the more authentic the information will be", we utilize source credibility of relevant articles to reflect indirectly their credibility. Specifically, we enrich source credibility by extracting meta-data features of source websites of relevant articles from Alexa 3 based on websites' domains (due to the original datasets do not include the details of these websites), where these meta-data features involve the following items from the perspectives of authority, activity, and popularity: 1)Authority: domain suffix, sites linking in, visited just before, and visited right after; 2) Activity: daily pageviews per visitor, Alexa rank 90 day trend, search traffic, bounce rate, and daily time on site; and 3) Popularity: site's audience interests, interest level, and top keywords by traffic. Then we quantify them and finally form a 300-dimensional credibility vector Acred i .

3 https://www.alexa.com/siteinfo

To deeply integrate the source credibility of relevant articles with their semantics, instead of the traditional concatenation, we elaborately devise gated affine transformation to incorporate the credibility vector Acred i into the outputs of IIL aiming at the article i. Specifically, we first employ linear transformation to map the credibility vector Acred i and the interactive semantics Ei to obtain the mapping credibility vector α(Acred i ), scaling vector β(Ei), and shifting vector γ(Ei), respectively. Then, a gate mechanism with a sigmoid function σ(·), generates a mask-vector from the scaling vector with values between 0 and 1 to select credibility semantics. Finally, the shifting vector adjusts slightly the credibility semantics to achieve the appropriate fusion. Formally, the process can be formalized as follows:

where Wα, Wβ, Wγ, bα, bβ, bγ are learnable parameters and ⊙ denotes element-wise multiplication. Especially, the integration of the average credibility vector Ecred avg of all source vectors is the same as the single relevant article's, Ecred avg reflects the overall credibility of relevant articles under the claim.

Global Interaction Inference. To balance and mitigate the interference of extreme semantics in some relevant articles, we introduce the average integration Ecred avg of all relevant articles into the interaction again between the claim and each relevant article. Specifically, we adopt self-attention networks to achieve the interaction of the three types of semantic features, where the structure of networks is same as self-attention module in IIL. Here, we adopt the outputs Oc of self-attention module in IIL at claims as query Q, key K is the outputs Ecred i of gated affine transformation concerning each relevant article, and value V is the average interactionEcred avg about average relevant articles. Finally, we adopt concatenation to integrate all outputs of GIL.

where Gi denotes the outputs of global inference layer of single relevant article i and Gall means the outputs of global inference layer of all relevant articles. At this time, F contains the highly trusted evidence semantics discussing the questionable parts of claim.

3.4 Task Learning Layer

As the last layer, softmax function first emits the prediction of probability distribution for task learning. Then, a global loss function forces the model to minimize the cross-entropy error for a training sample with ground-truth label y:

4 Experiments

4.1 Datasets and EvaluationMetrics

We use two public fact-checking datasets, i.e., Snopes and PolitiFact provided by Popat et al. [2018] for evaluation. Content. Both contain news claims along with their credibility labels, sets of relevant articles, and their respective web source domains. Labels. Snopes is labeled as true and false while PolitiFact is originally assigned six credibility labels: true, mostly true, half true, mostly false, false, and pants on fire. Like Ma et al. [2019], we merge the six labels into true, false, and mixed. In detail, mostly true, half true, and mostly false are consolidated as mixed, and treat false and pants on fire as false. Distribution. Snopes and PolitiFact include 4,341 and 3,568 news claims, and 29,242 and 29,556 relevant articles that both include respectively a total of 336 web sources.

Additionally, we adopt micro-/macro-averaged F1, classspecific precision (P), recall (R), and F-measure (F1) as evaluation metrics. We hold out 10% of all data as validation data for parameter tuning, and conduct 5-fold cross-validation on the remaining 90% of the data.

4.2 Settings

For parameter configurations, we apply the pre-trained BERTbase model [Devlin et al., 2019] to initialize our token embeddings and their length is set to 300. Due to no parameter depends on the number of relevant articles n, instead of intercepting a fixed number, we set n to vary with claims in datasets. In self-attention networks, attention heads and blocks are set to 6 and 2 respectively, and the dropout of multi-head attention is set to 0.7. Moreover, all the models are trained to use Adam optimizer [Kingma and Ba, 2014] with a learning rate of 0.002 and mini-batches of size 64 to minimize categorical cross-entropy loss. We employ L2-regularizers with the fully connected layer. Also, the dropout is 0.5.

4.3 Performance Comparison

We compare EHIAN and several state-of-the-art baselines.

• SVM detects fake news relying on manually extracted features (e.g., bag-of-words, ngrams, etc.) from relevant articles [Thorne and Vlachos, 2018].
• CNN [Wang, 2017] captures semantics by different convolutional window sizes for fake news detection. Here, we only use claim content without considering meta-data features.
• LSTM [Rashkin et al., 2017] takes the word sequence of a claim as the input and predicts credibility rating.
• DeClarE [Popat et al., 2018] presents attention networks to aggregate signals like salient words from external relevant articles and their sources for claim verification.
• HAN [Ma et al., 2019] focuses on learning coherent evidence as well as their semantic relatedness with the claim.
• HAN-ba [Ma et al., 2019] is a variant of HAN that uses the biaffine attention to replace gated attention as the coherence component.

We implement our model with Tensorflow 4 . Due to DeClarE is not open-source, we adopt the results provided by the experiments of Ma et al. [2019] with Theano 5 . Other baselines are implemented through their source codes.

Table 1 shows the experimental results and we observe that:

• CNN and LSTM only using content features are comparable with SVM incorporating many handcrafted features, which reveals neural networks indeed help to learn better hidden representation. DeClarE outperforms CNN and LSTM because it not only learns deep features via the neural model but also focuses on the differential features like salient words by attention networks. HAN and HAN-ba achieve better performance than DeClarE, which consider the relationship like coherence and textual entailment between relevant articles.
• Our model obtains competitive performance than other baselines. Unlike HAN and HAN-ba that treat all relevant articles equally, our model takes the credibility of relevant articles into account to strengthen high-reliability evidence semantics. Unlike DeClarE only capturing salient words as evidence, our model learns evidence semantics discussing the questionable parts of claims through hierarchical interaction between claims and articles.

4.4 Discussion

Ablation Analysis

To evaluate the effectiveness of different components of EHIAN, we ablate EHIAN into the following simplified models: -IIL denotes that EHIAN removes IIL. -Interact means EHIAN replaces symmetrical interaction module in IIL with concatenation. We remove GIL from EHIAN as -GIL. -Affine means gated affine transformation is replaced by concatenation in EHIAN. -Avg. means that the average semantics of all relevant articles are no longer the values of self-attention networks in GIL, but are replaced by the outputs of each gated affine transformation.

As shown in Table 2, we have the following observations:

4 https://www.tensorflow.org/

5 http://deeplearning.net/software/theano

• Effectiveness of internal interaction layer. EHIAN boosts about 7.1% and 5.7% in micF1 on Snopes and PolitiFact respectively compared with -IIL, which indicates the interaction between claims and articles via IIL is effective.
• Effectiveness of global inference layer. When compared with -GIL, EHIAN significantly improves performance with the help of GIL, showing 9% and 6.8% boost in micF1 on the two datasets respectively, which explains that the effectiveness of GIL to obtain more credible evidence semantics discussing the questionable parts of claims.
• Effectiveness of symmetrical interaction module. Analysis of the results of -Interact and EHIAN, EHIAN obtains the better performance relying on the interaction module, which illustrates that the effectiveness of EHIAN using symmetrical attention mechanism to achieve the interaction.
• Effectiveness of gated affine transformation. By introducing gated affine transformation, EHIAN improves the performance as compared with -Affine, showing 2.7% and 2.3% improvement in macF1 on the two datasets, respectively. It proves the effectiveness of gated affine transformation fusing meta data and semantics.
• Effectiveness of global interaction inference. EHIAN consistently outperforms -Avg., presenting 1.3% and 1.8% boost in macF1 on the two datasets, respectively, which confirms that EHIAN designing global interaction inference module to mitigate extreme bias voices is effective.

Evaluation of Gated Affine Transformation

To further evaluate the superiority of the fusion mode for gated affine transformation (i.e., gate+affine) in GIL, we conduct experiments to compare with the following strategies: conc.: Semantics and meta data (a.k.a. the credibility vector) are fused through concatenation as a baseline. add.: We use additions to integrate meta data and semantics as a baseline. heuristics: Matching heuristics [Mou et al., 2016] is used to fuse meta data and semantics. affine: Attentional affine transformation [Margatina et al., 2019] replaces gate+affine to incorporate meta data and semantics. gate: We adopt attentional featurebased gating [Margatina et al., 2019] to combine meta data and semantics. gate+emb.conc.: We utilize the combination of attentional feature-based gating and attentional concatenation [Margatina et al., 2019] to fuse meta data and semantics. The experimental results are illustrated in Table 3, we observe that:

• Compared with two baseline strategies, the improved methods achieve varying degrees of boost (up to 2.7% improvement in micF1) on the two datasets, where gate mechanism relying on filtering features gains better performance than heuristics and affine. gate+emb.conc. taking a combined way reflects the best performance in all improved methods.
• Our method outperforms the improved methods on the two datasets, showing from 1.1% to 1.4% boost in macF1 in comparison with gate+emb.conc., which illustrates the superiority of our gated affine transformation.

4.5 Case Study

The Visualization of Features Learned from GIL

We examine the capture of valuable features in two components of GIL. In the first component, we respectively map the outputs of both gated affine transformation and IIL (not using meta-data features of source websites) to the input elements

(a) The model using gated affine transformation (b)The model not using gated affine transformation

Figure 2: Interpretation via visualization of gated affine transformation in GIL. [magpies.net/tumblr.com] denotes the source websites.

of word-level. The visualized results are described in Figure 2, where the boxes are the captured evidence fragments. We observe that the model integrating meta-data features more focuses on the evidence fragments in the website with higher credibility ('tumblr.com'), i.e., 'article clearly reads as a joke', and accurately capture the evidence semantics 'clearly' and 'joke'. Conversely, the model not relying on meta-data features equally captures the keywords from relevant articles in different credibility websites, which misleads the model to obtain unreliable evidence fragments from low credibility website ('magpies.net'), resulting in some non-evidence salient words being captured, like 'receiving', and 'judge described'.

Additionally, in the same way, we respectively get the visualization whether the model integrates the average integration Ecred avg in global interaction inference, as shown in Figure 3. We observe that the integrated model captures more accurate keywords such as 'not gonna' and 'no doubt', which is not affected by some extreme words, like 'worst' and 'threaten', and finally determines that the claim is true. Conversely, the model not integrated does not consider the global semantics of all relevant articles so that some extreme words are captured, which ultimately misleads the evaluation result (i.e., false).

Figure 4: Interpretation via visualization of attention weights in EHIAN. [True/False] indicates the labels of claims.

The Visualization of Features Learned from EHIAN

We visualize the features finally learned from EHIAN. Specifically, we first look up these elements with the largest values from the entire outputs of GIL, and then these elements are mapped into the corresponding values in input embeddings so that we are capable of finding the specific tokens. The visualization results are depicted in Figure 4. We observe that EHIAN gives more attention to the words that are related to the claim, like 'free market' and 'taliban prisoners'. More importantly, we also observe that: Explainability. In claims, EHIAN captures the queationable parts of claims with high probability, such as 'destructive force' and 'taliban prisoners held guantanamo bay'. In relevant articles, EHIAN can also capture the key evidence fragments that discusses the questionable parts of claims, e.g., 'misquotation of clinton' and 'wrongly attributed a quote' related to false claim, 'true example collected via email' related to true claim, which is conducive to explaining the verification results of our model.

5 Conclusion

In this paper, we propose evidence-aware hierarchical interactive attention networks (EHIAN) to explore more credible evidence discussing the questionable parts of claims for explainable claim verification. EHIAN first strengthens the interaction between claims and relevant articles to discover key evidence fragments, and then incorporates source features of articles and mitigates the interference of extreme semantics to explore more credible evidence discussing the questionable parts of claims. Experiments on two datasets confirm the effectiveness and interpretability of EHIAN. In the future, we plan to expand the work by enhancing inference to discover conflicts between claims and relevant articles.

Acknowledgments

The research work was supported by National Key Research and Development Program in China (2019YFB2102300); The World-Class Universities (Disciplines) and the Characteristic Development Guidance Funds for the Central Universities of China (PY3A022); Ministry of Education Fund Projects (18JZD022 and 2017B00030); Shenzhen Science and Technology Project (JCYJ20180306170836595); Basic Scientific Research Operating Expenses of Central Universities (ZDYF2017006); Xi'an Navinfo Corp.& Engineering Center of Xi'an Intelligence Spatial-temporal Data Analysis Project (C2020103); Beilin District of Xi'an Science & Technology Project (GX1803). We would like to thank the anonymous reviewers for their constructive comments.