Learning to Identify Ambiguous and Misleading News Headlines

1 Introduction

With the rapid development of the Internet, a variety of online news websites spring up and attract a great number of readers by providing much convenience and a wealth of information. However, the prosperity of online journalism simultaneously brings about problems including inaccuracy, unfairness, and subjectivity. We are to solve the problem of inaccurate news headlines since they are frequently complained about by readers.

With the purpose of attracting clicks, online news publishers use diverse strategies to make their headlines catchy. Generally, those clickbait headlines can be classified into two categories, namely ambiguous ones and misleading ones [Marquez, 1980]. Ambiguous headlines make use of curiosity gap by concealing key information of a news event. As a result, readers tend to click on the links and find out the missing information. Misleading headlines always exaggerate or distort the fact described in news bodies. Readers will discover inconsistencies after reading the whole passage, only to be left disappointed.

News with inaccurate headlines is unacceptable. It is more than a violation of professional ethics for editors. From the perspective of users, the content of this kind of news hardly lives up to their expectations comparing with the catchy headlines. Readers usually decide whether it is worth their time to read a story through headlines, so tricky headlines will waste their time and degrade the user experience. More importantly, it has been proved that headlines can shape public opinion [Tannenbaum, 1953]. According to the study, readers think differently of the accused after reading news with headlines slanted toward either innocence or guilt. From the above, we can see that it is necessary to solve the problem. Hence, we turn to the study of automatically identifying ambiguous and misleading headlines.

In this paper, we redefine the problem using Journalism and Communication knowledge. Instead of simply treating each piece of news as clickbait or non-clickbait, we separately judge whether it is ambiguous and whether it is misleading. For the ambiguous headline detection task, we mine class sequential rules (CSR) to make use of sequential information. Previous work mostly employs features extracted from headlines [Chakraborty et al., 2016; Anand et al., 2016]. Biyani et al. [2016] utilized the informality of news body, but the consistency is still underappreciated. In our method for misleading headline detection, we extract features based on headlines, bodies and the relationship between them. Since we only annotate a small part of data set, we also design a semi-supervised method, co-training, to exploit the unlabeled news. Body-independent and body-dependent features are utilized to train the sub-classifiers.

The experiment results show the effectiveness of CSR features for the ambiguous headline detection task and the consistency features for the misleading headline detection task, and performance improves when co-training is used. With the final classifiers, we classify news crawled from four major Chinese news websites, which cover different categories such as sports, society and world news. Then we conduct a data analysis upon the results.

2 RelatedWork

In Communication and Psychology, there have been studies on the accuracy of news headlines since decades ago. Marquez [1980] divided news headlines into three types, namely accurate, ambiguous and misleading ones, and proposed specific definitions, which are subsequently used in our classification. Ecker et al. [2014] analysed the effect of misinformation in news headlines and showed that such headlines lead to misconception in readers. Several properties and structures of clickbait headlines were dug out manually [Molek-Kozakowska, 2013; 2014; Blom and Hansen, 2015], providing an entry point for preliminary automatic identification of clickbaits.

It was not until recent years that some studies on clickbait detection emerged in the field of artificial intelligence. Chakraborty et al. [2016] extracted a set of features from headlines to train a clickbait classifier. Instead of using hand-crafted features, Anand et al. [2016] tried RNN method with word embeddings as inputs. Some potential non-text cues, such as user behavior analysis and image analysis, were discussed but not implemented by Chen et al. [2015]. Biyani et al. [2016] further utilized both title-based and body-based (article informality) features to identify clickbait news. However, all the above works hardly considered the relationship between headlines and bodies, and focused more on the properties of headlines. In this paper, we clearly define the problem with Marquez's professional view [1980] rather than the general wording "clickbait", detecting ambiguous headlines and misleading ones separately. In the latter task, the consistency between headlines and bodies is especially taken into account.

3 ProblemDefinition and Corpus

3.1 ProblemDefinition

From the perspective of Journalism and Communication, news headlines are classified into the three categories below [Marquez, 1980]. Note that a headline can be both ambiguous and misleading. In this paper, we divide our task into two separate parts: ambiguous headline detection and misleading headline detection.

Accurate Headlines

An accurate headline is a headline that is congruent in meaning with the content of the news story.

Such headlines are non-clickbaits.

Ambiguous Headlines

An ambiguous headline is a headline whose meaning is unclear relative to that of the content of the story.

It is typical of ambiguous headlines to omit some key information. The lack of knowledge arouses reader's curiosity and lures them to click. For example:

/¤·Q†�›.)�§�y3���u�"0

("She once won the world championships, but now worries about making a living.")

Misleading Headlines

A misleading headline is a headline whose meaning differs from that of the content of the story.

The differences can be either subtle or obvious. Some common tactics are exaggeration, distortion, etc, most of which aim to cause a sensation. For example:

/�‡q~fi†/�{I����0

("Asian carp has become the nightmare of America!")

In the news body of this example, the overgrowth of Asian carp due to the lack of natural predators is described. The word "nightmare" and the exclamation mark actually exaggerate the problem.

3.2 Corpus

To cover a wide range of news, we crawled a total of 40 000 articles in six different domains (domestic, world, society, entertainment, sports, and technology) from four major Chinese news sites (Sina, NetEase, Tencent, and Toutiao).

Because of the exact definition of the problem, annotators have to read through the news headlines and bodies, which is a time-consuming and demanding job. Therefore, we randomly select 2924 pieces of news and employ 6 college students majoring in Chinese to label each news headline as ambiguous or not, and also label it as misleading or not. They have read relevant instructions before annotating and each piece of news is labeled by at least 3 people. To ensure the consistency of misleading headline annotation, we abandoned 316 controversial examples (at least one annotation differs from the others). The final labeled data set contains: 645 ambiguous and 2279 non-ambiguous; 843 misleading and 1765 non-misleading.

Note that 24 000 pieces of unlabeled news are used for cotraining, and a big data analysis is conducted upon the full data set. Some of the articles are not used for co-training because we continuously crawl them after the experiments.

4 Identifying Ambiguous Headlines

According to the definition of ambiguous news headlines, they usually deliberately omit some main elements of sentences to spur curiosity, which can be visually seen without reading news bodies. Thus, we firstly follow the previous practice to extract features from headlines. But the downside is that those features are mainly word-based, losing sight of sentence structures and sequential information. Thus, we secondly mine class sequential rules (CSR) and then derive CSR features. Finally, both the basic features and CSR features are utilized to train an SVM classifier [Joachims, 2002]. SVMmachine learning method is selected in this task for its outperformance over other methods. We use the SVM toolkit in scikit-learn 1 .

4.1 Basic Features

Table 1 lists the basic features. Those features were proved to be effective in English clickbait detection tasks [Chakraborty et al., 2016]. Clickbait words are phrases or words commonly used in catchy headlines, such as "You Won't Believe" and

1 http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html

"Will Blow Your Mind". We translate the English clickbait word list released by Downworthy 2 , and manually extend the vocabulary to adapt the characteristics of Chinese. We make use of "Chinese/English Vocabulary for Sentiment Analysis" released by Hownet 3 when counting sentiment words. This vocabulary contains six files, including words expressing sentiment degree, subjective opinion, positive evaluation, negative evaluation, positive emotion, and negative emotion. We match internet slang using the lexicons released by SogouInput 4 .

4.2 Class Sequential Rules Mining

We utilize a sequential pattern mining method described in [Liu, 2007] to find language patterns of ambiguous headlines and non-ambiguous ones, and then derive features based on the sequential rules.

Let I = {i1, i2, ..., in} be a set of items. A sequence s is an ordered list of items. s is denoted by 〈a1, ..., ai, ..., ar〉, where ai is an item in I. A sequence s1 = 〈a1, a2, ..., an〉 is called a subsequence of s2 = 〈b1, b2, ..., bm〉 if there exist integers 1 ≤ j1 ≤ j2 ≤ ... ≤ jn-1 ≤ jn such that a1 = bj1 , a2 = bj2 , ..., a1 = bjn .

The sequence database D is a set of pairs {(s1, y1), (s2, y2), ..., (sn, yn)}, where si is a sequence and yi ∈ Y is a class label. In the context of ambiguous headlines detecting, Y = {ambiguous, non-ambiguous}. A class sequential rule (CSR) is of the following form:

An instance (si, yi) covers the CSR if X is a subsequence of si. An instance (si, yi) satisfies the CSR if X is a subsequence of si and yi = y. The support (sup) of a CSR is the fraction of instances in D that satisfy the rule. The confidence (conf) of the rule is the proportion of instances that cover the rule which also satisfy the rule.

2 http://downworthy.snipe.net/

3 http://www.keenage.com/

4 http://pinyin.sogou.com/dict/

5 http://ltp.readthedocs.io/

To explain the above definitions, we consider the following example shown in Table 2. Setting the minimum support to be 0.2 and the minimum confidence to be 0.4, one of the CSRs is 〈1, 4, 7〉 → c1. The support of this rule is 0.4 since sequence 1 and 2 satisfy the rule, and the confidence is 0.67 since sequence 1, 2 and 5 cover the rule.

To build a sequence database, we transform the news headlines into sequences. The item set I contains 12 labels corresponding to word types mentioned in Table 1 such as WH-words and forward-reference words. I also contains 2 labels of temporal adverbs expressing "past" and "present", as well as 9 labels corresponding to different Chinese conjunctions. Temporal adverbs are used here because some headlines cause a sensation by presenting a contrast between the past and present. In Chinese, conjunctions may increase the attractiveness of a sentence by indicating antithesis, hypothesis, etc. For each word in a headline, it may be transformed into a type label. As a result, a headline, which is originally a sequence of words, is encoded into a sequence of type labels. If there is no match between a word with any items in the vocabularies, we just skip it and move on.

Take the sentence in Section 3.1 as an example. We search for each word in the lexicons, and the word "¤" (she), "Q †" (once), "�" (but) and "y3" (now) are successfully matched. Consequently, the word sequence is transformed into a label sequence as follows:

With selected minimum support (minsup) and minimum confidence (minconf), we utilize the CSR mining algorithm described in [Liu, 2007] to mine CSRs from the training database. Then the sequential pattern X in each CSR is treated as a feature, which is set to 1 when being contained by a headline.

After extracting features based on CSRs, we conduct experiments with CSR features to demonstrate the efficacy. We also train classifiers with all features mentioned above and see a performance improvement comparing with basic methods.

5 IdentifyingMisleading Headlines

A headline is considered misleading if and only if it differs from the news body. Hence, features extracted separately from headlines and bodies cannot provide enough evidence for this task. We select four groups of features which evaluate the consistency between headlines and bodies. Additionally, we utilize a bootstrapping method, co-training, to take advantage of the large set of unlabeled data.

5.1 Features

Body-independent Features

This set of features is extracted from headlines only, similar to those included in Table 1. Note that ForwardRef is abandoned here since this feature merely reflects ambiguity.

Body-dependent Features

Some of the following features are derived from news bodies, while others reflect the consistency between headlines and bodies.

1. Informality: We compute the frequencies of two informality indicators, namely internet slang and bait words. Additionally, the length of news bodies is also an input feature.
2. Sentiment: Sentiment feature consists of the frequencies of positive evaluations, negative evaluations, positive emotions, negative emotions and subjective words.
3. InformalGap: We calculate the absolute value of informality difference between headlines and bodies of each piece of news.
4. SentiGap: We calculate the absolute value of sentiment difference between headlines and bodies of each piece of news. Specifically, this feature set contains the differences in five frequencies mentioned in Sentiment.
5. Similarity: Misleading headlines often contains words that differ from those in news bodies. Thus, we firstly count the number of named entities that occur in a headline but absent in the corresponding news body. Secondly, for each word (except entities) hi in a headline, through calculating the cosine distance of word embeddings, we find its most similar word b in body, and record this largest cosine similarity as si. Then the following values are used as features:

where m is the number of words except named entities in a headline. Thirdly, tf-idf is used to compute the overall similarity between a headline and a news summary generated by PKUSUMSUM [Zhang et al., 2016].

1. Recognizing Textual Entailment(RTE): Textual entailment is defined as follows: a text T entails another text H if the meaning of H can be inferred from the meaning of T with common background knowledge. Previous studies proposed RTE methods based on the similarity of dependency trees [Kouylekov and Magnini, 2005; Wang and Neumann, 2007]. To simplify the problem, we parse the headline and sentences in body into dependency trees, and calculate RTE-score by matching pairs of governing and dependent words which are sentence skeletons. Specifically, we search sentences in the body for dependency pairs occurred in the headline. While matching, synonym, hypernym, hyponym, and antonym are taken into account and assigned different

weights. RTE-score equals the weighted sum of each match.

5.2 The Co-Training Approach

Identifying misleading headlines is a relatively complicated problem due to the variety of baiting strategies, while we only have a small set of labeled data. In order to build a robust classifier that is capable of handling different instances, we try to make full use of the larger unlabeled data set via a semisupervised method.

Co-training [Blum and Mitchell, 1998] is a typical semisupervised method, which demands features based on two independent views, but the independence assumption can be relaxed. Starting with the limited data set, co-training can increase the amount of labeled data by annotating the unlabeled data with two sub-classifiers. In recent years, co-training has been successfully applied to co-reference resolution [Ng and Cardie, 2003], sentiment classification [Wan, 2009], review spam identification [Li et al., 2011], etc.

Since we extract body-independent and body-dependent features from each piece of news, our task exactly satisfies the essential requirement of co-training. The algorithm framework is shown in Algorithm 1. In the experiments, we balance the parameter values of p and n at each iteration to maintain the class distribution in the labeled data. Through adding confidently predicted instances, two sub-classifiers gain useful information with the help of each other. Note that the examples with conflicting labels are excluded from Nh ∪ Nb.

Algorithm 1 Co-Training Algorithm

Given: Body-independent (headline-dependent) features Fh; body-dependent features Fb; a set of labeled news L; a set of unlabeled news U.

Loop for I iterations:

• 1: Learn the first classifier Ch from L based on Fh;
• 2: Use Ch to label news from U based on Fh;
• 3: Choose p positive and n negative most confidently predicted news Nh from U;
• 4: Learn the second classifier Cb from L based on Fb;
• 5: Use Cb to label news from U based on Fb;
• 6: Choose p positive and n negative most confidently predicted news Nb from U;
• 7: Remove Nh ∪Nb fromU;
• 8: Add Nh ∪ Nb with the corresponding labels to L.

During co-training, both the sub-classifiers will provide prediction scores for each instance. The prediction scores are normalized into [0, 1]. Finally, the average of the normalized values is used as the overall prediction score of each instance.

6 Evaluation Results

In this section, we set experiments to evaluate the performance of the two identification tasks. The corpus has been described in Section 3.2. The evaluation metrics are precision, recall, and F-score.

6.1 Ambiguous Headlines Identification

We randomly split the labeled data set into a training set and a test set in the proportion of 3:1. In the experiments, three versions of features are used for comparison:

Unigram Features: It is a baseline using the SVM classifier and the rbf kernel to identify ambiguous headlines, with unigram features provided. Unigrams here refer to Chinese words but not characters.

Basic Features: It uses the SVM classifier and the rbf kernel to identify ambiguous headlines, with only basic features provided.

CSR Features: It uses the SVM classifier and the rbf kernel with only CSR Features provided. When mining class sequential rules, we conduct experiments with different minimum support and minimum confidence. In Table 3, we list the result of minsup 0.02 and minconf 0.8 since this set of parameters fit well on the training set.

All Features: It uses the SVM classifier with both basic features and CSR features provided. The minsup and minconf for mining are the same as above.

Table 3 shows the comparison results. The method with unigram features does not perform well, because this task is different from traditional text classification tasks. The method with CSR features is proved to be effective and outperforms over basic features in precision, owing to its strictness with sentence structures. Our method with all features gets an increase in precision, recall, and F-score comparing with the method with basic features, demonstrating the contribution of CSR features. In addition, we conduct sign-test upon the predict results of basic features and all features. Sign test is a statistical method to test for consistent differences between pairs of observations. The p-value of sign test is 0.0022 < 0.05, demonstrating the significant efficacy of our method.

6.2 Misleading Headlines Identification

In this task, the labeled data set is randomly split into a training set and a test set in the proportion of 3:1. We firstly conduct supervised learning to compare the efficacy of different features on the labeled data set. For co-training, the same test set is used for evaluation, and the training set as well as the unlabeled data is used for learning.

Feature Validation

To validate the effectiveness of each feature, we conduct supervised learning with all features and then exclude each group of body-independent features to compare the performance. In addition, we list the results of the method with all body-dependent features and the method with all bodyindependent features. We select the SVM classifier since it performs best comparing with other methods. Rbf kernel is used in the experiments. The baseline uses unigram features and the SVM classifier. The theoretical result of random classification is also provided for comparison, which is 0.323 because of the imbalance of the data set.

Table 4 shows the result of feature validation. With all features, the classifier achieves the F-score 0.702, which significantly exceed both sub-classifiers. According to the results listed, all the six groups of body-dependent features are resultful. Among all features, InformalGap and SentiGap are relatively important. Thus, it can be seen that the consistency between news headlines and news bodies plays an important role in misleading headline detection.

Co-Training Results

In the previous section, the efficacy of two views of features is proved. In this section, we utilize the large number of unlabeled data by leveraging the co-training method. We compare the results of co-training and the supervised method to demonstrate the suitability of co-training in our task. The sub-classifiers use the same SVM-based machine learning method mentioned in the previous section. We use different sets of parameters and the experiment results listed in Table 5 is applied to the parameter set of p = 10, n = 20 and iteration number = 50. Our proposed co-training method outperforms the supervised method with all features and gains an increase of 0.022 in F-score. We conduct sign test again and our method passes the significance test with a p - value of 0.0183 < 0.05.

Parameter sensitivity: For different parameter sets, the performance of our co-training method is slightly different. We change p and n in experiments, and also observe the results in different iteration numbers. Note that n always equals 2p in order to keep the proportion of positive instances and negative instances. The results are shown in Figure 1. We can see that at the beginning, F-score is on the rise along with the increasing iteration number. After 45 iterations, the results achieve relative stability with slight fluctuation. Among different growth sizes, p = 10 and n = 20 perform best on our data set. When p = 5 or 2, F-score increases slowly during iteration. For a larger value of p = 20, the performance improves obviously in 20 iterations but no longer increase then, because a large growth size is less robust to noises.

However, only 39.9% of the news in Toutiao has accurate headlines, while misleading headlines account for more than half and ambiguous ones account for 38.6%. We can readily explain the difference. The first three websites are traditional online news organizations, while Toutiao is a kind of new media which combines various information sources without strict filtering. Lacking rigorous management, new media like Toutiao are prone to provide inaccurate information.

7 Data Analysis

We used 40 000 pieces of news crawled evenly from four major Chinese news sites (Sina, NetEase, Tencent, and Toutiao). The data covers six domains including domestic, world, society, entertainment, sports, and technology. After training the two classifiers, we identify news with ambiguous headlines and misleading ones. Then we analyze the statistics, getting some interesting and useful results.

Figure 2 shows the proportion of three kinds of headlines in different news websites. Note that each three columns do not add up to 1 because there can be an intersection of the first two categories. We can see a striking contrast. Among news crawled from Sina, NetEase, and Tencent, accurate headlines make up around 80 percent, with misleading and ambiguous headlines accounting for less than 20 percent.

Figure 3: Comparison of different news domains

The comparison results among news in different domains are shown in Figure 3. Domestic news and world news, which usually cover political and military events, have a high rate of accuracy. The highest incidence of inaccurate headlines in entertainment news may result from its lack of seriousness. Society news telling unusual stories in social life also tends to have tricky headlines. In addition, this figure reflects that almost half the inaccurate headlines are both misleading and ambiguous.

8 Conclusion

In this paper, we study the problem of inaccurate headline detection. We crawl news online and build a Chinese data set. We divide the problem into two tasks. For ambiguous headline detection, we proposed a method based on class sequential rules and demonstrate the efficacy. For misleading headline detection, features evaluating the consistency between headlines and bodies are utilized. To exploit the larger unlabeled data set, we apply the co-training method in the latter task. The experimental results demonstrate the robustness of our identifiers.

Utilizing the final classifiers, we identify ambiguous and misleading headlines in the full data set. The results of data analysis reflect the need for more stringent regulations in journalism, especially for new media and entertainment news.

Acknowledgments

This work was supported by 863 Program of China (2015AA015403), NSFC (61331011), and Key Laboratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology). We thank the anonymous reviewers for helpful comments. Xiaojun Wan is the corresponding author.