What Users Care About: A Framework for Social Content Alignment

1 Introduction

With the rapid development of social media, more and more people express their opinions and comments on daily news, products, movies and various Web documents through different social media platforms. For example, in many news portals, users can leave their comments on various aspects/topics of a news article. On Twitter, users post the URL of a Web document together with their personal views, followed by streams of posts generated by friends and other people. Figure 1 illustrates a news article about Boehner 1 in Yahoo! News, as well as corresponding comments dedicated to a specific topic in the article, such as vote, relief bill, tenure of office, national debt, etc.

A popular document often attracts a bulk of social discussions. Our preliminary statistics show that the average numbers of comments for top news in Yahoo! 2 and Sina 3 are 5684.6 and 9205.4 respectively (on Nov, 2012). What kinds of topics did the users discuss? How to align the user generated comments to the corresponding topics contained in the Web document? The objective of this research is to design a principled framework to automatically align the social content (user generated comments) to the topics contained in the Web documents, briefly referred to as topic level social content alignment. This can facilitate users to quickly focus on those topics that they are interested in and explore the user generated content at topic level.

1 http://news.yahoo.com/blogs/ticket/john-boehner-electedspeaker-house-190301689-politics.html

We propose a two-phase framework to address the social content alignment issue. In the first phase, we extract topics from both Web documents and social content. One straightforward idea is making use of probabilistic topic model on documents and social content independently, and then aligning the extracted topics on both sides, but it is difficult to guarantee the consistency between the two different topic sets. An alternative method is that we can model the Web documents and social content together, but there is a risk that topic bias may occur because of the extremely unbalanced volumes between them. Moreover, both methods ignore the dependency between social content and the Web document. We observe that the user generated comments are usually based on the content of Web document. In this paper, we propose a novel document-comment topic model which can effectively exploit the dependency between Web document and social content by using two correlated generative processes. Particularly, we first employ standard LDA to model sentences in Web documents, and for comments modeling, we use a Bernoulli distribution to determine whether it is generated from a document-related topic or a comment-specific topic.

The second phase aligns the topics detected in the first phase with the comments, using our proposed positive unlabeled learning technique. Intuitively, two kinds of machine learning methods [Sil et al., 2011b], namely unsupervised methods and supervised methods, can be employed for social content alignment. The unsupervised methods define and extract features (mostly uses terms) from both Web documents and social content, and assign weights to them (TF-IDF, ESA, etc.). Then based on the weights and features, we can use the method to calculate similarities between a document and a comment. Finally, for each topic extracted in the first phase, we select comments with the highest similarities. However, social content is usually quite short, informal, not well structured and it could be written using quite different vocabulary compared with Web documents, unsupervised methods thus cannot effectively match the topic and comments. Supervised learning methods, on the other hand, need users to provide training data, i.e. which sentences in Web documents or comments belong to which topics (treat each topic as a class). Then a classification model can be built to classify all the comments into one of the topic classes. However, in many real-life applications, it is difficult to have sufficient training examples to build accurate classifiers. As such, supervised learning methods will not work as well. In this paper, we propose a novel positive unlabeled learning method (PU learning) which can effectively leverage rich unlabeled data, i.e. those unlabeled comments for building an accurate classifier. Particularly, in our PU learning setting, given a topic, those sentences associated with the topic will be treated as positive data P, all the other sentences and comments are unlabeled data U. Our proposed algorithm is able to partition U into potential positive and negative data and build more accurate classification model. Note our PU learning model is built automatically and thus effectively facilitates the social content alignment process.

2 http://news.yahoo.com/most-popular/

3 http://news.sina.com.cn/hotnews/

The main contributions in this paper can be summarized as follows:

• We formally define the social content alignment problem and present a novel two-phase framework to address it.
• We propose an innovative topic model which can exploit Web document, social content and their dependency for accurately extracting latent topics and unifying both of them in the topic space.
• We propose a novel PU learning algorithm to build classifiers by exploiting the positive topic data identified by our topic model and a large amount of unlabeled data,
• which addresses the limited labeled topic data issue for effective classification.
• Experimental results show that our proposed framework can effectively address the challenging issues in the social content alignment problem, significantly outperforming the existing state-of-the-art methods.

The rest of the paper is organized as follows. We define the social content alignment problem and present our proposed algorithm in Section 2. Experimental results from extensive experiments are reported in Section 3. Finally, Section 4 concludes the paper.

2 Problem and Techniques

In this section, we first formalize the social content alignment problem, and then demonstrate our proposed techniques.

2.1 ProblemDefinition

While social content can be expressed through various ways, in this paper we focus on the most commonly used textual information, namely those social content dedicated to news, blogs and other Web documents posted through social media applications.

Preliminary Let d denote a Web document, consisting of a set of sentences S = {s1, s2, . . . , sM}, and associated with a social content set C = {c1, c2, . . . , cN}. Both the document and its associated social content cover several topics T = {t1, t2, . . . , tK}. Each s/c corresponds to a specific topic t, and contains a vector ws/c of Ns/c words, where each word ws/ci is chosen from a vocabulary of size V .

Definition 1: Social Content Alignment (SCA) Given a Web document with the sentence set S and the corresponding social content set C, the goal of social content alignment is to generate a set of matching pairs <social content, topic>, namely {(ci, tj)|where ci ∈ C, tj ∈ T ∪ ∅} which means social content ci discusses the specific topic tj.

As shown in Figure 1, the left is a news article entitled John Boehner re-elected as speaker of the House in Yahoo!

News, which discusses topics such as vote, relief bill, tenure of office and national debt. The right part lists a few examples of comments posted by users, where the arrows link them to the representative sentences of the topics in the document.

There are a number of work related to current research. For example, [Lu and Zhai, 2008] studied how to automatically integrate opinions by a well-written expert with lots of opinions scattering in various sources such as blogs, spaces and forums. A semi-supervised model was proposed to deal with it. [Yang et al., 2011] studied to leverage social information for Web document summarization and proposed a dual wing factor graph for summarizing news incorporated Twitter. The objectives of the two related work mentioned above are different from ours since they target at data integration and document summarization by exploiting additional social content. Recently [Sil et al., 2011a; 2011b] proposed to allow users to read news along with relevant comments and presented a supervised learning method for linking comments to news segments. As mentioned before, supervised learning methods typically need time-consuming effort to hand-label the training set. In addition, the training set labeled for one particular document and corresponding social content can only be used once.

In this paper, we propose to design a generic positive unlabeled learning method to automatically align the document with social content. Furthermore, our method is topic based which is more flexible than the segment based ones because all the sentences within a segment must be together in the document, while the sentences within a topic could distribute across the whole document. More specifically, we present a two-phase framework, a multi-source probabilistic topic model and a PU learning method, some of which are novel with respect to this task. The rest of this section demonstrates the details.

2.2 Document-Comment Topic Model

To tackle the sparse and non-uniform feature problem, one idea is enriching features with the aid of large scale data collections like Wikipedia [Phan et al., 2008], but it may introduce noise and often fails on newly-generated content. Considering such a scenario: when a user reads Web documents, he may have opinions on some inside topics, and he subsequently posts social content on the interested internal topics. Although he may use his own words, with the number of comments on this topic grows, the generality of them will be reflected. Therefore, extracting topic features is another direction to solve the problem.

For modeling documents from different sources, [Blei and Jordan, 2003] modeled pictures and their annotations, [Blei and McAuliffe, 2007] developed supervised topic models, where each document is paired with a response, to infer latent topics predictive of the response. Based on the previous works, [Wang et al., 2009] took the category information into consideration for picture modeling and classification. [Tang et al., 2009] proposed qLDA to extract an informative summary from a document collection for a given query, and [Tang et al., 2012] developed CTL to learn and differentiate collaboration topics from other topics. [Hong et al., 2011] extended standard topic models by allowing each text stream to have both local topics and shared topics.

Through observation (see Section 3.1), we find the social content heavily leverages the Web documents, and the latent topics build a bridge between them. The Web documents behave like a kind of background knowledge to guide the generation of social content. But all the existing works pay little attention on the dependencies, so here we develop a document comment topic model to model Web documents and social content simultaneously.

The basic idea is to use two correlated generative processes to model Web documents and comments. The first process is to model sentences in Web documents using standard LDA, and the second process is to model comments. For each word in comments, we use a Bernoulli distribution to determine whether it is generated from a document-related topic or a comment-specific topic. Figure 2 shows the graphical structure of the DCT model (For simplicity, we omit the modeling part for Web documents and focus on the modeling of comments).

Let us briefly introduce notations. θs, θc are topic models from documents and comments; x is a binary variable indicating whether the current word inherits the topic from document-related (x = 1) or by comment-specific topic (x = 0); α, β are the Dirichlet hyper parameters; λ is a parameter for sampling the binary variable x; γc, γs are beta parameters to generate λ.

Formally, the generative process is described in Algorithm 1: we extract topics from document according to the distribution p(θs|α), while for word wci in comment c, a coin x is tossed according to p(x|c) ∼ beta(γc, γs) to decide whether wci is sampled from a document-related topic or commentspecific topic.

To estimate the parameters for this model, we take the widely-used Gibbs sampling technique. First we sample the coin x according to the posterior probability:

where ncx0 is the number of times that x = 0 has been sampled in c; nzci is the number of times that topic z has been sampled from c and "¬" indicates excluding that instance

Algorithm 1: Generative process for DCT model.

Input: the priors α, β, γc, γs; S and C Output: estimated parameters θs, θc, λ and φ Initialize a standard LDA model over S;

foreach comment c ∈ C do

foreach word wci ∈ c do

Toss a coin xci according to bernoulli(xci) ∼ beta(γs, γc), where beta(.) is a beta distribution, and γc and γs are two parameters;

if xci = 0 then

Draw a topic zci ∼ multi(θc) from a comment-specific topic mixture;

else

Draw a topic zci ∼ multi(θs) from a document-related topic mixture;

end

Draw a word wci ∼ multi(φzci ) from zci-specific word distribution;

end

end

from counting. p(xi = 1|·) can be analogously defined as Equation 1.

Then the posterior probability of topic z is defined as:

where n¬ci zciwci , mzciwci denote the number of times that word wci has been generated by topic zci in comment and Web document respectively, and n¬ci czci and mczci are the number of times that topic zci have been sampled from commentspecific or document-related topic distribution.

During the parameter estimation, the algorithm keeps track of a (|S|+|C|)×K (sentence+comment by topic), a |C|×2 (comment by coin), a 2 × |K| (coin by topic) count matrixes, and a K × W (topic by word) count matrix. Given these matrixes, we can estimate the probabilities θs, θc, λ and φ.

After that, each sentence is associated with a probability distribution over topics, and we can obtain representative sentences for each topic by selecting those with highest probability over the given topic.

2.3 Learning fromPositive and Unlabeled Data

We are now ready to present our work on how to design a PU learning method to align the social content with the topics identified in the Section 2.2. Given a particular topic tj, all the sentences in S belonging to tj, will be treated as the positive set P . All the other sentences in S-P will be treated as unlabeled set U1, and all the social content will be treated as unlabeled set U2, as shown in Equation 3. We then build a PU learning model to classify all the social content, into either positive or negative class. Particularly, those social content that are classified into positive class will be aligned to the topic tj .

Learning from positive and unlabeled data (PU learning) was first proposed in [Liu et al., 2002]. The core idea can be characterized as the following two steps, namely: (1) identify a set of reliable negative examples from the unlabeled set U; and then (2) build a classifier using EM or SVM iteratively. The difference among the existing algorithms, [Liu et al., 2003], [Lee and Liu, 2003], [Li and Liu, 2003], [Li et al., 2007], [Li et al., 2009], [Li et al., 2010] and [Nguyen et al., 2011] lies in the specific algorithms used in these two steps.

The first step of existing PU learning methods, typically uses U as initial negative set to build a classifier with P to exact reliable negatives. However, since U contains hidden (false) positive data, the extracted negatives are not reliable. Different from the existing work, in this paper, we present a three-step PU learning method. We aim to first partition the U into potential positive set PP and potential negative set PN. In particular, we construct a hyper-sphere by calculating the centroid ⃗o of all the positive examples, and the average distance r (viewed as radius) between ⃗o and all the positive examples. Those examples which fall into the hyper-sphere will be treated as potential positive examples, while the others outside hyper-sphere are treated as potential negative examples, as shown in Equation 4. Note that the set PN will be much pure to serve as a negative set than the original unlabeled set U since we have taken out the potential positives PP in U. At the same time, the positive set P is relatively small, PP can be used to enhance the limited positive set P .

The second step of our method is to build a Rocchio classification model where we use P ∪ PP as positive training examples, PN as negative training examples. We construct positive and negative prototype vectors ⃗ p and ⃗n respectively for the positive and negative classes:

where μ = 16, ν = 4 as recommended in [Buckley et al., 1994].

For each unlabeled example u ∈ U1 ∪ U2, we calculate its cosine similarities with ⃗ p and ⃗n: if sim(u, ⃗ p) > sim(u, ⃗n), then u will be added to likely positive set LP; otherwise, it is added to likely negative set LN. We also compute the confidence score l as:

Note that we give the confidence score 1 to all the examples in the original positive set P .

Finally, we build the final classifier using Weighted Support Vector Machine (WSVM), whose optimizing goal is:

where CP , CLP and CLN represent the penalty factors of misclassification for three types of training examples, namely, original positive set P, likely positive set LP and likely negative set LN. We use the confidence score directly for each example in P, LP and LN as CP , CLP and CLN since we are more confident with positive set P than the likely positive set LP and likely negative set LN. Correspondingly, we give a larger penalty if examples from P are classified as negative class than if examples from LP are classified as negative or examples from LN are classified as positive.

After that, we use the our weighted SVMmodel to classify all the social content in C and those social content are classified as positive class will be aligned to the given topic tj. Apparently, we can accomplish the alignment task by repeating it on all extracted topics.

3 Empirical Evaluation

In this section, we evaluate the proposed alignment method using our manually created news data. We will first introduce the data set in Section 3.1 and then present the experimental results in Section 3.2.

3.1 Data Preparation

To the best of our knowledge, there is no existing benchmark dataset which can be directly used for our experiments. As such, we crawled news articles and corresponding social content from two popular news websites, namely Sina (China) and Yahoo!. We have chosen top 10 Chinese news from Sina and 12 English news fromYahoo! fromDec. 1st to Dec. 11th 2012 which have the most social content generated. Then we invited 7 annotators to build gold-standard link sets between the sentences in the news and social content/comments where we only include those links that majority of people agree (5 out of 7).

For both datasets, we perform preprocessing, such as remove stop words, and filter low-frequency words (frequency<= 3). Some statistics of the datasets after preprocessing are summarized in Table 1. Then we investigate the annotation results. Figure 3 shows the distribution of comments(or sentences) with respect to the number of sentences (or comments) they are linked to. We can observe that:

• 87% comments are linked to one or multiple news sentences, while the remaining 13% of comments are irrelevant to any sentences, indicating that it is reasonable to make use of comments to enhance topic detection in our document comment topic model.
• 22% news sentences can attract more than 10 comments, signifying it is important to automatically mine the relevant comments from large amount of social content so that we can quickly read through them to effectively understand what other people care about. We also notice there are 27% of sentences without any related comments. We found these sentences simply provide some background of the news and thus lead to "no comments" scenario.

3.2 Experiment

Baseline Methods. We use T-PU to denote the alignment technique that employs DCT model for topic extraction and weighted-SVM for the final classification. To test the effectiveness of T-PU, we compare it with four baseline methods:

VSM: a simplest similarity based method by using TF-IDF document representation method and cosine similarity.

BSVM: a classification based method. We build binary classifiers on sentences using the labeled data with libsvm 4 and test with five-folder cross-validation. The numbers of positive examples range from 0 to 78 (3.93 in average) while negative examples can be more than 100.

DCT: a straightforward method which classifies the comments directly by using the distribution obtained by DCT model.

T-SVM: a supervised method which needs users provide manually-labeled training examples. The only difference from BSVM is that classifiers here are built on the topics extracted by DCT model instead of individual sentences.

4 http://www.csie.ntu.edu.tw/˜cjlin/libsvm/

Metrics. The performance of text classification is typically evaluated based on precision and recall, but in our scenario precision is more important than recall because users prefer to read few accurate comments rather than many noisy comments.

For a sentence-comment set pair (S,C), let ri ⊆ S be the set of aligned sentences (labeled by different annotators) for comment ci. If |ri| > 1, then ci has multiple related sentences while |ri| = 0 indicates ci has no related sentences.

Let ti ⊆ S denote the result topic for ci found by our method, and ti is associated with several sentences ˜ ri. We consider it to be correct if ri ∩ ˜ ri ̸= ∅.

So the precision can be defined as:

To make the comparison fair, we use the DCT results for those methods without topic information.

Result and Analysis. Table 2 gives the overall performance of our method on the two datasets, and the comparison with other four techniques in terms of Precision is shown in Figure 4. We can see that:

• Our proposed method T-PU outperforms the baseline methods VSM and DCT on both datasets. This is because our T-PU method utilizes topic level features and word-level features in two steps while the other two use only one of them.
• Compared with BSVM, our method has significant improvement because it is difficult to build an accurate classifier with very few positive examples and many negative examples (contains noise inside). And our method can split the latter into potential positive and negatives examples, which can enhance the limited positive set as well as purify the negative set.
• T-PU is very close (-2.1% in Sina and -2.9% in Yahoo!) to T-SVM which performs best among all the methods. Note that T-SVM is a supervised method and its performance depends much on the quality and quantity of labeled data while our T-PU can achieve comparable results without using any labeled examples, so our method is more appropriate for the social content alignment task.

Furthermore, we investigate those comments that our method failed to find an appropriate topic for, and discover that the main reason is the comment chain and the topic drift caused by it. A drift example is shown in Figure 5. It is a report about rocket launch in North Korea. We can see that there is no comment on Topic 0 (background topic), and Topic 2 should have been talking about the cost but through the chain, it turns into food aid. Similar problem occurs in other news, which contributes most of the failed comments.

Data Sets

Figure 5: Topic distribution in news and comments

Hyperparameters. Following [Rosen-Zvi et al., 2004]'s guide, we take a fixed value for the hyper parameters α and β where α = 50/K, β = 0.1. As for the Beta parameters γc, γs, we tried two methods, calculating the common words ratio and referring the comment-sentence distribution in Section 3.1, and the results show DCT model is not very sensitive to the prior. And the number of topics is set to be 5 for all news and it works well in most cases.

4 Conclusions

In this paper, we address the most fundamental problem in social content analysis, that is, finding the corresponding article sentences for each piece of social content. We propose a novel two-phase framework to accomplish this task automatically. Specifically, we present a document comment model to extract topics from both the document and comments and introduce a positive and unlabeled learning method to build an accurate classifier which does not need users provide labeled examples but can achieve comparable good results compared with the supervised learning methods.

Social content alignment is a challenging yet interesting problem. Thus there are many potential future directions of this work. For example, we can study the alignment over similar Web documents (i.e. news reports on same event) and explore the comment patterns they may share. We can also investigate whether users' relationships can help the alignment, and study the topic drift in the comment stream.

Acknowledgments. Thanks anonymous reviewers for their valuable suggestions that help us improve the quality of the paper. The work is supported by NSFC (No. 61035004, 61222212, 61073073), NSFC-ANR(No. 61261130588), 863 High Technology Program (2011AA01A207), FP7-288342, and THU-NUS NExT Co-Lab.