Context Sensitive Topic Models for Author Influence in Document Networks

1 Introduction

The popularity of Web 2.0 applications has resulted in large amounts of online text data, e.g. weblogs, digital libraries of scientific literature, etc. These data require effective and efficient methods for their organization, indexing, and summarization, to facilitate delivery of content that is tailored to the interests of specific individuals or groups. Topic models such as Latent Dirichlet Allocation (LDA) [Blei et al., 2003] and Probabilistic Latent Semantic Analysis (PLSA) [Hofmann, 1999] are generative models of text documents, which successfully uncover hidden structures, i.e., topics, in the data. They model the co-occurrence patterns present in text and identify a probabilistic membership of words and documents into a much lower dimensional space compared to the original term space. Since their introduction, many extensions have been proposed.

One such line of research aimed at modeling the interests of authors to answer important queries about authors, e.g., who produced similar work [Rosen-Zvi et al., 2004], who belongs to the same research community [Liu et al., 2009; Wang et al., 2005] and who are the experts in a domain [Tu et al., 2010]. However, another fundamental question about the attribution of topics to authors still remains not answered: who influences the generation of new content in a particular topic of interest? In this work, we propose generative models that take the linkage between authors of citing and cited documents into consideration and explore various qualitative and quantitative aspects of this question.

Another line of research aimed at modeling topics for content and citations together to quantify the influence of citations over the newly generated content [Dietz et al., 2007; Nallapati et al., 2008; Chang and Blei, 2009; Kataria et al., 2010]. However, these statistical methods for parameterizing the influence of a document cannot easily quantify the influence of authors because one document often has multiple authors.

In this work, we exploit the complementary strengths of the above lines of research to answer queries related to authors' influence on topics. Specifically, we present two different generative models for inter-linked documents, namely the author link topic (ALT) and the author cite topic (ACT) models, which simultaneously model the content of documents, and the interests as well as the influence of authors in certain topics. As in the author topic model (ATM) [Rosen-Zvi et al., 2004], ALT models a document as a mixture of topics, with the weights of the mixture being determined by the authors of the document. In order to capture the influence of cited authors, ALT extends ATM to let the set of cited authors in a document be represented as a mixture of topics and again the weights of the topics are determined by the authors of the document.

Moreover, we hypothesize that the context in which a cited document appears in a citing document indicates how the authors of the cited document have influenced the contributions by the citing authors. ACT extends ALT to explicitly incorporate the citation context, which could provide additional information about the cited authors. Kataria et al. [2010] have previously used the citation context while jointly modeling documents and citations (without authors) and have shown that a fixed-length window around a citation mention can provide improvements over context-oblivious approaches. Unlike Kataria et al. [2010], we model the authors of the document along with the content and argue that a fixed-length window around a citation mention can provide either limited or erroneous information in cases where the context spans are larger or smaller, respectively, than the length of the window. Hence, we dynamically select an adaptive-length window around a citation that is statistically more likely to explain the cited document than a fixed-length window.

In summary, our contributions are as follows:

• We propose generative models for author-author linkage from linked documents conditioned on topics of interest to authors. Our models are able to distinguish between authors' interests and authors' influence on the topics.
• We utilize the context information present in the citing document explicitly while modeling the cited authors and obtain significant benefits on evaluation metrics on real world data sets. Moreover, we dynamically select the length of context surrounding the citation mention and circumvent the erroneous context inclusion by a fixed window approach.

2 Related work

One of the earliest attempts at modeling the interests of authors is the author topic model (ATM) [Rosen-Zvi et al., 2004], where the authors and the content are simultaneously modeled with coupled hyper-parameters for the interests of authors and the themes present in text (shown in Fig. 1(a)). The (latent) topics represent the shared dimensions among the interest of authors and the themes. Bhattacharya and Getoor [2006] extended ATM to disambiguate incomplete or unresolved references to authors. Another stream of author centric modeling deals with expert finding [Fang and Zhai, 2007; Balog et al., 2009; Tu et al., 2010] where an expert is defined as a person knowledgeable in the field. We define an expert/interested author as someone who has produced several contributions in a particular field whereas an influential author as someone who has certain key contributions in that field and gets cited more often. Therefore, given a field, an influential author is not necessarily an expert in that field, however, her key contributions have led several interested authors to contribute to that field. However our main goal is to model the influence of authors along with the interest of authors.

Linking to external content or entities is an important ingredient of social content such as citation graph of academic documents, asynchronous communications such as weblogs, e-mails, etc. The mixed membership model [Erosheva et al., 2004], also referred as linked-LDA [Nallapati et al., 2008], extended LDA to model links among documents with an additional parameter that governs link generation from citing documents to cited documents. Further extensions of linked-LDA analyzed the association between words and hyperlinks [Nallapati et al., 2008; Gruber et al., 2008; Chang and Blei, 2009], influence propagation [Dietz et al., 2007], community of links detection [Liu et al., 2009], context-sensitive citation and text modeling [Kataria et al., 2010]. To model the authors in an inter-linked corpus of documents, Tu et al. [2010] proposed an extension of the author topic model to inter-linked documents. In contrast to our approach, they consider the entire citing document as the context of the citation, which, as explained in § 4.2, can easily be considered as a special case of our approach. In addition, it performs inferior to dynamically selecting the context length.

Topic models have also been extended to social networks of entities where entity-entity relationships conditioned upon topics are explored. Mccallum, et al., [2007] extended the basic ATM to cluster the entity pairs based upon topic of conversation in e-mail corpus. Their approach assumes that the sender and the recipient both decide the entire topic of conversation. This assumption is not applicable in our setting because only the author of the citing document decides the topic of the document and every cited authors may not share the interest in all the topics discussed in citing document. Newman et al. [2006] and Shiozaki et al. [2008] proposed other entityentity relationship models for named-entities in news articles where documents are modeled as mixture of topics over both entities and words.

3 Models

Before presenting our models, we introduce some useful notations. Let V , D, A, ad and Nd denote the size of the word vocabulary, the number of documents, the number of authors, a set of authors and the number of words in document d respectively. Let T denote the number of latent topics, i.e., the latent variable z (see Fig. 1) can take any value between 1 and T inclusively. Suppose there exists a T ×V topic-word distribution matrix φ that indexes a probabilistic distribution over words given a topic and a T ×A topic-author distribution matrix θ that indexes the probability with which an author shows interest in a topic. The corresponding hyper-parameters for distributions φ and θ are αφ and αθ respectively.

3.1 Author Link Topic Model

Citations among documents exhibit the biases of citing authors towards certain influential authors who have key contributions in the topic of discourse. We quantify the influence of an author given a topic by the probability, denoted by ϕcz, that the author c's work gets cited when there is a mention of the topic z in a citing document. Since the Author Topic Model (ATM) does not model the citations among the documents, it is not possible to estimate the influence of an author given a topic. In contrast, Author link topic model (ALT) generates the references to cited authors along with the words from a mixture of topics. As in ATM, a set of authors ad decides to write a document. To generate each word, an author x is chosen uniformly at random from ad, and a topic is sampled from the chosen author's specific distribution. Then the corresponding word is generated from the chosen topic. For each author in the referenced set of authors in the document d, again an author x is chosen to generate a topic, and based upon the topic, an author c is selected from the topic specific distribution over authors. ALT model captures the intuition that given a topic and a list of relevant authors to be cited, authors from ad would choose to reference those authors's work that are influential in that topic. Fig. 1(b) shows the plate diagram for the ALT model.

In the following subsections, we will use w and c to denote the words and observed cited authors in a document and z to denote the vector of topic assignments in the document. With the model hyper-parameters αθ, αφ and αϕ, the joint distribution of authors x, the topic variables z, the document w and the cited authors c can be written as below. Here, Ld stands for the number of cited authors in the document d.

3.2 Context sensitive modeling of Author-linkage : Author Cite Topic Model

ALT model does not utilize the context in which a document cites an author. Although ALT models the cited authors in the citing document, yet, because of the bag of words assumption, the topic assignment to the authors does not explicitly depend upon the topics assigned to the content in that document. To enforce this dependence, we model the cited authors along with the context of the citation. In contrast to ALT, the Author Cite Topic (ACT) model associates cited authors and the words in the citation context of the cited authors with topic assignments to the context words. This association is based upon the assumption that given a topic, the choice of words and the authors to be cited are independent (see the plate diagram in Fig 1(c). With this independence assumption, the topic sampled for words in the citation context window generates both a word and a reference to the cited author. Since we observe a set of authors for a cited document, we treat c as hidden similar to x. The parameters of the ACT model remain the same as those of the ALT model, however the complete data log-likelihood function is different due to a difference in the generation process. The log-likelihood function to optimize can be written as below. Here, Cd is the total length (number of words) of all citation contexts in the document d.

Intuitively, Eq. 2 implies that the author first picks the words from the topic and then chooses to cite an author's work or vice versa. The product p(zn|x, θx).p(wn|zn, φzn ) acts as the mixing proportions for the author "generation" probability over the entire citation context of the corresponding citation. Therefore, one can expect that this explicit relation between citation generation probability and the word generation probability will lead to a better association of words and citations, and in turn authors, with documents than without utilizing the citation context explicitly.

3.3 Dynamic Selection of Length of ContextWindow

Since the ACT model imposes independence assumption in the context window surrounding the citation mention, it becomes important to identify the context that refers to the cited article. Previous work on context utilization in topic models, either assumes a fixed window of 10 words radius surrounding the citation mention [Kataria et al., 2010] or the whole document as the context for any citation mention [Tu et al., 2010]. However, the amount of relevant context in the vicinity of the citation anchor depends upon various factors such as the strength of the influence of cited article over the citing article, the location of the citation mention in the citing article, etc. Therefore, we propose to identify a dynamic window surrounding the citation anchor with the following method.

Let ←-d represent the cited document for a given citation anchor cdi , where i ranges over all citation mentions in the citing document d. Let S(cdi ) (or simply Si) represent the bag of words in the citation context surrounding cdi . The objective function that we choose to maximize is f(←-d |Si) which is defined as:

Here, Zp is the topic vector defined as 1 Np ∑n zp,n where n ranges over all the tokens in the bag p and Np denotes the cardinality of p. σ represents the sigmoid function and ′ .′ represents the dot product between two vectors. Intuitively, f(←-d |Si) represents the topical similarity between cited document and its corresponding context.

Next we describe our dynamic context selection procedure. We allow our window to grow over sentences beginning with the sentence that has the citation mention, although the method proposed is general enough to be applicable to any building block such as words or paragraphs. We choose to begin with the sentence that contains the citation mention as the sentence carries most of the information about the cited document. We let Si denote the current context window and sj and sk are the next left and right candidates to either include in the window or to let the growth stop in either direction. We update the window as defined below and continue to grow in the direction which maximizes the objective function 3.

3.4 Inference using Gibbs Sampling

We utilize Gibbs sampling as a tool to approximate the posterior distribution for both the models. Specifically, we want to estimate θ, φ and ϕ parameters of the multinomial distributions Multi(.|θ), Multi(.|φ) and Multi(.|ϕ), respectively, in Fig. 1(b) and 1(c).

According to Eq. 2, the joint probability distribution of the latent and the observed variables can be factorized as follows:

To generalize the notations, let n(b) a denote the number of times entity b is observed with entity a. Particularly, if an observation of topic z is made with author x, then n(z) x denotes the number of times this observation is made in the whole corpus. Similarly, we define n(t) z , n(c) z where t and c stand for term and cited author respectively. Here, we derive p(c|z, αϕ). Other factors can be obtained in a similar fashion. The target posterior distribution for cited author generation, i.e., p(c|z, αϕ), can be obtained by integrating over all possible values of ϕ:

With the likely treatment to other factors, the joint distribution can be written as:

Starting with a random assignment of topics z and authors x from the list of co-authors in a document, Gibbs sampler iterates through each word and cited authors in a document, for all the documents in the corpus. For the ALT model, we need to sample topic assignment for each word variable and cited author variable. Since we have two unobserved random variables x and z for both types of assignments, our Gibbs sampler performs blocked sampling on these two random variables. We draw a sample from p(zi = k, xi = x|z-i, x-i, w) for the word variable and from p(zi = k, xi = x|z-i, x-i, c) for the cited author variable. The subscript -i indicates that we leave the ith token out from the otherwise complete assignment. After algebraic manipulation to Eq. 7, we arrive at the sampling equations as given in Eq. (i & ii) in Table 1.

Unlike the ALT model, Author Cite Topic (ACT) model has one additional unobserved random variable c that appears inside the citation context of a given citation in any document. We initialize c from the co-authors of the cited documents by uniformly selecting one author. The remaining initializations remains the same as above. We block x, z and c while sampling and for each word in the citation context, we sample from the conditional distribution, i.e. p(zi = k, xi = x, ci = c|z-i, x-i, c-i,w). The algebraic form of the conditional distribution is given in Eq.(iii) in Table 1.

4 Experiments

We describe our data set and experimental settings below and, in § 4.2 and § 4.3, we provide the details of evaluation tasks with corresponding results.

4.1 Data Sets and Experimental Settings

We use two different subsets of scientific documents for our evaluation purpose. For the first dataset (referred as CiteSeerDS1), we use publicly available 1 subset of the CiteSeer 2 digital library. The data set contains 3312 documents belonging to 6 different research fields and the vocabulary size is 3703 unique words. There is a total of 4132 links present in the data set. The dataset contains 4699 unique authors 3 where 1511 authors are cited. After standard preprocessing of removing stop words, we supplement the data set with the context information for each citation.

We employ CiteSeer-DS1 because various previous studies [Nallapati et al., 2008], [Chang and Blei, 2009] have used the dataset for link prediction task, however CiteSeer-DS1 is a hand-picked dataset prepared for document classification purposes [Lu and Getoor, 2003]. For both qualitative and quantitative evaluations on a user selected scientific documents dataset in a collaborative setting, we also acquired dataset from CiteULike 4 for over 2 years from November 2005 to January 2008 (referred as CiteSeer-DS2). The dataset is available at http://citeulike.org. Overall, there are 33,456 distinct papers in CiteULike sample. We map the document ids of CiteULike documents to document ids of CiteSeer documents 5 to gain access to citation network of the sample. The resultant CiteSeer-DS2 contains 18354 documents in which 9571 documents are cited. There are a total of 29645 unique authors in CiteSeer-DS2 out of which 15967 authors are cited

1 http://www.cs.umd.edu/˜sen/lbc-proj/LBC.html

2 http://CiteSeer.ist.psu.edu/

3 we use disambiguated authors for each documents available at http://CiteSeerx.ist.psu.edu/about/metadata

4 http://citeulike.org

5 mapping is obtained from http://citeulike.org

Table 1: Gibbs updates for ALT(i,ii), ACT(i,iii)

at least once. We follow the same preprocessing step as the CiteSeer-DS1 dataset.

Experimental Set-up: We choose to fix the hyperparameters and evaluate different models with the same setting. We set the hyper-parameters to the following values [Rosen-Zvi et al., 2004]: αθ = 50/T , αφ = 0.01, αϕ = 0.01. We run 1000 iterations of Gibbs sampling for training and extend the chain with 100 iterations over test set. For dynamic window selection, we collect 10 samples from the chain after every 10 iterations starting from 1000 iterations, and compute the new window with the average of the samples using Eq. 4. After the window update, we let the chain converge and start to update the window again. Starting with the sentence that contains the citation mention, we allow our window to grow up to a maximum of 5 sentences in either direction. The multinomial parameters of the model are calculated by taking expectations of the corresponding counts from 10 samples collected during test iterations.

4.2 Model Evaluation on Unseen Content

This task quantitatively estimates the generalization capabilities of a given model on unseen data. In particular, we compute the perplexity on the held-out test set. We run the inference algorithm exclusively on the unseen words in the test set of documents, same as [Rosen-Zvi et al., 2004], to obtain the log-likelihood of test documents. Before extending the Gibbs sampling chain and sweeping the test set, we first initialize the topic assignment to authors and unseen words randomly and run the Gibbs iteration on the test set with following Gibbs updates:

Superscript (.u ) stands for any unseen element. The sampling updates in Eq. 8 can be used to calculate the model parameters, Π = (θ, φ, ϕ) for unseen documents as:

The predictive log-likelihood of a text document in the test set, given the model Π = (θ, φ, ϕ), can be directly expressed as a function of the multinomial parameters:

Next, we compute the perplexity as defined below. Here, Nw is the total number of word occurrences in the test set.

Baselines: We use following two baselines from [RosenZvi et al., 2004] and [Tu et al., 2010], namely Author Topic Model (ATM) and Citation Author Topic Model (CAT) respectively. Since ATM does not learn from links among documents, comparison with ATM signifies the importance of learning from links along with the text of the documents. CAT model treats all the content of a citing document as context for any cited document within, therefore, comparison with CAT highlights the importance of choosing a context window surrounding the citation mention. We compare these baselines against the proposed Author Link model (ALT), fixed length window Author Cite Topic Model (FixedACT) 6 and dynamically selected window based ACT model (Dynamic-ACT). For our experiments, the training data consists of 4 splits with 75% documents (training docs) along with the 25% words of the remaining 25% of the documents (test docs). The rest 75%words in test documents are used to calculate log-likelihood. The average value over the 4 splits are reported in the experiments.

Fig. 3 (a)&(b) show the comparison of perplexity on test set of CiteSeer-DS1 and CiteSeer-DS2, respectively. The ATM model performs slightly better than the ALT model. We believe that this is because the links considered separately from the content actually deteriorate the prediction capability of the models over words. In contrast, while training, links along with the content help to learn the topics better. However, when all the content is treated as context for every cited article [Tu et al., 2010] in a given citing document, the performance deteriorates significantly. Therefore, we argue that a wise selection of context window is essential when a context sensitive topic modeling approach is considered.

Dynamic-ACT outperforms all the other approaches (see Fig. 3 (a)&(b)). During our experiments, we observed that the length of a relatively large fraction of citation contexts was limited to a single sentence that contains the citation mention. The fraction decreases as we increase the number of topics. Specifically, for CiteSeer-DS1, 78% of the total citation contexts were composed of only one sentence when we set the number of topics to 10. This number drops to 65% with 100 topics. Also, we found the average window length on CiteSeer-DS1 to be 1.4 with 10 topics and 1.6 with 100

6 we set the radius to be 10 words from the citation mention after stop word removal, i.e., 20 words window

number of topics. We observe the similar trend with CiteSeerDS1 where 81% of the total total citation contexts were composed of only one sentence with topic count 10 whereas the number decreases to 62% with 100 topics. Considering that the topic assignment to words is fine grained with a large number of topics, the growth outside the window is more likely to explain the finer details mentioned in the cited document.

4.3 Cited Author Prediction

In this task, we evaluate the capability of the models to predict the authors that this document links to. That is, given the text of a test document, which authors' work should this document cite to? The experimental design for this task is very similar to the one in the previous subsection. We again perform the Gibbs update following the sampling from conditional distribution in Eq. 8 and calculate the model parameters. With the model parameters for the ALT and ACT models, the probability p(c|wd), where c is the author to be cited given a document wd is:

Baselines: Because the ATM does not model the links, it is not possible to treat ATM as a baseline for this task. We keep all the other four comparisons intact for this task. The training data consists of 4 splits with 75% documents and their outgoing links to cited authors (training docs) and the 25% outgoing links of the remaining 25% of the documents (test docs). The rest of 75% outgoing links in the test documents are used for this task. We set the number of topic to be 100 for this task. We use Precision@K as the evaluation metric. The average value over the 4 splits are reported in the experiments.

To evaluate the prediction accuracy for the proposed models, we first label the actual authors that are cited by a test document as its relevant result set. We rank the authors in the train corpus against each test document using p(c|wd) and compare the models based upon the precision of the retrieved results. Fig. 3(c) & (d) shows the results for the three methods on CiteSeer-DS1 and CiteSeer-DS2, respectively.

4.4 Anecdotal Evidences

Table 2 shows the most likely words, interested and influential authors in 6 topics from the CiteSeer-DS2 dataset obtained using the ACT model (e.g. Griffith, Beal, etc., as interested authors and Mackay, Ghahramani and Hinton as influential authors in Bayesian learning). For each topic shown in the table, most influential authors are well known in their respective areas and their authored papers gets cited in the respective fields.

5 Conclusion

We propose novel models for author-author linkage conditioned on topics latent in the content of the documents. We exploit the citations between documents to infer influence of certain authors over topics. We also propose context sensitive extensions of the proposed model that incorporates the context of the cited document and how it infers the topic of both cited and citing authors with better quality.

6 Acknowledgement

This work was partially supported by grant HDTRA1-09-10054 from DTRA. We are thankful to Dr. Lise Gatoor for making the Citeseer dataset publically available. We are also thankful to Dr. Frank Ritter for editing the final draft.