Prior-Based Dual Additive Latent Dirichlet Allocation for User-Item Connected Documents

1 Introduction

User-item connected documents, which are written by users for specific items, occur in many scenarios. For example, in online shopping websites (e.g., Amazon 1 ), users prefer to write reviews to express their attitudes towards some products that they have bought. These reviews have become a major reference for candidate buyers to make decisions. Another example is that in location-based social networks (e.g., Foursquare 2 ), users always propose tips for some point-of-interests (POIs), which are beneficial for other users to gain knowledge about the POIs they have not visited before. In summary, user-item connected documents own dual roles, with each document associated with one user and one item simultaneously. They not only reflect the core concerns of users, but also indicate the characteristics of items. Without specific explanation, we also use documents to denote user-item connected documents for simplicity in the rest of this paper.

1 http://www.amazon.com/

2 https://foursquare.com/

Characterizing content of documents is an important problem and has many applications in different fields, e.g., natural language processing and information retrieval. Topic models [Hofmann, 1999; Blei et al., 2003b] are elegant ways to solve this problem by assuming each document has a topic distribution and each topic is represented as a distribution over words. By this way, documents are summarized by their associated topic distributions in a high level. For user-item connected documents, not only the learned topic distributions are useful as usual, but if the topic factors of users and items can be differentiated, then they can be used for creating their self-introduction profiles. These profiles are beneficial for other users to gain knowledge about items and retailers to understand what their consumers care about. Further, current popular recommender systems can utilize them to make explainable recommendations through content matching.

While many topic models have been proposed in the last decade, almost all of them are designed for some specific tasks and do not emphasize the dual role phenomenon of useritem connected documents, let alone automatically infer topic factors of users and items simultaneously. For one user-item connected document, its associated topic distribution should be influenced by its corresponding user and item together, and for a user or an item, its topics consist in multiple documents related to it. Therefore, the main challenge is to connect user, item, and reviews in a unified topic model while ensure different combinations of user and item pair tend to generate distinct topic distributions. Besides, it is better for the topic model to simultaneously infer user and item topic factors. Existing approaches such as author-topic model [Rosen-Zvi et al., 2004] can only capture user's topic distribution but ignore item's topics.

To address the above issues, we propose a novel generative probabilistic topic model called Prior-based Dual Additive Latent Dirichlet Allocation (PDA-LDA) to model the useritem connected documents. This model is somewhat inspired by [Wallach et al., 2009] which points out that asymmetric Dirichlet priors over topic distributions can lead to additional benefit for topic models than symmetric priors. In PDA-LDA, we account for the dual role phenomenon by associating Dirichlet prior of each document with their corresponding user and item. More specifically, PDA-LDA first assumes each user or item is represented with a topic factor. When a component of a topic factor takes a larger value, it means the corresponding user or item concentrates more on that topic. Then an exponential function is utilized to combine a pair of user and topic factor for each document through its additive property of parameters to form a new Dirichlet prior and thus every component of the prior takes positive value. As a result, each document has a different Dirichlet prior. The comparison between priors of Latent Dirichlet Allocation (LDA) [Blei et al., 2003b], asymmetric Latent Dirichlet Allocation (AS-LDA) [Wallach et al., 2009], and PDA-LDA is shown in Table 1.

In short, LDA is assigned a corpus-level symmetric prior, ASLDA has a corpus-level asymmetric prior, while PDA-LDA owns document-level asymmetric priors. The main advantages of PDA-LDA lie in three aspects. First of all, it models text better by accounting for the dual role in topic generations. Second, it can automatically summarize users and items by the learned topic factors. And last, it constructs documentlevel priors with pairs of user and item topic factors. This enables prediction on test documents since after model learning procedure, user and item topic factors are known for these documents.

Contributions. In all, the main contributions of this paper are summarized as follows:

• We are the first to address the dual role phenomenon for topic generations of topic models in user-item connected documents to the best of our knowledge.
• We propose a new generative model called PDA-LDA which accounts for the dual role phenomenon by connecting user and item topic factors to Dirichlet priors through exponential additive transformation.
• We evaluated the proposed model on several real data sets to verify the effectiveness of PDA-LDA. The experimental results demonstrate that our model not only achieves superior held-out perplexity on test data, but generates better topics for its good performance in document classification application as well.

In what follows, detailed model specification is introduced in Section 2. Then we provide experimental results and some analysis in Section 3. Related works are briefly discussed in Section 4. In the last, we draw a conclusion for this work..

2 Approach

This section will be divided into three parts. The first part is mainly about the description of PDA-LDA, including the generative process of the model. Then we derive the Gibbs EM based learning algorithm for the model. Finally, a concise introduction to prediction on test data is given. Before we proceed, the main mathematical symbols we will use later are shown for reference in Table 2.

2.1 Model Description

Standard topic models such as LDA always assume each document relates to a distribution over K latent components called topics and each topic can be explained by a distribution over all words in vocabulary W. Both the topic and word distributions are sampled from a Dirichlet distribution Dir(α) known as prior in Bayesian learning. Our model follows the basic topic generation process of LDA. The key difference lies in how we set the parameter α of Dirichlet prior for each useritem connected document. The hyper-parameter α has been demonstrated to influence the result of text modeling [Wallach et al., 2009].

As we discussed, each user or item has its own characteristics. We assume each user is associated with a topic factor φu ∈ RK and the same for each item, i.e., φi ∈ RK. In topic factors, components with larger value denote users and items concentrate more on those corresponding topics. Topic factors should influence generation of topic distributions for documents. To bridge the gap between them, one natural idea is to associate topic factors with Dirichlet hyper-parameter α as in Bayesian learning framework, prior distribution reveals prior knowledge about how data will be generated before it really comes into action. This is consistent with the intrinsics of generation of user-item connected documents. Imagine that when a user plans to write something about an item, the first thing he will do is to choose some themes from what he often concentrates on and also conformed to the context of the item. Obviously, the themes are both relevant to the user and the item. These themes can be regarded as latent topics studied in topic models.

For a document du,v , we associate user and item topic factors with Dirichlet prior parameter of its topic distribution as follows,

where exponential transformation ensures each component of αu,v d to be positive, which satisfies the requirements of Dirichlet parameters. So far, topic factors of user and item can influence the generation of topic distribution of document du,v through αu,v d . One natural extension is to consider background topic factor φb additionally. It is necessary since some words such as conjunction words frequently occur in documents. The topic relates to these words are not specially owned by users or items. Therefore, the complete transformation between topic factors and Dirichlet parameter is defined to be

Based on the above analysis, we can summarize the generative story of PDA-LDA for user-item connected documents as below,

1. Draw background topic factor φb ∼N(λB).
2. For each user u, draw user topic factor φu ∼N(λU).
3. For each item v, draw item topic factor φv ∼N(λV ).
4. For each topic k, draw word distribution βk ∼ Dirichlet(η).
5. For each user-itemconnected document du,v in document collection D,
6. (a) Draw topic distribution θd ∼ Dirichlet(αu,v d ) where αu,v d is calculated through Equation(2).
7. (b) For each position j in the document du,v :
8. (b.1) Sample topic zj ∼Mult(θdu,v).

(b.2) Drawword wj ∼Mult(βzj ).

The complete graphical model representation of our proposed model is provided in Figure 1. The joint probability of a whole corpus D containing user-item connected documents and topic factors Θt = {φU , φV , φb} is defined as follows,

2.2 Learning

In this work, our goal is to learn optimal topic factors Θt and key latent variables θ and β by maximizing the log-likelihood of joint probability shown in Equation (3),

However, it is intractable to directly optimize the above function and compute the posterior distribution of θ and β due to the summarization of latent topics zi in discrete space and coupling of θ and β. Luckily, if the assignments of latent topics can be inferred for all words in reviews, then not only θ and β can be calculated, but also Θt are tractable to be optimized. This intuition leads to the idea of Gibb EM algorithm.

Gibbs EM learning algorithm is widely adopted in (partially) Bayesian latent factor models [Wallach, 2006; Liu et al., 2013] which alternates between sampling the value of latent variables (Expectation Step) and optimizing some model parameters (Maximization Step). More specifically, we adopt collapsed Gibbs Sampling to determine all the latent topics in E-step and optimize Θt through gradient ascent in M-step.

E-step

Given a word position j in document du,v , the key inferential problem in collapsed Gibbs sampling is to derive the posterior distribution of its associated topic, i.e., P(zu,v d,j = k|zu,v d,-j , wu,v d ) where zu,v d,-j denotes all topic assignments of words in document d are known except the word in position j. As Θt are fixed in E-step, αu,v d can be calculated through Equation (2). By applying Bayesian formula and conditional dependence property, it is easy to derive the following formal definition of the posterior distribution according to the results of [Griffiths and Steyvers, 2004],

where all the counts do not include the current word. The above equation is very similar to that in standard topic model except the Dirichlet prior αu,v d,k .

M-step

After getting the latent topic assignments for all words in reviews, the objective log-likelihood of joint probability becomes P(D,Z,Θt) and thus the summarization term in Equation (3) vanishes. Due to the conjugate prior property of Dirichlet multinomial distribution [Heinrich, 2004; Griffiths and Steyvers, 2004], θ and β can be integrated out and P (D, Z, Θt) is now reformulated as below,

where Γ is Gamma function with form Γ(x) = (x - 1)! when x is an integer.

By maximizing the log-likelihood of the above joint probability with gradient ascent algorithm, we can get optimal Θt. Gradient ascent consists of two steps. The first step is to educe the gradients of the parameters. For example, the gradient of φu,k is

where Ψ is Digamma function which is equal to logarithmic derivative of the Gamma function. The gradients of φv,k and φb,k are analogous to φu,k except the summarization condition and regularization term. We should emphasize although for one review, ∂φu,k and ∂φv,k are similar, different reviews connect to different users and items and thus the updates for the users and items are distinct from a whole perspective.

Based on the obtained gradients, the second step of gradient ascent is to update model parameters through

where ω is the learning rate of the algorithm. Superscript t denotes the finished number of iterations.

Besides, after the iterative learning process converges, we can calculate β and θ by collecting enough subsequent samples though a burn-in process as [Blei et al., 2003a],

where N u,v t,d,k denotes the samples collected in the t iteration. In summary, the whole learning algorithm for PDA-LDA is concluded in Algorithm 1.

2.3 Prediction

When the PDA-LDA model is utilized for document modeling on test dataset, Θt and β should be fixed to be the optimal values learned from training set. The only important step is to infer latent topic assignments over test documents by sampling from a resembling formula as Equation (5), which additionally incorporates relevant count variables of test documents.

3 Experiments

In this section, we first introduce the datasets we used and the preprocessing steps for them. Then we discuss the adopted comparison models and the hyper-parameter setting of PDALDA. Finally, we analyze the experimental results to demonstrate the effectiveness of our method.

Algorithm 1: The Gibbs EM Algorithm for PDA-LDA

Input: User-item connect document collection D.

Output: Optimal user, item, and background topic factors

Θt, document-topic distribution θ, topic-word distribution β

• 1 Initialize hyper-parameters and Θt
• 2 Randomly draw latent topic assignments of words in documents
• 3 tt = 0 4 while Not converged and tt ≤ Itermax do 5 E-step: 6 (a) Sampling all words' latent topic assignments 7 through Equation (5) 8 M-step: (a) Calculate the gradients through Equation (7)
• 9 10 (b) Update Θt through Equation (8) 11 tt ← tt + 1
• 12 Calculate θ and β through Equation (9) and (10) under a burn-in process

3.1 Dataset

We adopt three real data collections fromYelp 3 and [McAuley and Leskovec, 2013]. Based on their origins, we denominate the three data sets as Yelp, AmazonFood and AmazonSport, respectively. To clean text data, we adopt the following four processing steps: (1) converting all letters into lowercase, (2) filtering stop words 4 and punctuations, (3) removing reviews which are too short, and (4) saving frequent words to form a vocabulary. After cleaning text data, we further remove users and items with less than 5 reviews to ensure that users and items have enough related documents. Finally, we obtain the experimental data sets whose basic statistics are shown in Table 3. We randomly divide the two collections into train, validation, and test set with the ratio 7 : 1 : 2 for testing held-out perplexity and further binary document classification task.

3.2 ComparisonModels

To verify the effectiveness of our model, we first introduce the comparison models adopted in the experiments.

Latent Dirichlet Allocation (LDA): Although LDA [Blei et al., 2003b] can neither capture author topics, nor obtain item topics, it is still necessary to analyze its result to verify the effectiveness of AS-LDA and PDA-LDA for exploring asymmetric Dirichlet prior as we mentioned in Section 1.

3 http://www.yelp.com/dataset challenge

4 http://jmlr.csail.mit.edu/papers/volume5/lewis04a/a11-smartstop-list/english.stop

Author-topic Model (ATM): We adapt ATM [Rosen-Zvi et al., 2004] to our problem by regarding users as authors in the model. ATM somewhat resembles PDA-LDA as both of them can obtain users' topics. However, ATM cannot handle the double role phenomenon in user-item connected documents. Intuitively, it will performworse than the newmodel.

Gibbs Sampler for Dirichlet Multinomial (GSDMM): GSDMM [Yin and Wang, 2014] is a standard topic model for handling short text such as tweets in social media by assuming all words in a document have the same topic and used as a comparison here due to the short length of the documents shown in Table 3,

Asymmetric Latent Dirichlet Allocation (AS-LDA): ASLDA [Wallach et al., 2009] is a strong comparison as it addresses popularity bias for each topic by incorporating asymmetric Dirichlet prior. It can be regarded as the most similar method to PDA-LDA which only considers the background topic factor for topic generation process.

3.3 Parameter Setting

All the hyper-parameters are determined based on their performances on validation datasets. For all the comparison methods and PDA-LDA, we assign 0.1 to η. For comparisons except AS-LDA, we choose α to be 0.1 as well for its good performance. The concentration parameter α′ in AS-LDA is tunned to be 0.1. Apart from η, λU, λV , and λb are set to be 1 for PDA-LDA uniformly.

3.4 Results

Results on Text Modeling

We first compare all the adopted models in terms of perplexity on test datasets. Perplexity is widely used in probabilistic model for checking their quality. It is directly relevant to log-likelihood of probability and normally defined as below,

The detailed results are shown in Table 4. We notice that GSDMM performs not very well in user-item connected document modeling although the average length of the documents is short. One reason is that user-item connected documents are still longer than tweets in social media and very short text is more likely to contain only one topic.

ATM dose not show promising results although it models the role of users in topic generation. This may be explained by the fact that ATM is originally designed for the scenario where each document has multiple authors (e.g., research papers). As each user-item connected document is uniquely owned by one user, the ATMmay not handle it very well.

AS-LDA outperforms LDA consistently in three datasets. This makes sense that asymmetric Dirichlet prior captures popularity bias of topics in generation process from a corpus level while LDA ignores the bias. It demonstrates that the idea of modifying hyper-parameters of Dirichlet prior for topic distribution is promising and also provides evidence for PDALDA to further differentiate prior parameters.

It is clear that PDA-LDA achieves superior performances among all the adopted models. Especially, PDA-LDA behaves better than the strong comparison AS-LDA significantly. This indicates that asymmetric Dirichlet prior which capture topic concentration bias frommore grained document level is better than from corpus level. Moreover, we discover AS-LDA gains a minor improvement over LDA in the last dataset, while the decrease of perplexity of PDA-LDA is prominent, which indicates our model has wider applicability.

Dimension Influence on Text Modeling

We analyze the performance evolution with change of number of latent components from 10 to 100 with a step size of 10. We only compare LDA, AS-LDA, and PDA-LDA as their results are closer than the other two models. As Figure 2 shown, the perplexity of all the three models decrease steadily with increase of topic components. We choose K = 40 for other experiments in this work by considering a trade-off between efficiency and effectiveness as larger number of components costs more time to learn models. Besides, the perplexity differences between these models are relatively evident when K ≥ 40 and thus the comparisons in experiments will not be influenced. In general, PDA-LDA outperforms LDA and AS-LDA regardless of how the number of topic components varies.

Application to Document Classification

As the perplexity does not correlate with human judgments, it may be better to test topic models when using its generated topics for other tasks [Chang et al., 2009]. In this work, we apply LDA, AS-LDA, and PDA-LDA to binary document classification task by utilizing topic distributions learned from them as features for supervised classification models. We should emphasize our goal here is to test the quality of learned topics, but not to compare with state-of-the-art document classification methods. On the other hand, features of topic models can be regarded as a complementary for other supervised methods.

Based on the rating scores associated with the three datasets, we divide all the documents of training, validation, and test sets into two classes. The critical value between for all datasets is specified to be 3.5 ([1-5]). The number of documents in each class are shown in Table 6.

We compare LDA, AS-LDA, and PDA-LDA in terms of average accuracy, precision, and F1 metrics. Among them, precision and F1 are first computed in each class and then take weighted average over two classes. We adopt standard supervised classifier, i.e., Support Vector Machine (SVM), in the experiments. As the results shown in Table 5, PDALDA achieves best performance across all the datasets. Hence the quality of topics generated by PDA-LDA turns out to be better than those of the other models. Besides, although AS-LDA gains lower perplexity than LDA in Yelp dataset, it does not perform better in terms of classification metrics. This reveals obtaining lower perplexity does not ensure better discriminative power of features for classification performance definitely, which also reveals the robustness of PDA-LDA.

Case Study

To give an intuitive interpretation of PDA-LDA, we randomly sample two users and two items to reveal their topic factors in Figure 3, and show their representative topics (topic index corresponds to the largest value for each user and item topic factor) learned from AmazonFood data in Table 7.

We find that the sampled users and items only focus on several topics, for many values less than 0. Thus, they can be characterized by a few critical topics which have clear indications. For example, user-2 likes chips very much.

Table 7: Selected topics from AmazonFood data set.

T3: amazon, price, store, buy, local, order, product, find, bought, shipping

T9: food, cat, cats, chicken, eat, tuna, canned, fish, cans, flavors

T25: chips, flavor, salt, potato, sweet, bag, taste, chip, popchips, good

T27: bars, bar, fat, calories, snack, fiber, grams, protein, healthy, sugar

4 RelatedWork

Topic models are popularly utilized for modeling text. Different topic models concentrate on different aspects of text. For example, [Blei and McAuliffe, 2007] exploited the labels corresponding to documents to construct their supervised topic model. The authors in [Mei et al., 2008] considered the link relations between documents and assume linked documents have similar topic distributions. Recently, many topic models have been proposed for modeling review text [Titov and McDonald, 2008; Zhao et al., 2010; Sauper et al., 2011; Moghaddam and Ester, 2011; Mukherjee et al., 2014]. Many of them are designed for sentiment analysis, including aspects mining and rating prediction. While our model is not designed specially for review text, but a wider range of text, i.e., user-item connected documents and we concentrate on accounting for the dual role in the topic generation process of the text which are ignored in these models. Collaborative topic model (CTM) [Wang and Blei, 2011] considered the user and item information simultaneously for predicting ratings of users for items. Nevertheless, its topic generation process is the similar to LDA, which does not take the dual role into consideration. Author-topic model [Rosen-Zvi et al., 2004] is somewhat similar with our model as it can obtain the topic distribution of users. However, the topic generation is only influenced by users and it still cannot model the dual role phenomenon. Many papers deal with dual roles in different research fields, such as the dual role of hashtag in social networks [Yang et al., 2012] and users in question answering [Xu et al., 2012]. However, few previous works emphasize the dual role phenomenon in topic generation process of topic models.

5 Conclusion

In this paper, we propose a new probabilistic topic model called PDA-LDA to account for the dual role phenomenon of user-item connected documents. PDA-LDA models the topic generation process through the joint effect of user, item, and background topic factors on the Dirichlet prior of topic distribution. A Gibbs EM based learning algorithm is derived for the newmodel to learn optimal topic factors. The experimental results on real data collections have shown that PDA-LDA achieves better held-out perplexity and binary classification accuracy than several other models.

Acknowledgments

This work was supported in part by National Basic Research Program of China (973 Program) under Grant No. 2011CB302206, National Natural Science Foundation of China under Grant No. 61272088, and Tsinghua University Initiative Scientific Research Program.