A Correlated Topic Model Using Word Embeddings

1 Introduction

Conventional topic models, such as Probabilistic Latent Semantic Analysis (PLSA) [Hofmann, 1999] and Latent Dirichlet Allocation (LDA) [Blei et al., 2003], have proven to be a powerful unsupervised tool for the statistical analysis of document collections. Those methods [Zhu et al., 2012], [Zhu et al., 2014] follow the bag-of-word assumption and model each document as an admixture of latent topics, which are multinomial distributions over words.

A limitation of the conventional topic models is the inability to directly model correlations between topics, for instances, a document about autos is more likely to be related to motorcycles than to politics. In reality, it is natural to expect correlated latent topics in most text corpora. In order to address this limitation, the Correlated Topic Model (CTM) [Blei and Lafferty, 2006a] replaces the Dirichlet prior with logistic normal distribution which allows for covariance structure among the topics.

Nowadays, the rapidly developing technique in natural language processing - word embeddings [Bengio et al., 2003], [Mikolov and Dean, 2013] - provides us with the possibility to model topics and topic correlations in the continuous semantic space. Word embeddings, also known as word vectors and distributed representations of words, are real-valued continuous vectors for words, which have proven to be effective at capturing semantic regularities in language. Words with similar semantic and syntactic properties tend to be projected into nearby area in the vector space. By replacing the original discrete word types in LDA with continuous word embeddings, Gaussian-LDA [Das et al., 2015] has shown that the additional semantics in word embeddings can be incorporated into topic models and further enhance the performance.

The main goal of correlated topic models is to model and discover correlation between topics. And now we know that word embeddings are able to capture semantic regularities in language, and the correlations between words can be directly measured by the Euclidean distances or cosine values between the corresponding word embeddings. Moreover, semantically related words are close to each other in space and should be more likely to be grouped into the same topic. Since Gaussian distributions depict a notion of centrality in continuous space, it is a natural choice to model topics as Gaussian distributions over word embeddings in space. Therefore, the motivation of this paper is to model topics in the word embedding space, exploit the known correlation information at word level and further improve the correlation discovery at topic level.

In this paper, we propose the Correlated Gaussian Topic Model (CGTM) to model both topics and topic correlations in the word embedding space. More specifically, first we learn word embeddings with the help of external large unstructured text corpora to obtain additional word-level correlation information; Second, in the vector space of word embeddings, we model topics and topic correlations to exploit useful additional semantics in word embeddings, wherein each topic is represented as a Gaussian distribution over the word embeddings and topic correlations are learned among those Gaussian topics. Third, we develop a Gibbs sampling algorithm for CGTM.

To validate the efficacy of our proposed model, we evaluate our model on the 20 Newsgroups dataset and the Reuters21578 dataset, which are well-known dataset for experiments in text mining domain. The experimental results show that our model can discover more reasonable topics and topic correlations than the baseline models.

2 RelatedWorks

Correlation is an inherent property in many text corpora, for example, [Blei and Lafferty, 2006b] explores the time evolution of topics and [Mei et al., 2008] analyzes the locational correlation among topics. However, due to the use of the Dirichlet prior, traditional topic models are not able to model the topic correlation directly. CTM [Blei and Lafferty, 2006a] proposes to use logistic normal distribution to model the variability among topic proportions and thus learn the covariance structure of topics.

Word embeddings can capture the semantic meanings of words via low-dimensional real-valued vectors [Mikolov and Dean, 2013], for example, vector operation vector('king') - vector('man') + vector('woman') results in a vector which is very close to vector('queen'). The concept of word embeddings was first introduced into natural language processing by Neural Probabilistic Language Model (NPLM) [Bengio et al., 2003]. Due to its effectiveness and wide variety of application domains, word embeddings have garnered a great deal of attention and development [Mikolov et al., 2013], [Pennington et al., 2014], [Morin and Bengio, 2005], [Collobert and Weston, 2008], [Mnih and Hinton, 2009], [Huang et al., 2012].

Since word embeddings carry additional semantics, many researchers have tried to incorporate them into topic models to improve the performance [Das et al., 2015], [Li et al., 2016], [Liu et al., 2015], [Li et al., 2017]. [Liu et al., 2015] proposed Topical Word Embeddings (TWE) which combines word embeddings and topic models in a simple and effective way to achieve topical embeddings for each word. [Das et al., 2015] uses Gaussian distributions to model topics in the word embedding space.

The aforementioned models either fail to directly model correlation among topics or fail to leverage the word-level semantics and correlations. We propose to leverage the wordlevel semantics and correlations within word embeddings to aid us in learning the topic-level correlations.

3 LearningWord Embeddings

We begin our topic discovery process with learning the word embeddings with semantic regularities. Unlike the traditional one-hot representations of words which encode each word as a binary vector of N (the size of vocabulary) digits with only one digit being 1 the others 0, the distributed representations of words encode each word as a unique real-valued vector. By mapping words into this vector space, word embeddings are able to overcome several drawbacks of the one-hot representations such as the curse of dimensionality, the out-ofvocabulary words and the lack of semantics.

In this paper, we adopt a recently developed, very effective and efficient distributed representations of words based model called Word2Vec [Mikolov and Dean, 2013] to train word embeddings. In the learning process of Word2Vec, words with similar meanings gradually converge to nearby areas in the vector space. In this model, words in the form of word embeddings are used as input to a softmax classifier and each word is predicted based on its neighbourhood words within a certain context window.

Having learnt the word embeddings, given a word wdn, which denotes the nth word in dth document, we can enrich that word by replacing it with the corresponding word embedding. The following section describes how this enrichment is used in a generative process to model topics and topic correlations.

4 Generative Process

Trained word embeddings give us useful additional semantics, which helps us discover reasonable topics and topic correlations in the vector space. However, each document now is a sequence of continuous word embeddings instead of a sequence of discrete word types. Therefore, conventional topic models no longer are applicable. Since the word embeddings are located in space based on their semantics and syntax, inspired by [Hu et al., 2012] and [Das et al., 2015], we consider them as draws from several Gaussian distributions. Hence, each topic is characterized as a multivariate Gaussian distribution in the vector space. The choice of Gaussian distribution is justified by the observations that Euclidean distances between word embeddings are consistent with their semantic similarities.

The graphical model of CGTM is shown in Figure 1. More formally, there are K topics and each topic is represented by a multivariate Gaussian distribution over the word embeddings in the word vector space. Let µk and Σk denote the mean and covariance for the kth Gaussian topic. Each document is a admixture of K Gaussian topics. ηd is a K dimensional vector where each dimension represents the weight of each topic in document d. Then the document-specific topic distribution θd can be computed based on ηd. µc is the mean of η and Σc is the covariance of η. By replacing the Dirichlet priors in conventional LDA with logistic normal priors, the topic correlation information is integrated into the model. µ0, Σ0, ν0, µ, Σ and ν are hyper parameters for Gaussian topics and logistic normal priors.

Note that variables in bold font mean they are either vectors or matrices, for example, wdn. The generative process is as follows:

1. DrawΣc ∼W-1 (Ψ, ν).
2. Drawµc ∼N(µ, 1 τc Σc).
3. For each Gaussian topic k = 1, 2, · · · ,K:
4. (a) Draw topic covariance Σk ∼ W-1 (Ψ0, ν0).
5. (b) Draw topic mean µk ∼ N(µ0, 1 τ Σk).
6. For each document d = 1, 2, · · · ,D:
7. (a) Draw ηd ∼N(µc,Σc).
8. (b) For each word index n = 1, 2, · · · ,Nd:
9. Draw a topic zdn ∼Multinomial(f(ηd)).
10. Draw a word wdn ∼N(µzdn ,Σzdn ).

where τ and τc are constant factors; and f(η) is the logistic transformation:

The following conjugate priors are utilized for topic parameters: a Gaussian distribution N for the mean and an inverse Wishart distribution W-1 for the covariance. However, note that there is still a non-conjugacy problem between the logistic normal distribution and multinomial distribution, and we will solve this with data augmentation technique in the following section.

5 Parameter Inference

The observed variables are documents consisting of word embeddings, and our goal is to infer the posterior Gaussian distribution of each topic, topic assignment of each word, and topic correlations. Given D documents and the corresponding word embeddings w, the joint distribution of topic assignments z and logistic normal parameters η is:

where p(w|z) is the Gaussian probability of words w under topic assignments z. Because of the choice of conjugate priors for topic parameters, those variables can be integrated out and we can efficiently re-sample topic assignment for each word. However, due to the non-conjugacy between the logistic normal and multinomial distributions, regular Gibbs sampling scheme doesn't work for the logistic normal parameters. Thus we adopt Gibbs sampling with data augmentation technique to solve this non-conjugacy problem.

5.1 Sampling Topic Assignments

Since the topic parameters have conjugate priors, the sampling process of topic assignments is similar to the Gibbs sampling scheme for LDA [Griffiths, 2002]. Given η and z-dn which is the topic assignment scheme without considering the current word wdn, the topic of each word is drawn iteratively as:

where Tr(w|µ, Σ) is the multivariate Student's t-distribution for Gaussian sampling with (r = νk - dim + 1) being its degrees of freedom and dim being the dimensionality of word embeddings. (νk = ν + Nk) and (τk = τ + Nk) are the parameters of topic k, where Nk denotes the total number of words that are assigned to topic k.

5.2 Updating Gaussian Topics

Every time we re-sample topic assignment zdn, we need to update the two involved Gaussian topics because the current word wdn is either leaving or joining this Gaussian topic. Following [Das et al., 2015], we derive the updates for µk and Σk of the posterior Gaussian distributions for topic k:

where ¯wk is the sample mean of all the word embeddings assigned to topic k, and Ck is the scaled form of sample covariance of all the word embeddings assigned to topic k. These two intermediate variables are calculated as follows:

where δ(zdn, k) is the Kronecker delta function that δ(zdn, k) = 1 if zdn = k, δ(zdn, k) = 0 otherwise.

5.3 Sampling Logistic Normal Parameters

Given topic assignments, directly sampling logistic normal parameters η is difficult due to non-conjugacy. To address the non-conjugacy problem between the logistic normal distribution and multinomial distribution, following [Holmes et al., 2006], [Polson et al., 2013] and [Chen et al., 2013], we sample the logistic normal parameters η based on z with auxiliary variables. For document d, the likelihood for ηk d conditioned on η-k d is:

where zk dn is the topic indicator that zk dn = 1 if word wdn is assigned to kth topic, zk dn = 0 otherwise. ρkd = ηk d - ζ k d , ζ k d = log(∑j̸=k exp(ηjd )) and C k d is the number of words assigned to topic k in document d. Therefore, we obtain the posterior distribution of ηk d proportional to multiplying the likelihood by the prior:

For the prior part, it is a univariate Gaussian distribution conditioned on the other logistic normal parameters in the current document η-k d . Thus, given η-k d and µc, Σc of the multivariate Gaussian distribution over η, we have:

where Λ = Σ-1 c is the precision matrix. However, the nonconjugacy makes it difficult to directly calculate the likelihood l(ηk d |η-k d ) and thus unable to directly sample ηk d .

By introducing auxiliary Polya-Gamma variable λkd [Polson et al., 2013], we are able to get around the non-conjugacy problem and the likelihood l(ηk d |η-k d ) can now be expressed

where κkd = C k d -Nd/2 and p(λkd |Nd, 0) is the Polya-Gamma distribution PG(Nd, 0). As one can observe, Equation 9 implies that p(ηk d |η-k d , z,w) is the marginal distribution of the joint distribution:

Therefore we can sample ηk d based on the auxiliary variable λkd . The sampling procedure is as follows:

• Sampling λkd : according to Equation 10 and [Polson et al., 2013], we have the conditional distribution p(λkd |z, w, η) ∝ exp(- λkd (ρkd )2 2 )p(λkd |Nd, 0), which results in a Polya-Gamma distribution PG(Nd, ρkd ). Following the ideas in [Polson et al., 2013] and [Chen et al., 2013], Polya-Gamma variables can be drawn in O(1) time, and so a sample of λkd is obtained.
• Sampling ηk d : according to Equation 10, we can sample ηk d with posterior probability:

This results in a univariate Gaussian distribution N (γ k d , (τ k d )2 ) conditioned on the auxiliary variable λkd , where γ k d = (τ k d )2 (σ-2 d µkd + κkd + λkd ζ k d ) and (τ k d )2 = (σ-2 d +λkd )-1 . Thus, given the auxiliary variable λkd , ηk d can be easily drawn from a univariate Gaussian distribution.

5.4 Updating Topic Correlation

Given {ηd}Dd=1 , the logistic normal parameters µc and Σc are updated as:

where ¯η is the mean of {ηd}Dd=1 , and Q = 1D (ηd - ¯η)(ηd - ¯η)T .

6 Experiments

In this section, we carry out experiments on two realworld text collections - the 20 Newsgroups dataset 1 and the Reuters-21578 dataset 2 to demonstrate the efficacy of our proposed model. 20 Newsgroups contains approximately 20,000 text documents partitioned evenly across 20 different newsgroups. Reuters contains about 10,000 documents, but due to the imbalance of each category, only the largest 8 categories are selected in Reuters, leaving us with 7,674 documents in total. Both datasets have become popular datasets for experiments in many data mining tasks, such as text classification. Each document is associated with one single category label. For 20 Newsgroups, correlation is exhibited across different newsgroups (e.g. rec.sport.baseball and rec.sport.hockey), which makes this dataset a suitable choice to verify the effectiveness of topic correlation discovery for CGTM.

We compare CGTM with three topic modeling methods: LDA [Blei et al., 2003], CTM [Blei and Lafferty, 2006a] and Gaussian-LDA [Das et al., 2015]. CTM replaces the dirichlet prior in LDA with logistic normal distribution to capture the correlation among topic proportions. Gaussian-LDA was first proposed for audio retrieval [Hu et al., 2012] and then used to leverage word embeddings in the continuous vector space [Das et al., 2015].

To learn high quality word embeddings, we combine the current dataset with Wikipedia as the knowledge source. The motivation of using Wikipedia as the supplemental source lies in the sheer range of topics and subjects that are covered and it allows us to enhance the semantics of word embeddings extracted from 20 Newsgroups and Reuters. In the experiment, we set the dimensionality of word embeddings to 100, and the context window size to 12. We train word embeddings for 100 epochs.

We are interested to see if the learned topics can reveal a similar mixture and correlation with the ground truth text categories. Hence we set the number of topics K to the number of categories. For uniformity, all the models are implemented with Gibbs sampling and run for 100 iterations. The Gaussian topic hyper parameter µ0 is set to the sample mean of all the word vectors, the initial degree of freedom ν0 to the dimensionality of word embeddings, and Ψ0 to an identity matrix.

1 www.qwone.com/ jason/20Newsgroups/

2 www.daviddlewis.com/resources/testcollections/reuters21578/

6.1 TopicWords and Correlations

To investigate the quality of topics and the topic correlations discovered by CGTM, we visualize each topic with their top words as well as the topic correlations. To make the visualization clearer, we select only 6 categories from the 20 Newsgroups dataset whose topic words and correlations can be easily recognized and defined. The selected newsgroups are "rec.autos", "rec.motorcycles", "rec.sport.baseball", "rec.sport.hockey", "talk.politics.guns", and "talk.politics.mideast". Thus, in this experiment, we set the number of topics K to 6. As Figure 2 shows, we display top 10 words for each topic discovered by CGTM and map the corresponding word embeddings into a two-dimensional space via Principal Component Analysis (PCA). The size of each word varies with its relative frequency in the corresponding topic. The different colors and shapes of words indicate they are from 6 different topics. Each circle depicts the Gaussian distribution for each topic. The detected topic correlation is represented as a dashed line between topics. As one can observe, all the newsgroups, Hockey (topic 1), Baseball (topic 2), Mideast (topic 3), Guns (topic 4), Autos (topic 5) and Motorcycles (topic 6), are successfully discovered with reasonable topic words.

As the ground truth labels indicate, one can easily figure that Autos is correlated with Motorcycles, Baseball is correlated with Hockey, and Guns is correlated with Mideast. The dashed lines in the figure denote the automatically detected topic correlations by CGTM. With the help of word embeddings and Gaussian topics, topic correlations are also correctly detected, as the dashed lines show. We can see that, since word embeddings can capture the regularities in language such as synonyms, two topics tend to be correlated if their topic word embeddings overlap in the continuous vector space. This demonstrates how the known word-level correlation information can aid us in discovering the topic-level correlations.

In this subsection, we qualitatively exhibit the effectiveness of discovering topics and topic correlations of CGTM. In the following subsections, we will quantitatively evaluate CGTM on topic coherence and topic correlation discovery.

6.2 Topic Coherence

In order to quantitatively assess the topic coherence, we adopt a metric called coherence score of topics proposed by [Mimno et al., 2011] which is able to automatically evaluate the coherence of each discovered topic. Given a topic z and its top T words V z = {vz 1 , vz 2 , ..., vzT }, the coherence score of this topic is defined as:

where D(vz l ) is the document frequency of word vz l and D(vz t , vz l ) is the number of documents in which words vz t and vz l co-occurred. The coherence score follows the intuition that words from the same topic tend to co-occur in documents. This topic coherence score has been proven to be highly consistent with human coherence judgements [Mimno et al., 2011].

The topic coherence result on the 20 Newsgroups dataset is reported in Table 1. In order to investigate the overall quality of all the discovered topics, the average coherence score is reported, which is calculated as ¯ C = 1K ∑z C(z; V z ). To make this evaluation more comprehensive, the number of topic words T ranges from 5 to 50. For all the models, the topic words are ordered by word counts in each topic. Though for Gaussian-LDA and CGTM, topic words can also be ordered by word probabilities under each Gaussian topic, we still order them by word counts, since first, the Gaussian posterior probability information has already been fully utilized in the training phase and second, this coherence score is more appropriate to measure frequent words in a topic. The result shows that the topic words discovered by our model are more coherent than the topic words discovered by the baseline models.

6.3 Document Topics and Topic Correlation

We see that the topics discovered by CGTM qualitatively exhibit good topic words and reasonable correlations, and CGTM also outperforms the baseline models in terms of coherence score. But are the topics discovered by our model really corresponding to the coherent news categories? If yes, it would be very convenient for us to assess the quality of the detected topic correlations, because the correlations among the ground truth newsgroups labels are well defined. For example, 20 Newsgroups categories "rec.autos" and "rec.motorcycles" are clearly correlated, and Reuters categories "money" and "trade" should also exhibit correlations. To answer this question, we compare the ground truth document labels with the document-topic labels discovered by the models to see if they are consistent. The label of each document comes from the dataset and is used as the ground truth. The document-topic label of each document is assigned by the models. More specifically, for each model, we can assign one single topic to document d according to:

So this is a clustering evaluation problem where each document is a sample. To solve the cluster matching problem, e.g., ground truth label 1 may correspond to topic 5 instead of topic 1, we adopt pairwise comparison [Menestrina et al., 2010] to measure the consistency between the ground truth document labels and the learned topic representation of documents. The pairwise comparison is defined as:

where E and G are two clustering solutions corresponding to the document-topic clusters and the ground truth document labels respectively in our case, and pairE denotes the set of pairs in clustering result E, and ||pairE|| represents the number of instances in pairE. The experimental results of document clustering on 20 Newsgroups and Reuters are reported in Table 2 and Table 3 respectively. We can see that, with respect to the consistency between ground truth document labels and discovered topics, CGTM outperforms the other baselines on both datasets.

7 Conclusions

In this paper, we have proposed a correlated topic model using word embeddings. Word embeddings learnt from large, unstructured corpora, such as Wikipedia, can aid us in modeling topics and topic correlation by bringing in additional useful semantics. The known word-level correlation information in word embeddings is passed to topic-level correlation discovery task via Gaussian topics. In our case, the word embeddings are trained on the combined collections of Wikipedia and the 20 newsgroups dataset. We model each topic as a Gaussian distribution over word embeddings and directly learn topic correlations in the vector space. The experiments qualitatively show CGTM is able to learn meaningful topics and topic correlation, and quantitatively validate the effectiveness of our model in terms of topic coherence score and document clustering on two real-world datasets.

Acknowledgements

This work was supported in part by the US National Science Foundation under grants NSF IIS-1218393, IIS-1514204 and IIS 1319973. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.