Layer-Assisted Neural Topic Modeling over Document Networks

1 Introduction

Neural topic modeling [Miao et al., 2017; Srivastava and Sutton, 2017] refers to extract latent topics from text data by using Neural Variational Inference (NVI) [Miao et al., 2016], which combines stochastic variational inference with deep neural networks [Kingma and Welling, 2014; Mnih and Gregor, 2014; Rezende et al., 2014]. Thanks to the black-box nature of NVI, the neural topic models are flexible, efficient, and powerful for various types of model structures, beyond traditional topic models such as Latent Dirichlet Allocation (LDA) [Blei et al., 2003]. As the art marrying topic modeling with deep neural networks, it has recently attracted much attention from the machine learning community [Zhang et al., 2018; Dieng et al., 2020; Burkhardt and Kramer, 2019; Liu et al., 2019; Isonuma et al., 2020; Zhu et al., 2020].

∗Corresponding Author

†Contributing equally with the first author.

Nowadays, the text data with network links are becoming pervasive in many real-world scenarios, e.g., web pages with hyperlinks, scientific papers with citations, and Tweets with friendships, to name just a few. This kind of data challenges the neural topic modeling for simultaneously expressing the textual contents and network links of texts. More formally, we describe the text data with network links and the corresponding problem of neural topic modeling as follows: Consider a text document collection Ω = {X,Π}, where X = {xd}Dd=1 and Π = {πij}Di,j=1 denote the texts and the links between texts, respectively. Each text xd ∈ RV is represented by a vector of the vocabulary; and for each link, πij = 1 indicates that the texts xi and xj are connected, and πij = 0 otherwise. Generally, the objective of neural topic modeling over Ω is to extract latent topic representations of texts, which can effectively represent content themes and maintain the network links simultaneously.

To our knowledge, there are only very few previous investigations on this subject, e.g., the conventional Relational Topic Model (RTM) [Chang and Blei, 2009] and its neural variant Neural Relational Topic Model (NRTM) [Bai et al., 2018]. The NRTM can be interpreted as a deep autoencoder, where it encodes the texts into their corresponding topic representations and adopts them to separately reconstruct the texts and network links. In this work, we aim to enhance the topic representation encoding beyond NRTM by not only using text contents, but also the assisted network links. Motivated by this, we propose a novel neural topic model, namely Layer-Assisted Neural Topic Model (LANTM). which can also be interpreted from the perspective of Variational Auto-Encoders (VAE). Encoding: We treat the network links as a text graph, therefore we design an augmented encoder network with two channels, where one is the Multi-Layer Perception (MLP) for texts and the other is the Graph Convolutional Network (GCN) for network links. To extract high-quality topic representations, the two channels work in a layer-assisted manner, where the MLP representation of each layer is aggregated with the corresponding GCN representation learned from network links. Decoding: We adopt the topic representations to reconstruct the texts and network links by maximizing the likelihood of the generative process of LANTM. The overall framework of LANTM is illustrated in Fig.1. We use NVI to efficiently solve LANTM. We evaluate LANTM on topic quality, text classification and link prediction. Experimental results validate that LANTM significantly outperforms the state-of-the-art baseline models. To sum up, our contributions are outlined below.

1. We propose a novel neural topic model LANTM for the text data with network links, and describe its generative process and model training with NVI in Sections 3.1 and 3.2.
2. We develop an augmented encoder network, which jointly encodes the text content and network links into topic representations, described in Section 3.3.
3. We evaluate LANTM on 5 benchmark datasets. Empirical results indicate that LANTM significantly outperforms existing models, shown in Section 4 .

2 RelatedWork

To our knowledge, the hierarchy of topic modeling falls into standard probabilistic models and neural topic models.

The underlying idea of standard probabilistic models supposes that the word tokens of texts are drawn from the latent distributions associated with topics. For example, the representative LDA model [Blei et al., 2003] assumes that each word token is drawn from a selected topic distribution over words, and the topic is previously drawn from the topic proportions of texts. During the past decades, many extensions of LDA have been widely developed as surveyed in [Blei, 2012; Boyd-Graber et al., 2017]. However, to be efficiently inferred by conventional methods [Jordan et al., 1999; Griffiths and Steyvers, 2004; Mimno et al., 2012; Li et al., 2016], most of these methods are defined as shallow models with only three or four layers, .

Beyond LDA-based models, neural topic models [Miao et al., 2017; Srivastava and Sutton, 2017; Zhang et al., 2018; Dieng et al., 2020; Burkhardt and Kramer, 2019; Liu et al., 2019; Isonuma et al., 2020; Zhu et al., 2020] are solved by the generic black-box NVI [Kingma and Welling, 2014; Mnih and Gregor, 2014; Rezende et al., 2014]. Therefore, they are more flexible, and allowed for more complex and deeper generative processes, growing the expressive capacity. To be specific, under the spirit of NVI, the variational distribution of latent topical variables is defined as a variational neural network that ingests texts and outputs latent topical variables. With the reparameterization technique [Kingma and Welling, 2014], the variational objective can be approximated by drawing Monte Carlo samples, and then optimized by gradient-based methods, regardless of the network architecture. Or they can be interpreted as VAE, where the variational distribution serves as an encoder network from texts to latent topical variables, and texts are reconstructed by drawing from latent distributions associated with topics as the defined generative process of the model.

Existing neural topic models mainly focus on the pure text data, however there are rare works motivated by handling the text data with network links [Bai et al., 2018; Zhang and Lauw, 2020]. The NRTM [Bai et al., 2018] extends the traditional RTM [Chang and Blei, 2009] by leveraging the stacked VAE. The model defines an encoder-decoder process for texts, and jointly trains a link prediction network by treating the concatenation of topic proportions as the input. Besides, the Adjacent-Encoder-X (AdjEnc-X) [Zhang and Lauw, 2020] directly treats the network links as the supplemental features of texts, and constructs a noisy autoencoder to learn topic proportions. Orthogonal to those models, LANTM treats the text content and links as bag-of-words and graph data, respectively, and encodes them by using an augmented network with graph convolutional modules. Therefore, LANTM enables to learn better topic representations that benefits from the joint learning of different types of data and the layer-assisted manner.

3 The LANTMModel

In this section, we introduce Layer-Assisted Neural Topic Model (LANTM) for modeling text data with network links. We describe the generative process of LANTM as well as model training with NVI, and then interpret LANTM from the perspective of VAE.

3.1 Model Definition

Basically, LANTM can be regarded as an extension of RTM [Chang and Blei, 2009], thus its model definition is mainly inherited from RTM. We now introduce the generative process of LANTM for the content of texts and the links simultaneously. More formally, given a corpus we suppose that there exist totally K topics {φk}Kk=1 , where each topic φk ∈ RV is represented by a multinomial distribution over the vocabulary. And each text is represented by a mixture of topics. To generate a text d, it draws a topic proportion θd from a logistic normal distribution LN(µ0, Σ0) formulated below:

where δd denotes the corresponding unnormalized topic proportion. 1 Each word token in text d is generated by first drawing a topic indicator zdn from θd and then drawing xdn from the selected topic φzdn . On the other hand, for each text pair {i, j}, it generates a link indicator πij drawn from a Bernoulli distribution parameterized by the cosine similarity between their corresponding topic proportions. The motivation coincides with the fact that the texts with similar topic proportions are more likely to share a link.

For clarity, we summarize the generative process of LANTM as follows:

1. For each text d ∈ [D]
2. Draw an unnormalized topic proportion δd ∼ N(µ0, Σ0)
3. Compute the topic proportion θd = softmax(δd)
4. For each word token xdn, n ∈ [Nd]
5. Draw a topic assignment zdn ∼ Cat(θd)
6. Draw a word xdn ∼Cat(φzdn )
7. For each text pair i, j ∈ [D]
8. Draw a link indicator πij ∼ Bernoulli(cos(θi, θj))

By revisiting the model definitions of RTM [Chang and Blei, 2009] and LANTM, we show that the major difference is the generative ways of the topic proportion θ per-text, which can be interpreted as the latent representation of text. As a neural topic model with NVI, in LANTM the topic proportion θ can be inferred by more flexible and powerful encoder network based on the observations of word tokens and links, beyond RTM that directly generates θ from a Dirichlet distribution. We will detail the model training process and network architecture in the following subsections.

3.2 Model Training

Given an observation of collection Ω = {X,Π} with texts X = {xd}Dd=1 and links Π = {πij}Di,j=1 , the objective of LANTM training is to estimate the latent variables of interest, including the unnormalized topic proportions of texts δ and topic distributions φ. We neglect the topic assignments z since it can be analytically integrated out. Commonly, the model training can be achieved by maximizing the following log marginal likelihood of X and Π:

The above formula is intractable to maximize, since it involves a difficult integral over the (unnormalized) topic proportions. Accordingly, we resort to approximate training by leveraging the amortized variational inference [Gershman and Goodman, 2014]. To be specific, we define the variational distribution that depends on both observations {xd, πd} and the variational parameter γ, formulated below:

1 Following [Dieng et al., 2020] we fix the Gaussian prior N(µ0, Σ0) as the standard Gaussian N(0, I).

For each δd, the corresponding variational distribution is a Gaussian whose mean µd and covariance Σd are the outputs of the encoder network parameterized by γ. In other words, the variational distribution can be interpreted as an encoder network that ingests {xd, πd} and outputs the mean and covariance of δd.

We apply this family of variational distribution, thus under the spirit of NVI we formulate the following lower bound of Eq.(2), i.e., a variational objective of LANTM with respect to the topic distributions φ and variational parameter γ:

The conditional distributions within the first two terms can be presented as follows:

p(πij |δi, δj ) =λπij (1 - λ)(1-πij ) (6) where λ = cos(θi, θj ) = cos(δi, δj ).

Unfortunately, the variational objective of Eq.(4) is still intractable to maximize since it involves the expectations with respect to the (unknown) target variational distribution. To this end, we form a Monte Carlo approximation to the variational objective as follows:

where S is the number of samples; and the samples are exactly generated by leveraging the reparameterization trick [Kingma and Welling, 2014]:

Besides, the KL-divergence regularization per-text has an analytic form, which can be expanded as follows:

Algorithm 1 Training process for LANTM

Input: X ∈ RD×V : bag-of-words matrix; Π ∈ RD×D : text

links matrix; K: number of topics. Output: topic proportions of texts θ, topic distributions φ 1: Initialize network parametersW(g) ,W, b and topic distributions φ normally; 2: for epoch = 1 to maxEpoch do 3: for n = 1 to N - 1 do 4: Calculate GCN layer Hn = ψ( ˜ΠHn-1W(g) n ) ; 5: Calculate MLP layer Zn = ψ(ˆZn-1Wn + bn); 6: Combine two layers: ˆZn = ξHn + (1 - ξ)Zn; 7: end for 8: Calculate mean and variance by Eq.(13); 9: Calculate topic proportions θ by Eq.(14); 10: Calculate the gradients of {φ, γ} by backpropagation; 11: Update {φ, γ} with Adam; 12: end for 13: Return θ and φ.

where Tr(·) and det(·) denote the trace and determinant of a matrix, respectively. Accordingly, our LANTM can be approximately inferred by forming the gradients of Eq.(7) with respect to {φ, γ}, and updating them with any adaptive learning rate method.

3.3 Interpreting LANTM as VAE

We can interpret LANTM from the perspective of VAE, since the NVI method is adopted for model training. Revisiting Eq.(4), the first two terms play the roles of the reconstruction errors of observations. In other words, the variational distribution serves as the encoder that encodes the texts and network links to (unnormalized) topic proportions of texts {δ, θ}, and LANTM reconstructs the observations by maximizing their log marginal likelihood of the generative formulation given {δ, θ}. The overall framework of LANTM is illustrated in Fig.1. We now describe the encoder architecture in more detail.

Encoder architecture. We declare that X ∈ RD×V and Π ∈ RD×D denote the bag-of-words matrix and text link matrix, respectively. Let N denotes the number of layers in the encoder network. For the former N-1 layers, each of the n-th layer representations of the channels of GCN and MLP are described as follows:

Inspired by [Bo et al., 2020], we employ the layer-wise assistance, which combines each Zn and Hn with a combining coefficient ξ and sets the combined latent representation ˆZn as the input of the next MLP layer:

Finally, the last layer outputs the mean and covariance of δ as follows:

3.4 Training and Inference Summary

Model training. Following [Kipf and Welling, 2016], we set three layers for both channels of MLP and GCN. The sub-gradients of the encoder networks are computed by backpropagation, the Adam [Kingma and Ba, 2015] is applied for adaptively setting the learning rate. The overall training details of LANTM are summarized in Algorithm 1.

Inference for future texts. We directly use the fitted encoder network to compute the topic proportions of future texts. In practice, the network links of future texts may be unknown. To resolve this, following [Zhang and Lauw, 2020] we reuse the training texts as auxiliary data. Specifically, we apply the known text link matrix between training texts and future ones or construct the k-nearest neighbor matrix between them instead. We feed all texts and the auxiliary link matrix into the fitted encoder network to compute the topic proportions, and leave the ones of future texts only.

4 Experiment

Datasets. In the experiments, we apply the dataset of Cora 2 consisting of paper abstracts and citations [McCallum et al., 2000], and Reuters 3 (R8) without any links. The Cora dataset is divided into four subsets, namely Data Structure (DS), Hardware and Architecture (HA), Machine Learning (ML) and Programming Language (PL). We apply Cora processed by [Zhang and Lauw, 2020]. For R8, the standard stop words and infrequent words occurring in less than 5 documents are filtered out, and a k-nearset neighbor graph is constructed (k = 10). Statistics of datasets are presented in Table 1.

Comparing models. We totally select 7 existing baseline models for comparison, including 3 standard neural topic models, NVDM 4 [Miao et al., 2016], ProdLDA 5 [Srivastava and Sutton, 2017], ETM 6 [Dieng et al., 2020] and 4 methods handling data with network links, RTM 7 [Chang and Blei, 2009], NRTM 8 [Bai et al., 2018], VGAE 9 [Kipf andWelling, 2016], AdjEnc-X 10 [Zhang and Lauw, 2020].

For our LANTM, the combining coefficient ξ is tuned over {0.1, 0.2, . . . , 0.9}. For all baseline models, the default parameters are adopted. All methods are trained under same num of epochs and the topic numbers are set as {25, 50} for all datasets.

2 http://people.cs.umass.edu/mccallum/data/cora-classify.tar.gz

3 https://martin-thoma.com/nlp-reuters/

4 https://github.com/ysmiao/nvdm

5 https://github.com/akashgit/autoencoding vi for topic models

6 https://github.com/adjidieng/ETM

7 http://cran.r-project.org/web/packages/lda/

8 https://github.com/zbchern/Neural-Relational-Topic-Models

9 https://github.com/tkipf/gae

10 https://github.com/PreferredAI/adjacent-encoder

4.1 Evaluation of Topic Coherence

Topic Coherence (TC) is a popular metric to measure the topic quality by calculating co-occurrences of top-k topical words over an external corpus. In the experiment, we employ the public TC project Palmetto 11 [R¨oder et al., 2015], and the setting of CV is applied. We present the TC scores of top-10 topical words in Table 2.

We can observe that our LANTM outperforms baseline models in most settings, directly indicating LANTM can generate more effectively coherent topical words. For example, our LANTM is about 0.02 higher than baselines on DS when K = 50. The previous literature [Bai et al., 2018] reports that a shallower decoder limits topic quality performance, and we exactly observe that the baseline models ProdLDA and NVDM with shallower decoders perform worse than NRTM with deeper decoder. We kindly notice that LANTM is also with one-layer decoder, but it significantly performs better than NRTM. Therefore the results provide strong evidence to the effectiveness of layer-assisted encoder for generating coherent topics.

4.2 Evaluation of Classification

We compare LANTM against baseline models by classification. For each comparing model, the learned topic proportions of texts are used to train the SVMs classifier. 12 In both transductive and inductive settings, we conduct 5-fold cross-validation experiments, and report the average scores of Micro-F1 and Macro-F1 in Table 3.

11 https://github.com/dice-group/Palmetto/wiki/Coherences

12 http://scikit-learn.org/

Overall, we find that LANTM outperforms the baseline models, and achieves significant improvement in many cases. For example, the performance gain is about 0.22 and 0.28 on PL and R8, respectively. The results directly indicate the great advantage of LANTM on learning discriminative latent topic representations. More importantly, it can be seen that LANTM also beats baseline models in the inductive setting, empirically suggesting that LANTM enables to effectively fit future text data.

4.3 Evaluation of Link Prediction

Link prediction measures prediction capacity for unseen links. For each link πij, we predict it by using the corresponding topic proportions {θi, θj} to estimate the probability P(πij = 1|θi, θj) ∝ exp(-‖θi - θj‖2 ). In the transductive setting, following [Kipf and Welling, 2016; Zhang and Lauw, 2020] we randomly remove one link for the texts with more than 3 links, and predict the removed ones. In the inductive setting, we randomly select 80% texts and the corresponding links for training, and directly predict all links of the remaining 20% texts. We evaluate the results by Area Under the ROC Curve (AUC) and Average Precision (AP), computed by referring to [Kipf and Welling, 2016]. We report the average scores of 5 independent runs in Table 4.

In this evaluation, our LANTM consistently outperforms baseline models in the transductive setting, and ranks the first in most cases of inductive setting. In terms of ML, it achieves the highest improvements, i.e., about 0.160 and 0.147 on AUC and AP when K = 25. Besides, we observe that the link-based models RTM, NRTM, and AdjEnc-X almost perform better than the traditional models NVDM, ProdLDA, and ETM. This demonstrates the positive effect of network links for model fitting.

4.4 Parameter Evaluation

In this section, we evaluate the impacts of topic number K and combining coefficient ξ by TC and Macro-F1 scores in the transductive setting and plot results in Fig.2.

For the topic number K, we vary it from the set of {25, 50, 75, 100, 125}. We find that there exists a slight rising trend for TC on most datasets and best results mostly lies in K = 75. As for Macro-F1, our LANTM shows insensitivity on topic number which makes it practical in real applications.

For combining coefficient ξ, we vary it from an increasing set {0.1, 0.2, . . . , 0.9}. We find that LANTM has the best TC when ξ = 0.1 on most datasets. The reason may goes to the MLP module for document context matrix plays more important roles on extracting coherent topic representations. Meanwhile Macro-F1 shows rising trends and best scores achieve at ξ = 0.6 and 0.8 for most cases. This indicates that the GCN module may contribute to learning more discriminative latent topic proportions.

5 Conclusion

In this paper, we propose a novel encoder-augmented topic model LANTM for combining normal bag-of-words text data with ubiquitous network links to jointly learn latent topic representations. MLP and GCN are applied on these data in diverse structures and we employ layer-wise augmentation for combining each layer. Text data and network links are reconstructed separately according to different structures. Empirical studies on three commonly acknowledged metrics demonstrate our significant improvement against existing representative methods.

Acknowledgements

The work is supported by the National Natural Science Foundation of China (NSFC) (No.61876071) and Scientific and Technological Developing Scheme of Jilin Province (No.20180201003SF, No.20190701031GH) and Energy Administration of Jilin Province (No.3D516L921421).