Gaussian Embedding of Linked Documents from a Pretrained Semantic Space

1 Introduction

Linked documents are everywhere, from web pages to bibliographic networks (e.g., scientific articles with citations) and social networks (e.g., tweets in a Follower-Followee network). The corpus structure provides rich additional semantic information. For example, in Scientific articles, page limitation often leads to short explanations that become clear if we read the linked papers (i.e., citations).

Many recent approaches propose to use low-dimensional document representation as a proxy for solving downstream tasks [Yang et al., 2015]. These methods learn the representations using both textual and network information. They have many advantages: it accelerates the computation of similarities between documents and it drastically reduces the storage space needed. Moreover, they can significantly improve accuracy in information retrieval tasks, such as document classification [Yang et al., 2015] and link prediction [Bojchevski and G¨unnemann, 2018]. TADW is the first approach that embeds linked documents [Yang et al., 2015]. Subsequent methods are mainly based on matrix factorization [Brochier et al., 2019; Huang et al., 2017] and deep architectures [Liu et al., 2018; Tu et al., 2017; Kipf and Welling, 2016].

∗Contact Author

Most of those techniques learn documents as points in the embedding space. However, considering a measure of dispersion around those vectors brings useful information, as shown on corpus with no link between documents [Nikolentzos et al., 2017]. In Graph2Gauss [Bojchevski and G¨unnemann, 2018], each document is associated with a measure of uncertainty along with its vector representation. However, the objective function optimizes the uncertainty using the network information and it does not model the dispersion at the word level. Additionnaly, variational methods such as [Kipf and Welling, 2016; Meng et al., 2019] introduce gaussian posteriors, but the generative process uses the dot product between documents' mean only to model the adjacency and attribute matrix entries. Hence it is not clear what the uncertainty obtained from the variational variance captures.

Finally, none of earlier methods represents both documents and words in the same semantic space, as opposed to textbased methods such as [Le and Mikolov, 2014]. LDE [Wang et al., 2016] and RLE [Gourru et al., 2020] both build a joint space for embedding words and linked documents. However, these approaches do not take the uncertainty into account.

In this paper, we propose an original model that learns both a vector representation and a vector of uncertainty for each document, named GELD for Gaussian Embedding of Linked Documents. The uncertainty reveals both network and text variance. It will be higher if the document cites very different document sets and if it uses semantically distant words. In addition, documents and words lie in the same space: one can compute similarities between documents and words in this semantic space, enhance queries or describe document clusters by using close words.

After a review of related works in Section 2, we present our model in Section 3. We show that our representations outperform or match most of the recent methods in classification and link prediction on three datasets (two citation networks and a corpus of news articles) in Section 4. Additionally, we provide semantic insights on how to use the variance, and the shared latent space. We conclude and propose extensions of GELD in Section 5.

2 RelatedWorks

In this section, we present recent approaches for embedding documents organized in a network.

2.1 Document Embedding

Since Word2vec models from [Mikolov et al., 2013], representation learning for text has focused attention as it can improve many downstream tasks. Document embedding followed: many methods propose to represent documents as vectors. For example, [Le and Mikolov, 2014] extends the word2vec formulation. More precisely, the doc2vec models represent documents and words in the same space. One can therefore compute similarities between words and documents.

In many real-life problems, documents are organized as a network: the corpus forms an attributed network. Documents are nodes, citations are edges and the textual contents of the documents are the attributes. In the next section, we present several methods that take this network information into account when learning document representations.

2.2 Document Network Embedding

TADW is the first approach that embeds linked documents [Yang et al., 2015]. It extends DeepWalk [Perozzi et al., 2014], originally developed for network embedding, by formulating the problem as a matrix tri-factorization that includes the textual information. Subsequently, authors of GVNR-t [Brochier et al., 2019] propose to extend Glove [Pennington et al., 2014] in a similar way. AANE [Huang et al., 2017] applies Laplacian Eigenmap to attributed network using Alternating Direction Method of Multipliers to accelerate the learning phase. Recent works mainly use deep architectures: STNE [Liu et al., 2018] adapts the seq2seq architecture, CANE [Tu et al., 2017] uses an attention mechanism.

These methods do not learn representations for both documents and words, as opposed to LDE [Wang et al., 2016] and RLE [Gourru et al., 2020]. LDE uses an extended formulation of [Mikolov et al., 2013]. Nevertheless, it requires labels associated with nodes, which makes it a supervised approach. RLE [Gourru et al., 2020] uses both text and network to build a vector that projects documents onto a pretrained word embedding space.

All these methods learn a single vector by document. This assumption is limited as documents, especially long ones, might be semantically rich. To this end, several methods propose to learn a vector of uncertainty associated to the vector representation.

2.3 Gaussian Document Embedding

Variational methods such as VGAE [Kipf and Welling, 2016] and CAN [Meng et al., 2019] approximate the posterior of the latent variables (the document embedding) by gaussians. Nevertheless, the generative process uses the dot product between posterior means (the latent variables) to model the adjacency and attribute matrix entries. Hence, the variational variance is not an explicit measure of the document semantic and neighborhood uncertainty. The only method that explicitly model uncertainty is Graph2Gauss [Bojchevski and G¨unnemann, 2018]. A feed-forward neural network embeds the textual content and maps it to a mean and a variance, following an optimization process based on energy-based learning. The negative Kullback-Leibler divergence used as energy allows them to model the proximity between nodes. It should be higher between connected nodes than unconnected nodes. Therefore, Graph2Gauss does not explicitly model the semantic uncertainty of a document.

Another modeling assumption allows to learn documents as Gaussian Distributions. Documents are regarded as bag of word embeddings (see Figure 2), i.e. a multiset of vectors in the latent semantic space. Starting from this assumption, [Das et al., 2015] proposes a new way to extract topics from document sets. Topics are Gaussian distributions that generate word vectors in a semantic space, and therefore a bag of word embeddings for each document. In [Nikolentzos

Figure 2: The bag of word embeddings of a document.

et al., 2017], authors propose to model documents as Gaussians that generate bag of word embeddings. With pretrained word embeddings, they show that the optimal solutions for the means and variances are the empirical means and empirical variances of the vector representations of documents' words. Unfortunately, this solution does not hold with linked documents.

None of the above can, in the same time: i) represent documents and words in the same space, ii) learn a measure of uncertainty associated to the document embedding. We therefore propose a novel method, GELD, that has these properties.

3 GELD: Gaussian Embedding of Linked Documents

3.1 Data and Notations

We consider a corpus of n linked documents and a vocabulary of size v. We also consider a fixed representation in Rr for each word of the vocabulary. We write uk ∈ Rr the vector representation of the k-th word wk of the vocabulary, uk,r its r-th element. We note f the function that maps the word wk to its pretrained representation f(wk) = uk ∈ Rr . The user can either learn the word embeddings on the studied corpus with most recent methods [Devlin et al., 2019; Mikolov et al., 2013] or use a pretrained set of word embeddings built on a broader corpus 1 to reduce the computation time.

Each document di is then associated with the bag of word embeddings Dw i = {f(wi,1 ), f(wi,2 ), . . .}. In our notations, wi,1 is the first word used in the document i.

Our aim is to learn document representations as Gaussian distributions. Each document has two parameters: a mean µi in Rr and a diagonal variance σ2 i I, with σ2 i ∈ Rr , that reveals the document uncertainty. We can use the document mean as vector representation when it is needed, e.g., in classification tasks. We note g the function that maps the document di to its mean g(di) = µi ∈ Rr . Using the network information, we therefore have, for each document, a bag of document embeddings cited by the document di, noted Dli = {g(di,1 ), g(di,2 ), . . .}, where di,1 is the first document cited by the document i.

We introduce two notations: ci,k the number of times the ith document di uses the k-th word of the vocabulary and ai,l the number of times it cites the l-th document of the corpus.

By taking the union of these multisets, the corpus becomes n bags of vectors Di = Dw i ∪Dli :

1 e.g., https://fasttext.cc/

3.2 Model and Optimization

Similarly to [Das et al., 2015; Nikolentzos et al., 2017], we posit that the vectors in Di are independently drawn from an isotropic Gaussian distribution, meaning that

.

The Gaussian is parametrized by a mean and a diagonal variance characterizing the document embedding and its uncertainty. The parameters to learn for each document are σ2 i ∈ Rr and µi ∈ Rr .

The log-likelihood of the proposed model is, with |Di| the cardinality of Di:

where D = {Di}ni=1 , σ2 = {σ2 i }ni=1 and µ = {µi}ni=1 .

We split this log-likelihood between the drawing regarding words (Lw) and documents (Ld).

As citation and term frequencies are on different scales, we can observe imbalanced Lw and Ld. This problem is frequent when modeling heterogeneous data using the same generative process [Wang, 2001]. We therefore optimize an alternative weighted likelihood. By defining η ∈ [0, 1] denoting the importance given to the network information, we write the alternative function to optimize:

Computing and annealing the gradient, we get the optimal solutions µ∗i and (σ2 i,r )∗ :

Algorithm 1 GELD Algorithm

• Input: D, U Parameters: η, λ, k Output: µ, σ2 1: for each: document i do 2: initialize µ0i and (σ2 i )0 according to Equation 9 3: end for 4: k=1 5: repeat 6: for each: document i do 7: compute µ∗(k) i and (σ2 i )∗(k) using Eq. 6 and 7 8: update µ(k) i and (σ2 i )(k) with Equation 8 9: end for 10: k = k +1 11: until convergence 12: return µ, σ2

dataset, we filter the vocabulary by withdrawing stop words with the scikit-learn package 2 and we remove common and rare words (i.e., words appearing less than 4 times and in more than 25% of the documents). We obtain vocabulary size of 4,390 on Cora, 3,763 on Dblp and 6,407 for the Nyt dataset. Our method learns a mean and a variance for each document. In many real-world scenarios, downstream tasks require a single vector representation for a document. We therefore evaluate the relevance of using documents' mean in standard evaluation tasks: classification in Section 4.3 and link prediction in Section 4.4. We also provide qualitative insights on the variance possible use in Section 4.5. Besides, we also demonstrate the benefit of embedding documents and words in the same space.

4.2 Parameters Tuning

We compare our approach to recent baselines. We use four matrix factorization-based approaches: TADW, AANE, GVNR-t and RLE, and three deep neural network models: VGAE, Graph2Gauss and STNE. We also compare our method to DeepWalk, that considers the network information

2 https://scikit-learn.org/

We maximize ˜ L in each parameter and repeat the process until convergence. The optimization is iterative, and each parameter update depends on current values of the other parameters. Because of this dependency, we propose to adopt the Robbins-Monro method [Robbins and Monro, 1951] to prevent going too fast to a mediocre local solution. It yields good results in our experiments. The updated value of parameter µi at epoch k, given the optimal values at epoch k computed using Equation 6 we note µ∗(k) i , is:

We update σ2 i similarly. We propose to use λ(k) = (δk)-γ , γ ≤ 1 as done in [Barkan, 2017] to ensure convergence conditions. δ ∈ [0, 1] is the importance given to the optimal solution at the beginning of the optimization. In our experiments, we obtain higher results with low values of δ. In other words, we do not trust the first optimal solutions as the optimization is iterative.

The initial means and variances, noted by µ0i and (σ2 i )0 , are the empirical means and variances computed on the bag of word embedding only:

4 Experiments

4.1 Datasets and Evaluation Tasks

We experiment on three datasets. Cora [Tu et al., 2017] and Dblp [Tang et al., 2008; Pan et al., 2016] are two citation networks. Cora contains 2,211 abstracts of tagged scientific documents (7 classes) with 5,001 edges. Dblp has 60,744 documents titles (4 classes) and 52,914 edges between them. Additionally, we use the Nyt dataset from [Gourru et al., 2020] containing press articles from January 2007. It has 4 classes, 5,135 documents and 3,050,513 edges. For each only, LSA [Deerwester et al., 1990], that embeds the document with regard to the textual information, and a concatenation of embeddings obtained with those two methods we call "Concatenation" as done by [Yang et al., 2015]. We use all the original implementations provided by the authors.

We use hyper parameters recommended by the authors, for those who use similar datasets, and grid-search hyperparameters using the document classification task otherwise. For TADW, we use λ = 0.2 [Yang et al., 2015] and dimension 200 for the reduced representation of documents' content. For AANE, we use optimal λ and ρ obtained via gridsearch, as xmin for GVNR-t. For RLE we use λ = 0.7 and build the word vectors as specified by the authors. For VGAE, we use the author architecture, and K = 1 for Graph2Gauss. VGAE could not handle DBLP on our machine in reasonable time. For STNE, we determine depth using grid-search. For DeepWalk, we perform 40 walks of length 40 by nodes, and we set the window size to 10. We run all the experiments in parallel with 20 physical cores (Intel R© Xeon R© CPU E5-2640 v4 @ 2.40GHz) and 96GB of RAM. We use r = 160 as embedding dimension for every method following [Yang et al., 2015].

Similarly, we report the optimal parameters for GELD obtained via grid-search on the classification task: δ = 0.1, γ = 0.2, η = 0.99 for Cora, η = 0.8 for Dblp and η = 0.95 for Nyt. To learn word vectors, we adopt Skip-gram with negative sampling [Mikolov et al., 2013] implemented in gensim 3 . We use window size of 15 for Cora, 10 for Nyt, 5 for DBLP (depending on documents size), and 5 negative examples for both. It only takes 46 seconds on Cora, 84 on DBLP and 42 on Nyt.

4.3 Classification Results

We adopt standard evaluation tasks following similar works [Yang et al., 2015; Bojchevski and G¨unnemann, 2018]. We perform classification with a SVM classifier with L2 regularization. The optimal regularization is fixed, for each method and dataset, using grid search. We run the algorithms 10 times and report the mean Micro-F1 and standard deviation in Table 1.

Our method outperforms every competitor on each dataset (Table 1). To the exception of VGAE, linked document methods demonstrate higher accuracy than Deepwalk and LSA on Cora but they fail to outperform the Combination on Dblp and Nyt. GELD performs consistently better, possibly due to the impact of the variance during the learning phase: by inspecting Equation 6, we can see that the optimal value gives less weight to documents with high variance (i.e. uncertain or too general documents). Interestingly, with optimal L2 regularization, TADW yields better results than more recent baselines on every dataset with 50% train/test ratio. Figure 3 presents results with 50% train/test ratio with different dimensions for the four best models. GELD performs better in each dimension. TADW outperforms RLE in dimension 100, but adding dimension seems to deteriorate the results until convergence.

3 https://radimrehurek.com/gensim/

4.4 Link Prediction Results

For the link prediction task, we hide a random set of edges to learn the representations. Then, a random set of unconnected pairs of documents is drawn as negative examples. We compute cosine similarity between pairs of documents in the hidden edge set and the negative example set. We then report Area Under the Curve (AUC) in Table 2, obtained with 3 runs and different percentages of hidden edges.

GELD outperforms baselines on Cora and Dblp, but is beaten by TADW on Nyt (Table 2). Nevertheless, every method except LSA, AANE and GVNR-t obtains AUC between 88.2 and 88.5, for %25 and %50 of edges hidden. This is due to the nature of the network: mean degree is around 500, i.e. 11% of the network. Even with 50% of edge hidden, the network information is well represented. Furthermore, TADW fails to produce good representations for link prediction on Dblp which is less dense while GELD performs constantly for different network topologies.

4.5 Qualitative Insights

As stated earlier, GELD represents words and documents in the same space. To demonstrate the interest of this property, we compute, for each annotated class of Cora, the average vector of document means µi inside this class. We present the five closest words to these class centroids in Table 3. It is easy to grasp the class content by looking at these descriptors. For example, Class 3 contains documents on Bayesian models and Class 1 on neural networks.

In Table 4, we present the six closest documents to "Latent Dirichlet Allocation", along with their variance (the sum of each axis variance). The papers "A perceptual hashing algorithm using latent Dirichlet allocation" and "Spatial Latent Dirichlet Allocation" both apply LDA to images. Therefore, they have a greater variance as they must cite papers from different areas. Meanwhile, "Latent Dirichlet Co-clustering" is in the same class than LDA, and it is more likely to cite documents from this class. Interestingly, papers with lower variance are all in the same class than "Latent Dirichlet Allocation" (Class 3).

5 Conclusion

We presented Gaussian Embedding of Linked Documents (GELD), a generative model that represents a document as a Gaussian Distribution in a word vector space. Learned through maximum likelihood estimation, it outperforms existing methods on Cora and Dblp, and it matches baselines on a New York Times dataset. Adding the variance during the learning phase seems to provide better vector representations since it gives less importance to documents with high variance when updating parameters. In further studies, we will focus on: 1) integrating the variance in downstream tasks, 2) developing a fully Bayesian version of our model to add priors on mean and variance.

Acknowledgments

We thank Adrien Guille for his support, and for helping to write the code used in our experiments.