Deep Context: A Neural Language Model for Large-scale Networked Documents

1 Introduction

Text mining applications use quantitative representations of documents to analyze and compare them to one another. One popular approach to text modeling represents each document as a vector of its word frequencies. Due to its conceptual simplicity and computational efficiency, this approach is used widely in information retrieval, text summarization, and personalized recommendation. However, representing documents by their word frequencies has significant disadvantages that limit the utility of this representation. The principal of these is that word frequencies fail to capture word meaning. Individual words may be highly ambiguous: the same word can often mean different things, and different words frequently have the same meaning. To address this challenge, modern text analysis methods take advantage of language models (n-grams or topic modeling algorithms) which include the context of words in document representations, where a word's context is provided by the neighboring words, phrases in sentences and co-occurred words in same documents. Then they use quantitative methods to find statistical dependencies among words across documents. The intuition behind this approach is that surrounding words in a document, though themselves ambiguous, collectively help to pin down a given word's meaning.

However, words often have a deeper context that extends beyond nearby words, phrases, and sentences in the same document to other relevant documents and concepts. For example, the online encyclopedia Wikipedia is composed of a network of Web pages describing interconnected concepts

Kristina Lerman

USC ISI lerman@isi.edu

and entities. Figure 1 illustrates a small portion of Wikipedia related to the concept "cat". The text of the page describing "cat" references other pages describing concepts "felidae", "mammal" and "vermin". Similarly, the Wikipedia page describing the concept "dog" links to related pages describing "wolf", "pet", "carnivora". In order to get a complete picture of what "cat" and "dog" are, one has to read the descriptions in the linked pages. Similarly, in order to understand a scientific article, a reader must rely not only on the text of the article, but also on the background knowledge and supporting evidence that is described in other articles. Thus, the meaning the reader perceives is not simply derived from the words and sentences appearing in the article, but is the inferential result of the connections the article makes to other articles and the concepts expressed in them. These connections, and the text used by them to express concepts and themes of the documents, provide a deeper context for understanding the current document. Automatically learning these deep contexts from data will help us create better models of text documents that not only help to better solve existing tasks, such as finding documents similar to the given document, but also address novel text mining tasks that existing tools cannot solve.

In this paper, we address the problem of learning the deep contexts of documents with neural language models [Bengio et al., 2003]. We focus on leveraging information in document text and the links that exist between documents in document networks. We describe a model that captures the generative process of document creation in which language and semantics are intricately linked across document networks. Authors read existing documents to draw inspiration for the vocabulary, grammar, and style and learn how to describe the concepts and ideas in their own works. Language influences propagate between documents through citations and crossreferences, providing a deeper context for understanding the semantics of text. The question is how to capture the hierarchy of contexts of a word in a sentence, including the surrounding words, the sentence semantics, underlying theme of the article and influence from cited articles. To this end, we devise a new language model that accounts for these contexts in text documents.

2 RelatedWork

We begin with a review of two lines of related work.

2.1 Neural Language Models

Neural language models are based on the idea of distributed word representations [Bengio et al., 2003]. Instead of "onehot" representations for words in vocabulary, neural language models use continuous variables to represent words in vector space, in the hope of improving the generalization of classic n-gram language models. This idea of distributed word representations has been successfully applied to many tasks in natural language processing and data mining, such as modeling unigram words and phrases [Mikolov et al., 2013], sentences and documents [Le and Mikolov, 2014], relational entities [Bordes et al., 2013; Socher et al., 2013], general text-based attributes [Kiros et al., 2014], streaming documents [Djuric et al., 2015] and semi-supervised learning of text embedding [Tang et al., 2015a]. Representative applications of neural language models also include learning representations of nodes in networks [Perozzi et al., 2014; Tang et al., 2015b; Chang et al., 2015; Grover and Leskovec, 2016; Wang et al., 2016].

2.2 Semantic and Link Analysis

Modeling underlying structure of text documents and learning semantic representations is critical to various applications including information retrieval, text summarization and personalized recommendation. Approaches of learning latent semantics (notably "topic modeling" methods) include probabilistic Latent Semantic Analysis (PLSA [Hofmann, 1999]) and Latent Dirichlet Allocation (LDA [Blei et al., 2003]) are proposed. However, most of these models do not account for data with the link structure where text documents lie on. Research work in link analysis [Hoff et al., 2002; Kemp et al., 2004; Airoldi et al., 2009] attempt to model network structure in latent space. However, these approaches cannot be easily applied to node with text content. The goal of our model is closest to recent work in joint content and link analysis. Several models [Mccallum et al., 2005; Dietz et al., 2007; Nallapati et al., 2008; Mei et al., 2008;

Chang and Blei, 2009] have been proposed based on topic modeling, and links between documents are explained by the similarity of topic mixtures. However, our model goes beyond the bag-of-words representations where these models bear in common, and exploit word order information in sentences.

3 Deep Context Neural Language Model

In this section, we describe our model in detail. To begin with, we review the log-bilinear language model [Mnih and Hinton, 2007], which is the foundation of our model.

3.1 The Log-Bilinear Model

We represent each word w using an F-dimensional realvalued feature vector, vw ∈ RF . The log-bilinear model (LBL [Mnih and Hinton, 2007; Mnih and Kavukcuoglu, 2013]) specifies the bilinear energy function of a sequence of context words (w1, · · · ,wn-1) and the predicted next word wn:

where ˆv is the predicted representation of next word, defined as

where ⊙ denotes element-wise multiplication, and ci is the weight vector for the context word in position i. For symmetry, we use the same set of word representations for both the words being predicted and the context words.

The resulting probabilistic distribution of next word is given by a softmax function:

where Zc = ∑Kwn=1 exp[-E(w1:n-1; wn)] is the context dependent normalization factor, and K is the vocabulary size.

3.2 Model Architecture

Our model assumes that the generation of the next word in a word sequence depends not only on the preceding words, but also on the global context of the document and the other documents it references (and possibly the documents these references cite, and so forth). Figure 2 illustrates a general architecture, where we use the word sequence example "dogs are our best · · · " in the document related to the concept "DOG". The concepts "PET" and "CAT" are also used to predict the next word "friends", as they are neighbors of "DOG" in the document network. More precisely, we firstly consider there is global semantic context for word sequences in each document d, which we also represent as a real-valued feature vector, vd. This idea is same as in [Le and Mikolov, 2014]. We also consider the semantic influence from the neighborhood of d in the document network (i.e., documents that can be reached from d in a few hops), and jointly model text in documents and their link structure. For simplicity, we learn word and document representations in the same Fdimensional vector space, and consider vw ∈ RF , vd ∈ RF .

Given a sequence of words (w1, · · · ,wn) in a document d1, we define the energy function as:

where N m d1 is the set of neighbors of d1 that can be reached in ≤ m hops along the links in the document network. We take ˆv to be the predicted representation of the next word:

where dp ∈ N m d1 , and c(d) p ∈ RF is the context parameters defining the weights of dp. We may reuse c(d) p for any dp that can be reached in the same number of hops. The probabilistic distribution of the next word is defined as

By learning next word's representation using features from its global document vector and the linked documents, our model captures the deep context for generating each word in a document. In this sense, we call the vectors learned by our model Deep Context Vectors (DCV). However, the model in Eq. ( 6) is computationally expensive due to potentially exponentially large number of documents within m hops, N m d1 . To reduce computation, we introduce a practical algorithm that draws node sequences to characterize the neighborhoods.

Randomized Document Sequence

Given a sequence of words (w1, · · · ,wn) within a document d1, we draw an associated sequence of documents (d1, · · · , dm+1) from N m d1 by following the directed links from document d1. The document sequence can be sampled based on a probability distribution P(d1:m+1) = ∏m+1 j=1 P(dj|d1:j-1). For simplicity, we adopt first order random walk scheme. In the first step, we start from the original document d1. In each following step, we uniformly sample dj+1 from the set of documents that the current document dj links to. Although higher order random walk is more powerful, we leave studying this direction as future work.

Figure 3: An alternative model architecture. We refer to it as DCVivLBL as it performs inverse language modeling.

We hence redefine the energy function in Eq. (4) as:

where ˆv is the predicted representation of the next word, and is learned from the feature vectors of preceding words as well as associated documents:

Here c(d) j ∈ RF is hop-dependent and specifying the weights of the feature vector of j-th document in the document sequence.

The conditional word distribution is given by

3.3 Learning with Negative Sampling

The word and document vectors are initialized with random values and trained by minimizing the negative log-likelihood of the data. The parameter updates are performed using stochastic gradient descent, and can be derived from Eq. (9):

where θ = {vw, vd, c(w) , c(d) } is the set of parameters to be learned, and ϵ is the learning rate. The computation of gradients obtained in Eq. (10) involves the normalization term and is expensive since the complexity is proportional to the vocabulary size K. In this paper, we use a scalable optimization algorithm without calculation of the gradients of normalization, that is negative sampling [Mikolov et al., 2013]. Negative sampling trains a logistic regression to distinguish between data samples of wn from "noise". In our model, the objective is to maximize

where σ(x) is the sigmoid function. Pn(w) is the global unigram distribution of the training data acting as the noise distribution where we draw k negative samples.

3.4 Improving with Alternative Architectures

We explore different model architectures, in the hope of improving the performance with ensemble of feature vectors learned with them separately. We present an alternative architecture which uses similar scheme of Skip-gram [Mikolov et al., 2013]. Figure 3 shows the architecture, which utilizes the current word to predict its word context (including preceding and following words), along with a weighted vector learned from the document sequence and also used to predict the word context. As it performs inverse language modeling, we refer to it as DCV-ivLBL and it's a counterpart of inverse vector LBL [Mnih and Kavukcuoglu, 2013]. The objective is to maximize the log-likelihood for a given word sequence (wt-b, · · · ,wt, · · · ,wt+b)

where b is the context size of words. The current word wt and d1:m+1 are used separately to predict the context words. The probability distributions is formulated as

P (wt+i|d1:m+1)

where Z w c and Z d c are the normalization term.

4 Experiment

In this section, we present evaluation results of our model. We begin with a description of our data sets.

4.1 Data Description

We select data sets from different domains including Wikipedia pages, scientific papers and legal opinions:

• Wikipedia: A dump of Wikipedia pages 1 in October 2015 is used in our experiments. Wikipedia data provides rich text with interlinked knowledge concepts. The document network is dense, with around 40 out-links per document on average.
• DBLP: We download the DBLP data set [Tang et al., 2008] 2 , which contains a collection of papers with titles and citation links. This represents a type of documents with extremely short text, with only about 9 words per document.

1 http://dumps.wikimedia.org/enwiki/latest/ enwiki-latest-pages-articles.xml.bz2

2 http://arnetminer.org/lab-datasets/ citation/DBLP_citation_2014_May.zip

• Legal: We collect a large digitized record of federal court opinions from the CourtListener 3 project in our study. The text in each case of legal opinion is mainly about the discourse of a legal case. The citation network is sparse and the lengths of documents are much larger than that of Wikipedia and DBLP, with more than 2 thousand words on average.

The statistics of the three datasets are shown in Table 1.

4.2 Methods for Comparison

We compare our model with the following baselines and the state-of-art approaches for learning document representations:

• BOW. We use bag-of-words representations as baseline, which is simply the frequency of unigrams in a document. We report performance on TFIDF weighted BoW.
• LDA [Blei et al., 2003; Hoffman et al., 2010]: Latent Dirichlet Allocation is used to learn document-specific topic distributions. By comparing our models with LDA, we may understand how important it is to modeling word order and the context of linked documents.
• Skip-gram [Mikolov et al., 2013]: We learn word vectors using Skip-gram model. Each document is represented by averaging vectors of all words in it.
• PV [Le and Mikolov, 2014]: Paragraph Vector (PV) learns distributed representations for variable length pieces of text, such as sentences, paragraph and documents. By comparing our model with PV, we may see how informative the links are.
• DeepWalk [Perozzi et al., 2014]: DeepWalk is used for learning distributed representations of nodes in a network. In order to understand how informative the word content are, we consider the word-word co-occurrence network Gww and word-document network Gwd. In addition, we also use document-document network Gdd to understand the effects of links.
• LINE [Tang et al., 2015b]: LINE models first-order and second-order proximity between nodes. LINE is optimized with edge-sampling and trained on wordword network Gww, word-document network Gwd and document-document network Gdd.

We perform experiments on a single machine with 64 CPU cores at 2.3 GHz, and 256G memory. Asynchronous stochastic gradient descent algorithm is used with 40 threads to optimize our models. Models are trained for 20 epochs on each dataset.

3 https://www.courtlistener.com/

4.3 Document Classification

In this experiment, we perform document classification and evaluate how discriminative the learned feature vectors are.

Experiment Setup

For each dataset, all text, links of all documents or the combination (listed in Table 1) are used for learning document representations in unsupervised manner. For evaluation, we sample a subset of documents with balanced category distribution for each dataset. (1) For Wikipedia dataset, we randomly select 10,000 samples from each category among the seven diverse ones "Arts", "History", "Human", "Mathematics", "Nature", "Technology", and "Sports"; (2) For DBLP papers, we select five research fields including "Artificial Intelligent", "Computer Graphics", "Computer Networks", "High Performance Computing", "Theory" as class labels. For each field, we randomly sample 10,000 papers; (3) In legal opinions, legal code are cited to support the judgement of a case. Each statute has a standard format, with title, code and section/subsection numbers. We use the statute title cited most by each case as its category label. We choose the eight most cited statute titles, namely "Title 8: Aliens and Nationality", "Title 11: Bankruptcy ", "Title 15: Commerce and Trade", "Title 18: Crimes and Criminal Procedure", "Title 21: Food and Drugs", "Title 28: Judiciary and Judicial Procedure", "Title 29: Labor" and "Title 42: The Public Health and Welfare". We randomly select 10,000 samples for each statue title.

The dimensionality of word and document vectors are fixed as 400 for all learning models. The number of negative sampling is fixed as 5 for Skip-gram, PV, LINE and DCV. We set the word context window size n = 5 in DCV-vLBL and b = 5 in DCV-ivLBL. The context window of document sequence m is fixed as 1 by which we only consider the immediate neighbors that the current document links to. Increasing m, DCV may yield performance gain, but with higher cost of computation. To construct word-word network Gww, we consider co-occurrences of words with context window size 5. We also test model performance using concatenation of document vectors trained using PV-DBOW and PV-DM, as well as DCV-vLBL and DCV-ivLBL. For LINE, we use concatenation of the vectors trained on first and second-order proximity [Tang et al., 2015b].

Results of Document Classification

We train one-vs-rest logistic regression on the resulting vector representations of documents.Table 2 shows the average Micro-F1 and Macro-F1 scores. The results are average over 5-fold cross-validation on the sampled data. The baseline BOW works well on all datasets, but with features in dimensionality of millions. Learning models with 400 dimensional vectors reduce the document feature space by at least 99.8% for any dataset. Our DCV consistently performs better than PV, LINE, DeepWalk and LDA. The performance gain of DCV is significant on DBLP where the documents (paper titles) are extremely short. The performance of PV degrades in modeling short text as the vectors learned will overfit the few words in each document. However, DCV does not suffer from this problem as the links provide discriminative information for learning better document representations. On Legal dataset, it's worth noticing that most of documents in Legal dataset are long documents with thousands of words. DCV is not easily tuned in this case, but still perform well. In most cases, models that use text features only can achieve good performance. The link information can slightly help document classification, but models using link information solely cannot perform well. We also present the running time to train vectors for each methods on Legal dataset. As we can see, DCV can efficiently train vectors on 2M documents with 6.3B words and 4M links in about 10 to 26 hours, which is quite scalable.

Dimension of Vectors

We vary the dimensionality of vectors in DCV, and report the classification performance on Wikipedia and Legal datasets in Figure 4. With larger dimensionality of vectors, the vectors are trained at a higher computational cost. We can see the performance of DCV increases when dimensions of vectors increase to 200 or 400, which is suitable for both datasets. However, when dimensions increase to 800, the performance gain is marginal.

4.4 Link Prediction

In this experiment, we investigate how much information the learned vectors can provide for predicting unseen links between documents.

Experiment Setup

For Wikipedia and DBLP datasets, we hold out all out-links of the test data as described in the task of document classification (Section 4.3) for evaluation. For Legal dataset, we hold out all out-links of legal opinions of 62,018 supreme court decisions, each has about 4 links on average. For each test document, we rank other documents by Cosine similarity in vector space, instead of predict whether it will or not link to any other documents. In this experiment, we fix the number of hops m as 2 in sampling document sequence.

Results of Link Prediction

Table 3 shows the results averaged over all test documents. We use Precision at the cut-off 10 (P@10) and Mean Average Precision (MAP) for evaluation. As we can see, DCV consistently performs better than other methods, and the performance gain is significant on Wikipedia dataset which has relatively denser document-document network. DCV also performs well on DBLP dataset. This demonstrates the capacity of DCV in modeling short text with link information. Exploiting both text and link information, DeepWalk, LINE and DCV generally perform better than using text or link solely. LDA performs worst in this task. This indicates the topic distributions learned from bag-of-words features do not provide much information for link prediction.

Number of Hops

We explore the effects of the number of hops, m. Figure 5 shows the link prediction results using DCV with different number of hops used. We observe for DBLP dataset, the performance of DCV does not change much with hops in the range of 1∼ 3. Hence small number of hops are preferable due to its low computational cost. DCV performs better with the number of hops is set as 2 than other values for DBLP and Legal datasets. Sampling long document sequence has high computational cost and is not effective in real applications.

5 Conclusion and FutureWork

In this paper, we proposed a neural language model called Deep Context Vectors (DCV) for linked documents in document networks. We described two model architectures and showed how to simultaneously learn word and document vectors in our framework. By modeling the deep contexts of words in linked documents, our model can learn better document representations in a low-dimensional vector space. Experimental results showed that our model outperforms other neural language models, such as Paragraph Vector [Le and Mikolov, 2014], that do not take into account the links between documents. Interestingly, the vectors learned by DCV, which have on the order of few hundreds of features, can perform better or comparably to the bag-of-words model, which uses millions of features.

In future work, we would like to use side information such as the relative location of each word and the hyperlinks or citation in the paragraph, to determine which linked documents should be modeled for each word.

Acknowledgements

This work was supported in part by the National Science Foundation under grant SMA-1360058 and in part by Leidos. We thank anonymous reviewers for their helpful comments.