Self-paced Compensatory Deep Boltzmann Machine for Semi-Structured Document Embedding

1 Introduction

There have been massive documents in many web applications with the development of Internet. One kind of documents, which consists of the plain text and a group of metadata, are ubiquitous source of information. This type of documents can be called the semi-structured documents (SSDs) [Zhang et al., 2009; Li et al., 2013b; Soto et al., 2015; Li et al., 2016]. It is an important task to learn a good representation for a semi-structured document. The aim of document embedding is to generate such representations that are useful for document classification and retrieval tasks. When embedding the SSDs, besides the plain text in a document, the structured metadata such as authors and keywords in a paper and the director and actors in a film may contain rich information and can provide a great help to extract more meaningful document feature in a similar way to how human understand unstructured text. For example, when we are reading a textual description about a movie "A team of explorers travel through a wormhole in an attempt to ensure humanity's survival.", we would have no sense about what movie it refers. Then some keywords (one type of metadata) such as "Father daughter relationship, Space travel, Time paradox" can tell us that it is a Sci-Fi and father-daughter emotional movie. So these metadata would help us to extract more meaningful features about the text. Moreover, when another type of metadata "Christopher Nolan, Matthew McConaughey" is given, it is easy to know that this movie is "Interstellar (2014)". From this example, we can see that people could better understand unstructured text when provided hierarchical background knowledge. This is consistent with the behavior of the brain that people constantly add background knowledge to more accurately understand what he/she reads. This paced process shows an effective way to exact latent features for text, especially when different types of metadata are provided.

∗ Part of this work was performed at Microsoft Research Asia.

Thus, it would be better to utilize the metadata information when modeling the text. Recently, many models have been proposed to model the text as well as the metadata, such as [Li et al., 2013b]. Li et al. propose a method to learn from the SSDs using directed topic model, where the major drawback is that it treats all the metadata equally, which ignores the fact that the metadata may belong to different types, and could have very different importances and concepts for the documents. It is obvious that the different types of metadata need to be considered gradually, which means that the types of metadata should be modeled into a hierarchical structure, just like the way of human learning that people would gradually take advantages of different auxiliary information in reading. Another challenge is that the order of different types of metadata need to be considered when we build this hierarchical structure. Similarly when reading a document, we may think of different types of metadata in different concept layers.

Thus, we propose a novel Self-paced Compensatory Deep Boltzmann Machine (SCDBM) that combines deep learning models with a latent topic model for semi-structured document embedding, in which each type of metadata is selected in different layers of a deep Boltzmann machine [Salakhutdinov and Hinton, 2009a] as compensatory information. The proposed SCDBM takes the corpus with the metadata as the input to obtain the documents' embeddings after an effective combination of bag-of-word information and the metadata through a deep Boltzmann machine. We build the metadata information as one of teacher networks described in [Hu et al., 2016], where the metadata can be used as one type of learning resources.

It has two principal contributions. First, through simulating the process of understanding text of human, this model can embed the documents with the benefits of integrating metadata information. Second, it provides an automatic method to handle different types of metadata in a self-paced way to add them into respective hidden layers in the deep Boltzmann machine, and has an efficient learning procedure to estimate the model parameters for each metadata-compensatory hidden layer.

In the experiment section, we will present the experimental results on two corpora, Wikipedia and IMDB, to show the performance of the proposed model on semi-structured document embedding for document classification and retrieval and tag prediction when comparing with state-of-the-art baselines.

2 RelatedWork

To date, there has been lots of work on developing document embedding using undirected graphical models or directed graphical models. As an undirected graphical model, the Replicated Softmax Model (RSM) [Salakhutdinov and Hinton, 2009b] can be used to model and extract low-dimensional latent semantic representations from unstructured collection of documents. Nitish et al. provide a type of deep Boltzmann machine, called the over-replicated Softmax model in [Nitish et al., 2013] to extract distributed semantic representations from unstructured documents in a different way. Directed graphical models, such as Latent Dirichlet Allocation (LDA) [Blei et al., 2003], CTM [Blei and Lafferty, 2005] and RATM [Li et al., 2017], have been extensively used for building generative probabilistic models of the bag of words in a document. Several topic models have been proposed to take advantage of the metadata given in documents, such as Tag-Weighted Topic Model (TWTM) [Li et al., 2013b], TagWeighted Dirichlet Allocation (TWDA) [Li et al., 2013a], Author Topic Model (ATM) [Rosen-Zvi et al., 2004], Labeled LDA [Ramage et al., 2009], and so on. These models are useful for modeling sparse count data based on the assumption of bag-of-words, and provide methods to handle the word count representations in a document and to extract its latent topics.

Models based on deep neural networks have shown a powerful capability on image processing, speech recognition and many other areas [Hinton et al., 2006]. Le Roux and Bengio has proved that adding hidden units yields strictly improved modeling power, and that restricted Boltzmann machines (RBMs) are universal approximators of discrete distributions to represent any distributions [Le Roux and Bengio, 2008]. Thus, each layer in the hierarchical (deep) architecture of deep Boltzmann machine (DBM) [Salakhutdinov and Hinton, 2009a] may provide a particular representation about the input data. However, there are few works investigated document modeling based on deep neural networks in semi-structured document embedding. In our paper, we take advantage of the capability of DBM to get the document representation by adding each selected type of the metadata information into the DBM architecture level by level.

The curriculum learning [Bengio et al., 2009] provides a learning paradigm where a model is learned by gradually including more complex samples in the training process, which can be used to select the metadata to integrate the text understanding process layerwisely. Self-paced learning [Kumar et al., 2010] [Jiang et al., 2015] present a way to learn the priority ordering of each data point in the training process, which just matches the metadata selection process in our work. Thus, we take advantages of the self-pace learning to learn the ordering of different types of metadata selected for each layer.

3 Self-paced Compensatory Deep Boltzmann Machine

3.1 Notation

Consider a semi-structured corpus D = {(w1,m1), ..., (wd,md)}, where each 2-tuple (w,m) denotes a document, where w represents the bag-of-words representation of the document, and m represents the metadata in the document. m = (m (1) ,m (2) , ...,m (k) , ...,m (t) ) where m (k) denotes the k-th type of the metadata in the corpus and t is the number of the types of metadata in the corpus. Similar to deep Boltzmann machine, we define h = (h (1) , h (2) , ..., h (i) , . . . , h (t) ) as the hidden layers in our model.

3.2 The proposed Model: SCDBM

To form a self-paced compensatory metadata structure, we gradually add the types of metadata layers as the compensatory information into the construction of deep Boltzmann machine layer by layer. Figure 1, left panel, shows an example of the proposed model with three hidden layers h = (h (1) , h (2) , h (3) ) and three types of metadata considered in a corpus where m = (m 1 ,m 2 ,m 3 ). We build a deep multilayers Boltzmann machine first, and then combine each hidden layer with metadata by learning a joint RBM. As shown in the left panel, Figure 1, we first select one type of metadata m 1 as the compensatory metadata layer, then combine the input layer w to form an RBM with h (1) , where the compensatory metadata set 〈m〉 = {m 1 } . For the next level, we select another type of metadata to add to the compensatory metadata layer, where the set 〈m〉 = {m 1 ,m 2 }, to combine with h (1) , h (2) to form an RBM. For the last level, the remaining type of metadata is added to compensatory metadata layer m, where the set 〈m〉 = {m 1 ,m 2 ,m 3 }, and then we learn an RBM to get the final hidden layer h (3) as the embedding of (w, m).

Clearly, we need to build a t-layer of deep structure if there are t types of metadata in the corpus. Consider a t-layer SCDBM model. The energy of the state {w,m 1 , ...,m t , h (1) , ..., h (t) } is defined as (ignoring bias terms on the hidden units for clarity):

where |m| t = m 1 |m 2 |...|m t means we concatenate the types of metadata as the compensatory metadata layer. In the above equation, θ = {W 1 , ...,W t , ˆW 1 , ..., ˆW t } are the model parameters: W 1 and W t are w-to-h (1) and h (t-1) -to-h (t) symmetric interaction terms, and ˆW i is symmetric matrices of weights associated with the connections between hidden layers and compensatory metadata layers. The probability that the model assigns to word data w and metadata (m 1 , ..., m t ) is:

where Z(θ) is the partition function.

3.3 Self-pacedMetadata Selection and Pretraining

As described above, the SCDBM selects the types of metadata layer-wisely to add into the deep structure. Then how to define the order of the types of metadata selected needs to be considered. The sequence of gradually added training samples, or other data, such as types of metadata can be called a curriculummentioned in [Bengio et al., 2009]. A straight forward way to select the sequence of metadata is to use metaheuristic measurements [Spitkovsky et al., 2009]. While as described in [Kumar et al., 2010], self-paced learning is a recently proposed learning regime inspired by the learning process of humans and animals that gradually incorporates data into training process. Actually, the proposed compensatory deep Boltzmann machine can be treated as one special selfpaced learning problem where the main difference lies in the training data. Self-paced learning defines the order of training samples, and in our proposed model, the training samples are naturally separated into different data types, such as different types of metadata (the words in a semi-structured document can also be treated as one special type of metadata). Moreover, in the proposed model, we only consider the sequence of the different types of metadata where the number of types could be very small. The benefit is that we can easily learn the sequence through defining a problem-specific self-paced learning process. Thus, in the following, we introduce a tailored self-paced metadata selection process during pretraining layer-wisely.

We take advantage of a greedy layer-wise training process [Bengio et al., 2007] based on learning a stack of "modified" RBMs using Contrastive Divergence (CD) algorithm [Hinton, 2002] for model pretraining and self-paced metadata selection. For the k-th layer, the input layer in the "modified" RBM consists of by two parts: the hidden layer h (k) and the compensatory metadata layer m (k+1) , where the m (k+1) contains k + 1 selected types of metadata 〈m〉selected. The output layer is h (k+1) and the candidate types of metadata set is denoted by 〈m〉le f t .

In order to select one type of metadata from 〈m〉le f t, we train the "modified" RBM with all the types of metadata from 〈m〉le f t with m (k+1) and h (k) , and the goal is to jointly learn the model parameters θ and a latent weight variable v = [v1, ..., vt-k-1] by maximizing the Pseudolikelihood of this "modified" compensatory RBM:

where λ is the parameter of learning pace and we set it to a constant here. We define v as an indicator vector that only one dimension equals 1, because we need to select one type of metadata. Unfortunately, jointly training the "modified" compensatory RBM with all the types of metadata involved using Eq. (3) would be very slow. Note that with the assumption of v as an indicator vector, the training process can be simplified, since we can search the space of v to find the solution, especially when the dimension of v is small in real. Thus, maximizing the Pseudolikelihood of this "modified" compensatory RBM is equivalent to maximizing the decision function:

For the first layer, we use the w as the input layer and select one type of metadata as the compensatory layer by the above decision function. Note that, we only choose one type of metadata to add into the compensatory metadata set 〈m〉

for each layer, so the depth of the proposed model is determined by the semi-structured corpus, where k types of metadata determine k layers of our model. The intuitive explanation is that the selection process is greedy, and the metadata which have more contribution to model the words in a document would be selected prior to that having less contribution. This process could mimic the human learning process that the human gradually take advantages of different auxiliary information when learning.

To train the "modified" compensatory RBM with each candidate type of metadata, the conditional distributions over h (k) , 〈m〉selected, h (k+1) and the candidate m c can be easily derived as follows:

where |m, m c | = m 1 |m 2 |...|m k |m c , and g(x) = 1/(1+exp(-x)) is the logistic function. We can also use rectified linear units described in [Nair and Hinton, 2010].

Metadata Embedding

In many applications, different types of metadata in one semi-structured corpus come from different kinds of information, such as the discrete and sparse category information, the real-valued and dense image pixel or timestamp information. This would make it difficult to learn the SCDBM model directly with the various forms of the metadata information without any preprocessing. Thus, we embed different kinds of metadata using RBMs (or their generalizations to exponential family models which have powerful representation as shown in [Le Roux and Bengio, 2008] and [Welling et al., 2004]) as shown in Figure 1, right panel. We let ˜m denote the embedding of metadata. In particular, the traditional restricted Boltzmann machine could map the discrete and sparse data into low-dimensional representation, and the Gaussian RBM can be used to handle the real-valued data (e.g. image patches).

The main purpose that we do the metadata embedding is to reduce the dimensions of the metadata. When one type metadata is noisy, this would degrade the performance of the proposed model. Moreover, the observed metadata is typically high-dimensional in many applications, and if we use the observed metadata directly, the high-dimensional representation of m would make the model learning slowly. Another intuition behind the metadata embedding is that each type of metadata may have very different statistical properties that make it difficult to directly combine with hidden layers as compensatory information. The way of metadata embedding is to learn latent representations of metadata to remove such type-specific property so that the metadata information can be compensated to the corresponding hidden layers. On the other side, the metadata can be treated as one type of instance learning resources, or the simple rules that guide the building process of the network as shown in [Hu et al., 2016]. The metadata embedding process may enhance the benefits of the metadata for performance improvement of text embedding.

Similar to the metadata, instead of directly using the wordcount vectors of documents as input of the deep network, we use latent topic models (such as LDA or RSM) to extract the latent topic vectors before SCDBM learning. As shown in the right panel of Figure 1, ˜w represents the latent topic distribution obtained via a LDA or RSM, and then we use ˜w as the input of the first layer in SCDBM.

3.4 Approximate Learning and Inference

Exact maximum likelihood learning in this model is intractable. Thus, we carry out an efficient approximate learning process using the mean-field inference to estimate datadependent expectations. [Salakhutdinov and Hinton, 2009a] describes a mean-field inference to estimate data-dependent expectations in DBM and an MCMC based stochastic approximation procedure to approximate the DBM's expected sufficient statistics. In this work, we use the same procedure for SCDBM model learning. Specifically the true posterior p(h|˜w, ˜m; θ) with a fully factorized approximating distribution over the t sets of hidden units is approximated as:

where µ = {µ (1) , ..., µ (t) } are the mean-field parameters with q(h ( j) i = 1) = µ ( j) i for j = 1, ..., t. Thus, the variational lower bound on the log-probability of the data can be formulated as follows:

where H(q) is entropy of the variational distribution and |˜m| (t) denotes the t-th layer's metadata concatenation. The learning process is as follows. First, we find the value of µ which maximizes the lower bound logP(˜w, ˜m; θ) given the current value of model parameters θ. Then, given the variational parameters µ, we can update the model parameters θ to maximize the variational bound using stochastic approximation [Salakhutdinov and Hinton, 2009a; Tieleman, 2008]. Actually, the number of types of metadata would be very small in real applications, and the training process is very efficient, especially after metadata embedding.

3.5 Metadata Prediction

After training the SCDBM model, we can infer the type of metadata compensated at the each level of the structure. For example, when we need to predict the metadata at layer i which means the i + 1 type of metadata in the compensatory sequence is missing, m i+1 , we first infer the values of the hidden variables h (i) with w and the metadata sets 〈˜m〉 (i) , then we can perform alternating Gibbs sampling using the following conditional distributions until convergence:

where W i is the weight matrix between h (i) and h (i+1) , and ˆW i b is the weight matrix between h (i+1) and ˜m (i+1) . We use the distribution of p(˜m i+1 |h (i+1) ) to sample ˜m i+1 . The sample ˜m (i+1) can then be propagated back to generate a distribution over the vector space of metadata m (i+1) , which can be used to sample the real metadata.

For the self-paced metadata selection process when pretraining, the complexity is same as a traditional RBM in each layer for one type of metadata selection. Thus, the total complexity in self-paced metadata selection process is t times for a traditional RBM, where t is the number of type of metadata (both the depth of the network). For approximate learning process, the complexity is same as a traditional DBM, since the compensatory metadata for each layer has fixed.

4 Experiments

4.1 Description of Datasets

In the experiments, we used two semi-structured corpora. The first dataset is from Wikipedia. The Wikipedia corpus used in our experiment contains D = 213,600 articles, with W = 70,061 words in the vocabulary by removing the stop words. Each article contains two types of metadata. The first type of the metadata m 1 is the entity information observed in the abstract of the article with the hyper links. The m 2 is the category information normally found at the bottom of an article page. There are 974 categories in the category dictionary and 10,208 entities after removed low-frequency items. Each article belongs to one of 18 classes, such as history, education, technology, art and so on. The second corpus is the data from Internet Movie Database (IMDB). The data set we used includes 106,432 movie story lines, and 15,703 words after removing the stop words and low frequency words. Each movie contains three types of metadata. a) m 1 , the crew of the movie including the director, the writers and the actors. b) m 2 , a list of keywords. c) m 3 , the movie information including rating, production, country, language, the length of duration, sound mix and date. For m 3 , we simply used a 6 dimension realvalue vector to represent. There are 5,192 persons appeared in all the movie by removing the persons appeared less than 5 times, and 5,773 keywords in the keyword dictionary. Each movie belongs to one of 29 genres.

4.2 Experimental Setting

We compared the SCDBM with LDA, Replicated Softmax Model (RSM), and TWTM [Li et al., 2013b]. Because LDA

(a) Lh(2) = 300 onWikipedia and RSM are the models handling the unstructured documents without the metadata information, so we treated the given metadata as word features for them. For the Wikipedia, we treated all the metadata, including entities and categories, as the words to train LDA and RSM, which are called LDAEC and RSM-EC, respectively. Likewise, we also trained LDA and RSM on the IMDB with the metadata as the words features, and we called the models LDA-M and RSM-M. For the RSM model, we subdivided datasets into minibatches, and each contains 100 training cases, and updated the parameters after each minibatches to speed-up learning in the same way as RSM. Moreover, we choose the 2,500 most frequent words as the feature representations in the training dataset. With the three models, we embedded the documents into 300dimensional latent representations.

When training the proposed SCDBM, we first embedded the metadata of the two corpora into low dimension using RBMs. For Wikipedia, we let L˜m (1) = 300, and L˜m (2) = 300. For IMDB, we let L˜m (1) = 300, L˜m (2) = 300 and L˜m (3) = 6. To model the word count vector, we trained LDAs with 300 topics on the two corpora, and treated the latent topic distributions as ˜w at the bottom level of our model. The dropout rate is set to 0.5 as described in [Srivastava et al., 2014] when training our model.

Since the number of types of metadata in each corpus is small, the starting value of v is becomes critical. Thus, after tried some methods to initialize the v, we use a logarithmic scheme to initialize the v as shown as in [Jiang et al., 2015] to get the best results. After pretraining and selection for the types of metadata to be added into the compensatory metadata set, we obtained the following sequence for each corpora. For Wikipedia, we built a two-layer network with 〈m 1 〉 (1) , 〈m 1 ,m 2 〉 (2) as the compensatorymetadata layers, and let Lh (1) = Lh (2) = 300. For IMDB, we built a three-layer network with 〈m 2 〉 (1) , 〈m 2 ,m 1 〉 (2) , 〈m 2 ,m 1 ,m 3 〉 (3) as the compensatory metadata layers, and let Lh (1) = Lh (2) = Lh (3) = 300.

4.3 Document Classification and Retrieval

In our first set of experiments, we evaluated the single-class classification performance utilizing feature sets generated by SCDBM and other baselines on the Wikipedia and IMDB. As we have mentioned, we used the latent features of the semistructured documents dataset embedded by SCDBM to train a classifier based on LIBSVM with the Gaussian kernel and the default parameters. Figure 2 shows the classification results on the two corpora with mean square errors. Intuitively, SCDBM takes full advantage of metadata, especially with the depth increases. In order to test the way of adding the different types of metadata into different hidden layers respectively, we also trained a variant model with adding all the metadata into the lowest level of a DBM, which we called DBM+M, compared with the self-paced metadata-compensatory way proposed in this paper. These comparisons indicate that the way of self-paced metadata-compensatory could bring the performance improvement. The main reasons are following. First, different types of metadata would have different concept properties, the way of self-paced selection in our model is just to find a better way to take advantages of this metadata. Second, under the setting of SCDBM, the metadata which is Table 1: Some examples of the predictive categories generated by SCDBM on Wikipedia, where Column 2 is the real categories appeared in the test pages, and Column 3 in red is the highly correlated categories generated by SCDBM.

Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17)

selected in the first layer could influence the structure to the last layer, which means that the different types of metadata have different weights during the training.

We also used IMDB to evaluate the performance on a document retrieval task. To decide whether a retrieved document is relevant to the query document, we just check whether they have the same class label. In order to do retrieval, we represent each document w using the latent vector representation generated by SCDBM and other models in comparison. The corpus was randomly divided into two part: 80% the database documents and 20% the query documents. For each query, documents in the database were ranked by using the cosine distance as the similarity metric. We averaged the results of all the queries by using the F-Measure (F1-score). As shown in Figure 3(a), the SCDBM outperforms other models on F1score. We also show the performances of different models on document retrieval by drawing the precision-recall curves. Figure 3(b) shows the results, from which the SCDBM also has a better performance than other baselines, particularly when retrieving the top few documents.

4.4 Results onMetadata Prediction

In this experiment, we show the generative aspect of our model qualitatively. We randomly selected 1,360 documents from the Wikipedia dataset as the test set, and treated the left 10,000 documents as the training set. Before training, we embedded the entities and categories of each document into low-dimensional space using the RBMs. When inferring a document, we only used the words (w) and entities (m (1) )

of the document to generate the vector h (1) , then generated the distribution ˜m (2) by the method which we mentioned in Section 3. Since the value of the unit in ˜m (2) is the distribution of p(m (2) = 1), We simply ranked the values in ˜m (2) and selected the top 20 as the predictive categories generated by SCDBM. Table 1 shows some examples of the predictive categories generated by the SCDBM for a group of pages in the test set. The first column shows the articles in test set we inferred, and columns 2 and 3 are the categories predicted by our model; Column 2 is the real categories appeared in the test pages, and Column 3 (red) is the highly correlated categories generated by SCDBM. By taking the entity of "David Beckham" for an example, the categories of "English footballers" and "UEFA Euro 2004 players" are generated by our model, which are the real categories given in the article of "David Beckham". The categories (red) in Table 1 are highly correlated with the input article, even though some of them are wrong in fact. In other words, the SCDBM may have the capability of learning the internal relations among the metadata items, as well as capturing relation between words and metadata.

5 Conclusion

We proposed a self-paced compensatory deep Boltzmann machine to model documents by combining the word features and metadata information by simulating the process of people understanding text. The model fused the word count vector and metadata into a jointly hidden representation, and provided improved capability in terms of classification and document retrieval compared to state-of-the-art models. Besides, the model has demonstrated the ability to predict the missing metadata in some applications.

Acknowledgments

This work was supported in part by the National Key Research and Development Program of China under Grant 2016YFB0201900. Also, this work was supported in part by NSF of China under Grant 61672548, U1611461, and the Guangzhou Science and Technology Program, China, under Grant 201510010165. We thank the support of Key Laboratory of Machine Intelligence and Advanced Computing (Sun Yat-sen University), Ministry of Education, China.