Dataless Short Text Classification Based on Biterm Topic Model and Word Embeddings

1 Introduction

It is very popular for people to obtain and exchange information in different applications and websites on the Internet, such as instant messages, social media and news media. A huge number of short texts are generated every day, and it is crucial to acquire important and interesting information from them. Short text classification plays a fundamental role in short text processing. Many recent studies on this task were based on supervised deep learning methods [Wang et al., 2017; Zeng et al., 2018], which achieved significant improvement over traditional classification models. However, the lack of plentiful training data limits the application of these models, since labeling documents is very expensive and time consuming for domain experts.

∗Corresponding author

Dataless text classification has attracted more and more attentions, as it only requires a small set of seed words for each category [Liu et al., 2004; Druck et al., 2008], which are much cheaper than labeling documents. Many existing approaches [Chen et al., 2015b; Li et al., 2016b; Li et al., 2018a] based on topic models took advantage of seed words and word co-occurrence information in a weaklysupervised manner, and performed significantly well in many scenarios. However, these models aim at normal documents, but are not suitable for irregular short texts on the Internet.

Different from long documents, short texts are extremely sparse and only limited word co-occurrence information can be utilized to form a topic and further correspond to a category, so the models above get unsatisfactory results for classifying short texts. [Li et al., 2019] proposed a seed-guided topic model for dataless short text classification and filtering (SSCF), but it costs much time and neglects other types of information to enhance the correlation between categories and words. [Shalaby and Zadrozny, 2019] proposed a concept raw context model (CRX), which employs raw concept mentions from the knowledge base to learn concept embeddings for dataless text classification, but an appropriate even domainspecific knowledge base is indispensable.

Comparatively, word embeddings are widely used nowadays and relatively easy to access. Recently, many researches [Li et al., 2016a; Xun et al., 2016; Li et al., 2018b] made use of word embeddings in topic models for short texts, and demonstrated that word embeddings can effectively help to identify similar words in both syntactic and semantic levels, and this kind of external information can successfully alleviate the data sparsity. Based on this idea, we propose a novel model named Seeded Biterm Topic Model (SeedBTM), which incorporates pre-trained word embeddings into the classical short-text topic model BTM [Yan et al., 2013]. Specifically, we calculate the maximum similarity between a corpus word and seed words of a category to get the categoryword similarity score, served as the prior topic-in-set knowledge for topic-word distributions [Andrzejewski and Zhu, 2009]. Moreover, the key idea in this approach can also be extended to other biterm-based topic models. We further proposed Seeded Twitter Biterm Topic Model (SeedTBTM) based on Twitter-BTM [Chen et al., 2015a] when the user (author) information of short texts is available.

In summary, the contributions of this paper include:

1. An approach is presented to solve the task of dataless short text classification with seed words, by combining word co-occurrence information and category-word similarity based on word embeddings. To the best of our knowledge, it is the first successful work to utilize both short-text topic model and word embeddings to classify short texts in a dataless manner.
2. Two models SeedBTM and SeedTBTM are respectively proposed through applying our approach on short-text topic models BTM and Twitter-BTM. That indicates our approach is applicable on different topic models by effectively integrating meta information of short texts, such as users (authors).
3. Informative Experiments are conducted on five realworld datasets to show that our models significantly outperform the state-of-the-art baseline methods, especially when the categories are overlapping and interrelated.

The remainder of the paper is organized as follows. In Section 2, we review recent related work. Section 3 introduces the two models with the uniform approach. In Section 4, the experimental results on real short-text datasets are shown. Section 5 concludes this work and discusses future work.

2 RelatedWork

2.1 Dataless Text Classification

Many researchers focused on dataless text classification because the proposed models can successfully reduce the effort in labeling documents for text classification. Some work tried to exploit auxiliary knowledge bases to accomplish dataless classification [Chang et al., 2008; T¨urker, 2019; Shalaby and Zadrozny, 2019]. However, external knowledge bases are hard to obtain in many scenarios and these models are not suitable for classifying documents into specific and fine-grained categories.

Another group of researchers proposed dataless classifiers by utilizing category seed words. Some of these studies were based on pseudo-labels obtained approximately from seed words [Liu et al., 2004; Druck et al., 2008], but these pseudolabels often result in noisy training data. [Meng et al., 2018] proposed a novel weakly-supervised text classification model (WeSTClass). It at first constructs a semantic space by learning vector representations, then extends category seed words in the semantic space to generate pseudo-documents for training a neural classifier, and finally fits unlabeled data through bootstrapping. However, this model would fail when the categories are interrelated, as that means the semantic sub-spaces of these categories are overlapping and confusing.

Recently, several topic model based approaches have been proposed for dataless text classification with seed words. TLC++ [Hingmire and Chakraborti, 2014] labels topics clustered by Latent Dirichlet Allocation (LDA) [Blei et al., 2003] with seed words based on information gain, and transforms topic-word distributions into category-word distributions for classifying documents. [Chen et al., 2015b] proposed descriptive LDA (DescLDA), which constructs some descriptive documents with seed words to guide topic-word distributions, and then labels documents via clustering documenttopic distributions. [Li et al., 2016b] proposed Seed-Guided Topic Model (STM). It assumes that each document is associated with a single category topic and a mixture of general topics. The former consisting of seed words is used to determine the document category, and the latter help to capture more word co-occurrence information. As an extension of STM, [Li et al., 2018a] proposed DFC for dataless text filtering and classification. However, these models focus on long documents and cannot accommodate the data sparsity of short texts. Furthermore, [Li et al., 2019] proposed SSCF to handle short texts, which estimates the co-occurrence correlation of seed words and corpus words with a word network topic model (WNTM) [Zuo et al., 2016], but this approach is time consuming and does not make use of word embeddings to supplement word similarity information.

2.2 Topic Models for Short Texts

For short texts, applying conventional topic models directly is less effective due to document-level data sparsity, so many researches aggregate short texts to pseudo-documents through additional metadata [Weng et al., 2010] or heuristic strategies as pre-processing [Quan et al., 2015]. Some other models are based on the assumption that each document has only a single latent topic, like Twitter-LDA [Zhao et al., 2011] and Dirichlet Multinomial Mixture (DMM) [Yin and Wang, 2014]. This assumption is proved to be effective in many scenarios, but sometimes it is too strong to cluster well.

As another branch, [Yan et al., 2013] proposed Biterm Topic Model (BTM), which explicitly models word cooccurrence patterns in the generative process of the whole corpus. [Chen et al., 2015a] further proposed Twitter-BTM by combining BTM and Twitter-LDA, which introduces user information and the background topic into BTM. In addition, [Jiang et al., 2016] proposed biterm pseudo-document topic model (BPDTM) using the word co-occurrence network to construct biterm-based pseudo-documents. [Li et al., 2018b] proposed relational BTM (R-BTM) to link short texts via a similarity matrix of words computed by word embeddings. Our approach selects BTM as the base model, since it is flexible for different scenarios and can easily be extended to combine various aggregation strategies with metadata.

3 ProposedModels

In this section, we at first present the overview of our approach and then elaborate the key step of estimating the category-word similarity. After that, we propose two models SeedBTM and SeedTBTM based on this approach in detail.

3.1 Approach Overview

Given a corpus of unlabeled documents D = {d1, ..., dN} and a set of target categories Z = {z1, ..., zK} with respective seed words, our approach aims to assign a category label to each document, taking advantage of external word embeddings. It consists of two steps as Figure 1 illustrates. The first is to estimate the category-word similarity matrix, which provides prior knowledge about the relation between topics (categories) and words. The second is to infer documenttopic distributions, incorporating the information above into the generative process of a biterm-based topic model to adjust and optimize topic-word distributions.

3.2 Estimating Category-Word Similarity

To effectively utilize seed words to guild the classification of short texts, we follow the idea that word similarity scores based on word embeddings could represent the semantic correlations between words [Li et al., 2016a; Li et al., 2018b]. In our models, with a vocabulary W = {w1, ..., wV } including V words, each category corresponds to a topic z, and has a category-word similarity vector δz as the prior topicin-set knowledge. Next, we will introduce how to calculate the value of each dimension δz,w(w ∈ W) with seed words.

At first, for a category seed word s and a corpus word w, we get the word vectors vs and vw through word embeddings, and then calculate the word similarity sim(s, w) as follows:

Notice that the range of the cosine function is [-1, 1], but we hope the similarity score sim(s,w) can be regarded as a positive weight and lies in (0, 1], so here we set a threshold ϵ > 0 to give a lower bound on sim(s, w).

Then, for a category z with multiple seed words, sz,1, ..., sz,nz , we compute the category-word similarity δz,w as the maximum similarity between each seed word and w:

3.3 Seeded BitermTopic Model

With the similarity scores, we extend the base model BTM and proposed SeedBTM. In BTM, A biterm bi,j contains two words wi and wj co-occurring in a short text, regardless of the order. Given a sampled topic z, we think the generation of a word w in SeedBTM is influenced by both prior categoryword similarity δz and the topic-word distribution φz. That drives the model to induce category-aware topics in the inference process. As Figure 1 illustrates, the generative process of SeedBTM is as follows:

1. Draw a distribution over topics: θ ∼ Dir(α)
2. For each topic k = 1, ..., T
3. (a) Draw a topic-word distribution φz ∼ Dir(β)
4. (b) Modify the topic-word distribution φ′z ∝ δz · φz
5. For each biterm b in the biterm set B
6. (a) Draw a topic zb ∼Multi(θ)
7. (b) Draw two words to form b: wi, wj ∼ Multi(φ′z )

Compared to BTM, our model transforms the topic-word distribution φz to φ′z by multiplying the category-word similarity vector δz (2b), and samples both words of a biterm following the new distribution φ′z (3b).

Inference via Gibbs sampling. Similar to BTM, after random initialization on the Markov chain, we iteratively calculate the conditional distribution P(zb|z¬b,B,δ) for each biterm b = (wi,wj), where z¬b denotes the topic assignments for all biterms except b, and B is the set of biterms in the whole corpus. The formula can be easily obtained by applying the chain rule on the joint probability for all biterms.

where nz is the number of biterms assigned to the topic z, and nw|z is the number of times when the word w is assigned to the topic z. Note that δz,wi and δz,wj are the added terms to introduce the prior knowledge for the two words respectively.

Predicting document category. Like BTM, SeedBTM treats the expectation of the topic proportions of biterms in a document as the topic proportions of the document:

where P(z|b) can be calculated by Equation 3 and P(b|d) is estimated based on the relative frequency of b in d. Finally, for document d, the category label zd can be predicted as the topic with the highest probability:

3.4 Seeded Twitter BitermTopic Model

The key idea above can be extended to other biterm-based topic models. We change the base model BTM to TwitterBTM, which incorporates user information and the background topic into BTM, and propose SeedTBTM. In this model, each user u is associated with a multinomial distribution θu over K topics drawn from a Dirichlet prior Dir(α), and the background topic B is associated with a multinomial distribution φB following a Dirichlet prior Dir(β). Each word w in any biterms has an indicator y. y = 0 means the word w is a background word, while y = 1 indicates w is a topic (category) word. y is associated with a uniform Bernoulli distribution π for all users drawn from a Beta prior Beta(γ). When generating a topic word, the modified distribution φ′z is adopted similar to SeedBTM. The generative process of SeedTBTM is omitted due to the page limit.

Inference via Gibbs sampling. We also perform Gibbs sampling to calculate the conditional topic distribution of a biterm for a user as follows:

where nuz is the number of biterms assigned to topic z for user u. wu,b,m and yu,b,m are respectively the word and the indicator of the m-th word of biterm b for user u. The distribution of the indicator yi can be calculated as follow:

where n(0) and n(1) are respectively the number of words assigned to the background topic B and category topics. Note that B does not have seed words as well as the prior similarity scores with corpus words, so for a fair comparison, we need to normalize the modified term for category-word proportions in the calculation of P(yu,b,m = 1). The prediction for the category label is same as SeedBTM and omitted here.

4 Experiments

4.1 Datasets

We show the model effectiveness on five real short-text datasets, the last two of which contain user information.

• SearchSnip [Li et al., 2019]: Search Snippets is a widely used dataset for short text classification. These texts were selected from results of web search transactions using predefined phrases in different domains. This dataset contains 8 categories and 12340 web search snippets.
• Reuters10T: Reuters-21578 1 is a dataset including news on Reuters newswire in 1987, and we just use the news titles as short texts. We choose 10 largest categories and remove the documents belonging to multiple categories.
• AGnews [Zhang et al., 2015]: It is another news article dataset, and the titles of 4 largest categories are selected.
• TwitterTrends10 2 : It contains 140K tweets of 10 trending events from 2015 to 2017, and can verify the capability of our models in event classification on social media.
• News Category Dataset 3 [Misra, 2018]: It is obtained from HuffPost and contains around 200K news headlines from 2012 to 2018. There are totally 42 categories, and many of them are overlapping, like college and education. We choose one 10-category sub-dataset HuffN10 and two 4-category sub-datasets, HuffN4-SWPB and HuffN4-ECMC, focusing on overlapping categories.

For all datasets, we at first lower and lemmatize all corpus words, remove stop words and then filter out the documents containing only one word. The categories and statistics of these datasets after pre-processing are shown in Table 1. Nu indicates the average document number of users and Nd indicates the average word number of documents.

4.2 Baselines

We evaluate our models against five baseline methods. They all deal with dataless text classification with seed words. The first two of which make use of word embeddings to compute category-word similarity, while the other three are based on topic models to capture word co-occurrences, but none of them combine the effects of these two kinds of information.

1 http://kdd.ics.uci.edu/databases/reuters21578/reuters21578.html

2 https://www.kaggle.com/laiyin/trendtweet10

3 https://www.kaggle.com/rmisra/news-category-dataset

• WeSTClass 4 [Meng et al., 2018]: It is a state-of-the-art weakly-supervised text classification model. It utilizes seed words and word embeddings learnt from the corpus to generate pseudo-labeled documents for training a classifier, and then bootstraps it with unlabeled data.
• WeSTClass*: In order to make explicit evaluations for our models, we implement an intermediate method replacing the self-trained embeddings in WeSTClass with external and pre-trained word embeddings, Glove Common Crawler 5 [Pennington et al., 2014], which is also used in our models.
• STM 6 [Li et al., 2016b]: It is a successful approach to dataless text classification for long texts based on topic model, utilizing the co-occurrence information among seed words and corpus words.
• SSCF [Li et al., 2019]: It is a state-of-the-art topic-based model for this task, specially aiming at short texts.
• DescLDA [Chen et al., 2015b]: It is an extended model of LDA by constructing descriptive documents with seed words to guide the classification results.

For WeSTClass(*) and STM, we adopt their implementation codes directly. For SSCF and DescLDA, all parameters are tuned according to the suggestions provided by the authors. As to user information, SeedTBTM integrates it in the generative process, while for baseline models, the user name is regarded as a part of text contents (words).

4.3 Experiment Settings

Seed word settings. There are two ways (sources) to get seed words as problem input, the category label itself consisting of several words (usually only one word, denoted by SL), or multiple descriptive words derived from them (denoted by SD ). The former type of seed words can be directly obtained from the corresponding dataset, while for the latter type, the selection strategy we adopt is similar to other work. We firstly run SeedBTM with SL, and then manually choose for each category 3∼10 suitable and representative words from the top 30 topic words. For the sake of fairness, our models share the same seed words (both SL and SD ) for each dataset with all baseline methods, except that SSCF and DescLDA have only SD type, which is consistent with their papers.

4 https://github.com/yumeng5/WeSTClass

5 http://nlp.stanford.edu/data/glove.840B.300d.zip

6 https://github.com/WHUIR/STM

Parameter settings. For all datasets, we set the topic number K′ = K (category number), α = 50/K + 1, β = 0.1, γ = 1 and ϵ = 0.0001. We set the number of iterations to 50 as our models achieve competitive performance since then. For word embeddings, we employ the widely used GloVe Common Crawl as mentioned before. It contains 840B tokens, 2.2M vocab and 300d vectors.

4.4 Experimental Results

We at first evaluate the classification performances of SeedBTM using Macro-F1. We run every model 10 times on each dataset to get the average value shown in Table 2.

We can observe that SeedBTM performs significantly better than baseline models on 6 of 7 datasets except AGnews. Specifically, the Macro-F1 value increases about 1.2∼6.3 percent compared to the best baseline result in these datasets. That indicates our model is able to enlarge the guiding role of seed words in short text classification with the help of easily accessible word embeddings. In addition, the improvement over WeSTClass* confirms that our model can utilize word embeddings more effectively through fusing word cooccurrence information in short texts. As an exception, our model underperforms WeSTClass and WeSTClass* on AGnews, since the categories in this dataset can be easily distinguished, and the word co-occurrence information is thus not so helpful. That explains our approach is more suitable for datasets with overlapping and interrelated categories. Moreover, The performances of baseline models fluctuate among different datasets, but our model can get more stable and effective classification results, so behaves domain insensitive.

Similarly, we evaluate SeedTBTM with user information on TwitterTrends10 and the three sub-datasets from HuffPost. The average Macro-F1 results are shown in Table 3.

Among all methods, SeedTBTM stably achieves the best performance on all datasets. Also, SeedTBTM significantly increases Macro-F1 scores about 0.4∼16.3 percent compared to SeedBTM on the four common datasets. That indicates our approach can effectively utilize metadata like user information to improve the accuracy of dataless short text classification, which is impossible to realize for supervised methods without sufficient labeled data for each user.

For different types of seed words, although the results of our models with SL is generally worse than those with SD except the last dataset, the gap is distinctly reduced compared to baseline methods. That implies with fewer seed words, our approach can achieve comparable performances via combining the information of word co-occurrence and category-word similarity, which simplifies the subtle process of selecting descriptive words automatically or manually.

4.5 Parameter Study

We now study the impact of different parameter settings on the classification performance of SeedBTM by using SD as seed words. When paying attention to one parameter, other parameters are fixed to the default values given in Section 4.3.

The impact of iteration number. This is an important factor for our models, because a smaller value means costing less time to obtain classification results. We vary the number in the range of [1,100], and the results are shown in Figure 2(a). We can see that SeedBTM can achieve good classification performances when the number of iterations is 10, and the F1-score is almost unchanged after that point. The fast convergence should give credit to the regulating effect of the prior knowledge from word embeddings.

The impact of threshold ϵ. We vary the lower bound for the word similarity in Equation 1 from 0.0001 to 0.5 and observe the influence on model performances shown in Figure 2(b). The flat lines certify that our approach is insensitive to this threshold, especially for smaller values less than 0.2.

The impact of word embeddings. In Table 2, WeSTClass outperforms WeSTClass* with SD on TwitterTrends10 and HuffN10 datasets, but not for others. That indicates word embeddings trained from the corpus may be more suitable in some scenarios. Thus, we replace the external general word embeddings in SeedBTM with self-trained word embeddings and name the new model as SeedBTM-Self for comparison. From Figure 2(c), we find that SeedBTM-Self performs worse than SeedBTM on SearchSnip, AGnews and HuffN10, and is even significantly poor on Reuters10T, while on the larger dataset TwitterTrends10 with event-related documents, SeedBTM-Self is better than SeedBTM. The results demonstrate that when the categories are domain-specific, external word embeddings are probably insufficient to characterize ad-hoc word correlations, but self-trained word embeddings from plentiful documents can lead to better performances. Therefore, our models can choose suitable word embeddings depending on the scope and granularity of categories to be predicted.

5 Conclusion

Aiming at the task of dataless short text classification with seed words, we propose a novel approach integrating both word co-occurrence information and category-word similarity information, which are respectively brought by bitermbased topic model and widely used word embeddings. The proposed models significantly outperform other baseline methods, especially for difficult and confusing classification problems. Moreover, the flexibility on the base topic model can effectively incorporate metadata like users to further improve the classification accuracy, which is hard to realize for supervised methods due to the lack of specific labels.

In the future, we plan to extend this approach to deal with the hierarchical short text classification task. In addition, it is interesting to study how to optimize the embedding-based similarity computation and its integration with topic models.

Acknowledgements

This work was supported by the National Key R&D Program of China (2017YFC0803805, 2018YFC0116703), and the Strategic Priority Research Program of Chinese Academy of Sciences, Grant No. XDC02060500.