TextGTL: Graph-based Transductive Learning for Semi-supervised Text Classification via Structure-Sensitive Interpolation

1 Introduction

Text classification is a fundamental and long-standing task of Natural Language Processing, which has been shown useful in countless intelligent services, such as sentiment analysis [Wang et al., 2020a] and fake news detection [Wang et al., 2020b]. Different from traditional sequential learning methods [Iacobacci and Navigli, 2019; Kim, 2014; Blei et al., 2003], some recently proposed studies adopt graph learning approaches to model free text, aiming to improve classification performance based on text representation with higher quality. More concretely, all these works utilize different media (e.g., words [Yao et al., 2019; Liu et al., 2020] and topics [Wang et al., 2019; Hu et al., 2019]) as bridges to associate free text, form heterogeneous text graph, and introduce graph neural networks to capture global and local information simultaneously.

Although these graph-based methods have yielded promising results on free text classification, they are mostly intuition-based and lack theoretical guidance, leading to limitations in real-world setups. On the one hand, as previously mentioned, they all rely on constructing heterogeneous graphs which not only hugely expand the scale of model parameters [Yao et al., 2019; Liu et al., 2020], but also introduce subsequent problems such as node redundancy, information missing, and error cascade propagation (illustrated in Figure 1). On the other hand, while the motivation of incorporating graph structures for text modeling is to promote information sharing along edges and break the independence across samples (nodes), thus alleviating the data sparsity and over-fitting issues, existing methods can merely achieve this goal on datasets where edges naturally have meanings. More particularly, this line of algorithms has only been successfully applied on data with explicit graph structures (e.g., citation [Kipf and Welling, 2017; Velickovic et al., 2018] and molecular bond [Fout et al., 2017]) but have never been used to model free text under more strict semi-supervised conditions.

In this work, we propose a semi-supervised pipeline framework, namely Text-oriented Graph-based Transductive Learning (TextGTL), which explores attacking the problem of semi-supervised classification for free text with a graphbased architecture. Moreover, to the best of our knowledge, we are the first to construct non-heterogeneous graphs for text classification, which greatly reduces the parameter complexity and makes the trained model lightweight.

Specifically, following the theoretical insights from document similarity research [P¨orner and Sch¨utze, 2019], we first design a set of novel strategies to respectively construct text graphs based on multiple linguistic information sources (i.e., semantics, syntax, and sequential context). Next, inspired by the works of [Li et al., 2018; Yang et al., 2019; Li et al., 2020], TextGTL refines the graph topology by strengthening the connection density and information consistency within the same dense substructure of the text graph, so that it can better serve the information propagation. We also obtain high-quality features through attribute space interpolation according to the structural tightness and estimate their low-entropy labels as new training samples, which substantially mitigates the data sparsity issue through data augmentation. Finally, to boost the performance of semi-supervised text classification, we design multi-graph parallel Graph Convolutional Networks [Kipf and Welling, 2017] to share hidden layer signals between multiple text graphs.

In order to demonstrate the effectiveness of our proposed method, we conducted extensive experiments on five popular text classification datasets and benchmarked TextGTL against seven strong baselines. Empirical results show that TextGTL outperforms its counterparts on all tested datasets, especially in the challenging corpus with long sentences. We further performed ablation studies to investigate how each component affects the final model performance as well as the relationship between text graphs via different linguistic information.

The main contributions of this paper can be summarized as follows:

• By proposing novel non-heterogeneous graph construction methods for free text, we significantly reduce the parameter scale and optimize the graph quality.
• Based on our unsupervised text graph topology refinement and data augmentation schemes, we successfully strengthen the useful information propagation among nodes and achieved graph-based semi-supervised free text classification.
• Through joint training on multiple text graphs, the proposed TextGTL system outperforms state-of-the-art methods in extensive experimental setups.

2 RelatedWork

2.1 Graph-Based Text Classification

Different from the existing sequence learning model, the graph-based methods expect to establish label or feature associations for independent samples by constructing a graph structure. In the initial stage, such methods are more common in text graph topology with real meaning, e.g., citation networks [Kipf and Welling, 2017; Velickovic et al., 2018], protein networks [Fout et al., 2017], and social networks [Liu and Wu, 2018]. They can learn higher-quality text representations through smoothing attributes along the edges. With the development of multiple text graph construction methods, some works began to lean toward free text as the object of analysis. More methods focus on constructing heterogeneous graphs, using words, topics, and other elements as bridges to realize the information propagation [Yao et al., 2019; Liu et al., 2020; Hu et al., 2019; Chen et al., 2020; Ye et al., 2020]. In addition, the work of [Zhang et al., 2020] use graph neural networks to capture the information of the internal structure of the text (e.g., co-occurrence). Some studies also construct text or online word graphs to alleviate the problems of sample imbalance and poor scalability [Tian et al., 2020; Huang et al., 2019].

However, existing methods often use heterogeneous structures to model free text, which greatly increases the parameter scale. Meanwhile, the construction method is mainly based on empirical design, without theoretical guidance.

2.2 Semi-Supervised Text Classification

The prior semi-supervised methods can be roughly divided into two categories: latent variable model and embeddingbased model. The former expands the topic model based on labeled samples and then relies on the posterior topic-label distribution for label prediction. The latter uses the given labeled samples as a seed to derive the embedding and labeling of the pending text. Among them, some semi-supervised methods introduce graphs as the basis for derivation. For example, PTE [Tang et al., 2015] first represented the labeled information and different levels of word co-occurrence information as a large-scale heterogeneous text network, which is then embedded into a low dimensional space through an efficient algorithm. HAN [Wang et al., 2019] and HGAT [Hu et al., 2019] designed heterogeneous graph to achieve more flexible feature capture. EGNN-Proto [Lyu et al., 2020] utilize an edge-labeling graph neural network to implicitly models the intra-cluster similarity and the inter-cluster dissimilarity of the documents

However, graph-based semi-supervised methods generally do not consider the derivation strategy [Gururangan et al., 2019; Meng et al., 2018; Miyato et al., 2019] in classic semisupervised methods. Hence, we propose a novel graph-based data augmentation technique and corresponding graph neural network structure to fill this gap.

3 Methodology

In this section, we will introduce the proposed pipeline framework TextGTL, as shown in Figure 2.

3.1 ProblemDefinitions &Notions

Semi-supervised text classification is a common natural language processing task. Given a text corpus D = {d1, d2, · · · , dn}, where n = |D|, and di denotes a document. Among D, every text di has a corresponding label yj ∈ C, where C = {c1, c2, · · · , cnc } is the label set, and nc = |C|. X ∈ Rn×dem is the attribute set of D, xi ∈ X denotes the initial attributes of di, and dem is the deminsion of attributes. Y ∈ Rn is the label set of D, and due to the semi-supervised setting, |Yobs| ≪ |Y - Yobs|, where Yobs is the training set.

There exist settings for this task, namely transductive learning and inductive learning, which are different in information visibility. While the former can fully observe and utilize X in both learning and inference stages, the latter is blocked from partial information during the learning stage but are fed with the complete dataset during testing. In this paper, we focus on transductive learning.

3.2 Non-Heterogeneous Text Graph Construction

In this section, we will explore how to construct meaningful text graphs and propose novel strategies to associate text. Unlike heterogeneous graphs that use words, topics, etc., as bridges to associate documents, we intend to use different linguistic information to consider whether the information is propagated between documents. To capture various information, TextGTL evaluates the propagation probability from three different linguistic perspectives as follows.

Semantics Text Graph

Inspired by the task of unsupervised semantic textual similarity, we design meta-document embedding to integrate multiple pre-trained document vectors based on various methods to represent the documents, and then determine if the association exists [P¨orner and Sch¨utze, 2019]. Given zero-mean random vectors v1, and v2, Canonical Correlation Analysis (CCA) finds linear projections θ1 and θ2 such that θ⊤ 1 v1 and θ⊤ 2 v2 are maximally correlated. [Bach and Jordan, 2002] showed that CCA reduces to a generalized eigenvalue problem, and Generalized CCA (GCCA) generalizes CCA to K random vectors v1, v2, · · · , vK ∈ Rdemk . Then, the generalized eigenvalue problem finds scalar-vector pairs (ρ, θ) that satisfy

where ΣK,K is the covariance matrix and ΣK,1 is the crosscovariance matrices. For stability, we add τσkIk, k ∈ [1,K] to every Σk,k, where τ = 0.1 is a hyperparameter, Ik is the identity matrix and σk is the average variance of vk. We stack the eigenvectors of the top-dem eigenvalues into operator matrices Θk ∈ Rdem×demk and define the GCCA meta-

where hdm,k ∈ Rdemk is the kth kind of pre-training initial document representaiton of dm. For each document pair (di, dj) to be determined for establishing an association or not, we use a simple classifier as follows:

where y(di,dj) denotes adding an edge between (di, dj) or not, ReLU(·) is an activation function, α is a trainable parameter, and || denotes concatenation. The positive and negative examples used in training are derived from the classification training set sample combination. Then, we can minimize the cross-entropy loss for data distribution for training

When the edge between (di, dj) is determined to be added, we will use wsem i,j = σ(hGCCA di · hGCCA⊤ dj ) as its initial weight and normalize it on whole graph.

Syntax Text Graph

Dependency parsing is considered to be useful for guiding the understanding of sentences in a document, so we are ready to calculate the contact ratio of syntax dependency between documents. First, for the document di, we use the parser of Stanford CoreNLP to get the smallest dependency unit (i.e., [word, dependency-relation, word]) that exists in the document and generate a set. Next, like the bag-of-words, we represent all documents as a vector hsyn di = [0, · · · , nj, · · · , 0], where nj is the number of jth dependency unit in di. Similar to the construction method of the semantics text graph, we also build a classifier to add edges and determine the weight of edges wsyn i,j , so we omit it.

Context Text Graph

The sequential context describes the language characteristics of local co-occurrence between words, which has been widely used in text representation learning. In TextGTL, to evaluate the sequential context similarity between two documents, we design a sliding-window-based calculation method as

where wm and wn are a pair of words which simultaneously appear in both di and dj, and p(wm) is the probability of the word wm appeared in a sliding window. p(wm,wn) is the probability of the word pair (wm, wn) co-occurring in the same sliding window, which can be estimated as the fraction of the total number of sliding windows over di and dj and the number of times that wm and wn co-occur in the same sliding window.

In the end, as shown in the Figure 2, we obatin three independent text graphs constructed based on different linguistic information.

3.3 Topology Refinement &Data Augmentation

Based on the widely-adopted perspective of existing graphbased semi-supervised models, i.e., attributes exhibit smoothness along the graph edges [Stretcu et al., 2019; Xu et al., 2020; Li et al., 2018; Li et al., 2020], some works propose a criterion to assess both training samples and given graph topology, which is highly correlated to the subsequent classification performance. Criterion C: The higher the consistency of the node information (i.e., attribute and label) within the same dense subgraph, the more reasonable and sufficient the information propagation. This criterion is highly consistent with the basic assumptions (i.e., graph homogeneity) of traditional label propagation algorithms, and has been proved by mass experiments, so we use it as our guide to refine text graph topology.

Subgraph Density

Specifically, when we refine the three text graphs' topology respectively, we first need to reasonably strengthen the density of the subgraphs in each text graph (All operation in each of the three text graphs is independent, and we take one as an example.). As shown in Figure 3, through the structure-based overlapping clustering algorithm [Epasto et al., 2017], we obtain the dense subgraphs in the text graph, which are regarded as the range that the information propagation should cover. However, due to the existence of long-distance dependence, information often cannot be propagated adequately in GNNs that only stack about two or three layers. This is the reason why some nodes' representations are under-smoothing.

Therefore, by generating super nodes and connecting them with corresponding subgraph members, we limit the maximum distance of nodes in the subgraph to 2, which will effectively alleviate the phenomenon of under-smoothing. In particular, Dropedge [Rong et al., 2020] shows that too dense the graph structure is also not conducive to deep node representation learning. This is because in a graph with too high connectivity, information propagation becomes easy, which can easily lead to the convergence of nodes in the entire graph, i.e., the problem of over-smoothing. Hence, we randomly remove some edges in the text graphs. The weights of new edges are also calculated from section 3.2.

Information Consistency (Data Augmentation)

From another perspective, improving the consistency of node information within the subgraph can also improve the performance of the downstream tasks. As the sum of all nodes of the corresponding cluster, the super node can be regarded as multiple sampling of the same label distribution. This operation is similar to mixup [Zhang et al., 2018], but different from augmenting the marginal sample to improve the model in a supervised setting, our goal is to obtain samples with higher class separability in a semi-supervised setting. Specifically, according to the Law of Large Numbers [Pr´ekopa, 1972], it should fit the original distribution more closely than the original node [Li et al., 2020]. On this basis, as shown in Figure 3, some super nodes can be directly labeled based on their containing training samples in clusters. By introducing a simple classifier trained by such high-quality labeled super nodes, we can obtain more super nodes with high-quality features and low-entropy labels, and provide more training data for downstream models, which realizes data augmentation.

3.4 Joint Training onMultiple Text Graphs

After obtaining the three updated text graphs and the augmented training sample set, we focus on exploring an efficient graph neural network framework to learn document representation and achieve classification. We consider that, on the one hand, crudely compressing three different text graphs together will inevitably confuse the various structure information and lose the freedom of the model. On the other hand, the basic principle of graph neural network learning is that nodes propagated information to each other and update their feature embeddings by propagating information among other neighboring nodes. Therefore, we attempt to transmit information within the text graph independently while sharing information between different text graphs to promote a more flexible propagation of information throughout the model.

We stack a 2-layer GCN. Among them, the first layer is a general GCN [Kipf and Welling, 2017], and three graphs are fed independently. The first layer of each text graph is the same and independent, as follows:

where H(1) denotes the hidden representation matrix of D at the 1st layer, H(0) is the initial embedding of D, W (1) is a learnable linear transformation matrix, and σ(·) is an element-wise nonlinear activation function. Among them, ˜ A = M - 1 2 (A + I)M - 12 represents the symmetric normalized adjacency matrix, A, and M are the weighted adjacency matrix and degree matrix. Next, by combining hidden representations from three text graphs, the second layer as follows:

In the last layer of TextGTL, we perform a mean pooling over graphs to obtain the final representation of documents for classification.

4 Experiment

4.1 Datasets &Baselines

We choose a suite of recently widely used benchmark datasets for experiments and analysis. We conduct extensive experiments on 5 benchmark text datasets: 20-Newsgroups dataset, Ohsumed dataset, R52 Reuters dataset, and R8 Reuters dataset, and Movie Review dataset. These datasets involve many life genres, such as movie reviews, medical literature, and news documents, etc. The Movie Review dataset is designed for binary sentiment classification. The 20Newsgroups dataset, R52 Reuters dataset, R8 Reuters dataset are news classification datasets. The Ohsumed dataset is medical literature. To evaluate the performance of the semisupervised learning model, we gave 20 labeled data per class. We split the same number of samples of the verification set as the training set from the dataset, and ensure that there is no sample imbalance in the testing set. A summary statistics of the benchmark datasets is presented in Table 1.

To comprehensively evaluate our proposed method for semi-supervised text classification, we compare it with the following three groups of state-of-the-art methods: SVM [Blei et al., 2003]: SVM classifiers using TF-IDF features. CNN [Kim, 2014]: CNN classifiers using averaged word embeddings as sentence embedding, whose word embeddings are pre-trained with Wikipedia Corpus based on GloVe. LSTM [Iacobacci and Navigli, 2019]: A LSTM sentence encoder using pre-trained word embeddings as CNN. PTE [Tang et al., 2015]: A graph-based semi-supervised representation learning method for text data. PTE firstly learns word embedding based on the heterogeneous text networks containing three bipartite networks of words, documents, and labels, then averages word embeddings as document embeddings for text classification. TextGCN, TensorGCN [Yao et al., 2019; Liu et al., 2020]: A GCNbased methods on sequential contextual graph and multiple text graph. HAN [Wang et al., 2019]: It embeds heterogeneous information network by first converting a heterogeneous information network to several homogeneous subnetworks through pre-defined meta-paths and then applying graph attention networks. To be fair, all of our models use a uniform pre-training vector as the initial representation of the document. The results of models with ∗ come from original code or replication.

4.2 Experiment Settings

We use K = 4 kinds of document initial embeddings in the semantics text graph that are derived from [P¨orner and Sch¨utze, 2019], set the length of sliding-window as 20, and the dependency parser used in the syntax text graph is Stanford CoreNLP. We equip TextGTL with the EgoSplitting [Epasto et al., 2017] as an overlapping clustering algorithm (resolution=1.0). During data augmentation, we select GBDT as a simple classifier, and iterated 10 rounds to obtain more labeled super nodes to add to the training set. As mentioned in section 3.3, in this study, we adopt a two-layer GCN to achieve. The initial features of the first-layer nodes in the three different text graphs are all TF-IDF attributes from TfIdfcounter in sklearn, the output feature dimension is 200, the input feature dimension of the second-layer nodes is 200×3, and the output feature is the number of classes. During the training process, the dropout rate is 0.5, and the L2 loss weight is 5e-6. We use Adam optimizer with a maximum of 200 epochs and a learning rate of 0.002, when the verification loss is not reduced for 10 consecutive epochs, an early stop will be executed. All results reported in this study are the average of 10 independent tests.

4.3 Results Analysis

A comprehensive experiment is conducted on the benchmark datasets. The results presented in Table 2 show that our proposed TextGTL significantly outperforms all baselines (including some state-of-the-art sequential learning and graphbased models). Among them, we notice that the sequential learning method is generally lower than the graph-based method in corpus sets with a larger average text length, which we suspect may be due to the limitation of the representational ability of the sequential model when processing multiple sentences.

4.4 Discussion of Text Graphs

We discuss the specific performance of the text graph proposed in this paper by testing the effectiveness of the three text graphs. The results are shown in Table 3. We notice that the performance of any single text graph is lower than the result of joint training as shown in Table 2, which shows that the three text graphs are complementary in information propagation. In addition, after statistics, the edges and nodes in the text graphs of TextGTL are far less than the existing heterogeneous graph methods.

4.5 Ablation Analysis

Finally, we performed an ablation analysis of the components in our proposed TextGTL to confirm the improvement of model performance brought by different components. The experimental results are shown in Table 4. We notice that the performance of the model based on only three original text graphs is only competitive, which is slightly lower than TextGCN. We guess the main reason is the undersmoothing of information propagation and noise in the text graphs. However, each of the remaining components has brought certain improvements to the performance of the model, especially topology refinement and data augmentation. It proves that Criterion C correctly guides to topology refinement.

4.6 Impact of Number of Labeled Data

To explore the performance of different models given different amounts of labeled samples, we experiment with TextGTL and baseline methods on the MR dataset. The experimental results are shown in Figure 4. With the increase of labeled documents, all methods have achieved better results in terms of test accuracy. And under different settings, TextGTL always outperforms all other methods. If fewer labeled documents are provided, the performance of the baseline will be significantly reduced, and our model can still achieve higher performance. It illustrates that our proposed TextGTL can more effectively use limited label data for text classification.

5 Conclusion

In this work, we propose a pipeline framework Text-oriented Graph-based Transductive Learning to build text graphs, refine topologies, data augmentation, and design GCNs that share hidden layer information for semi-supervised text classification. Experimental results show that TextGTL can effectively improve the performance of semi-supervised text classification. In addition, we further discussed the performance improvement brought by different strategies in ablation analysis. In the future, we will continue to explore more development possibilities for graph-based text classification tasks.

Acknowledgements

The corresponding author is Jianxin Li. This work was supported by the NSFC program (No. U20B2053, 62002007 and 61772151), S&T Program of Hebei through grant 20310101D, and SKLSDE-2020ZX-12.