Correlation-Guided Representation for Multi-Label Text Classification

1 Introduction

Multi-label text classification (MLTC) deals with real-world objects with rich semantics, where each text is simultaneously associated with multiple class labels that tend to be correlated. It is a fundamental task in natural language processing (NLP), which aims to learn a predictive model that assigns an appropriate set of labels to an unseen text. It is worth noting that to learn from multi-label text data, one needs to pay attention to two key factors: 1) How to generate more discriminative text representation; 2) How to effectively mine correlations to facilitate the learning procedure.

∗Equal contribution.

†Work done during an internship at Tencent.

Text representation is critical in multi-label text classification. Transformer-based studies [Devlin et al., 2019; Lan et al., 2019] demonstrate the effectiveness of Transformer module for capturing the dependencies of all words in a sequence and provide a contextualized representation for classification tasks. Nevertheless, utilizing only contextual information to generate the text representation is suboptimal as it ignores the information conveyed by class labels and thus fails to take advantage of potential correlations among label-label and word-label. The fact that different labels may share the same subset of words is beneficial to help generate strong text representation. For example, academic literature containing keywords such as "neural network" is often tagged with "artificial intelligence" and "deep learning". Closely related labels tend to co-occurr. Therefore, it is rather desirable to fully exploit potential correlation information in text representation generation, which could be investigated in two-fold: 1) On one hand, label-label correlations can be exploited to extract latent inter-dependent features; 2) On the other hand, context-label correlations can be exploited to enhance discriminative abilities of extracted features. As far as we know, the simultaneous exploitation of both correlations has still not been well studied.

Generally, the learning process must be facilitated by exploiting correlations among labels in order to tackle the challenge of an exponential-sized output space for MLTC. Specifically, CNN-RNN [Chen et al., 2017] presents an ensemble method of CNN and RNN to capture semantics and model label correlations. SGM [Yang et al., 2018] captures high-order correlations between labels through the sequence generation model. We argue that correlations change dynamically in different contexts, so if we can learn words and labels jointly in the same space, we will get better label-label correlations as well as context-label correlations that fit a text. To further model the context-label correlations, several label embedding methods, including C2AE [Liu et al., 2017], LEAM [Wang et al., 2018], LSAN [Xiao et al., 2019], X-Transformer [Chang et al., 2020], etc., are proposed to take advantage of label information and construct label-specific text representation through the refinement of the word embedding. However, they fail to provide implicit information among label space, which leads to the prediction bias in favor of the majority classes while ignoring the minority classes. Furthermore, such methods are limited to a certain extent when the labels do not carry semantic description information. As an example, "deep learning" is a label with description, but the symbol "DL" has no description. Providing labels with abbreviations or symbols in a dataset can lead to poor prediction performance or inapplicability.

Inspired by the potential of correlations, we import label semantics as auxiliary information by a global embedding strategy. The encoder learns word-word, label-label, and word-label correlations globally through Transformer module. Since not all text words contribute equally to the prediction, we construct an attention vector from the word-label correlation matrix to extract more discriminative words. The attention mechanism can improve performance with interpretability for text classification, which means that it helps relevant words to get higher attention than irrelevant words. To the best of our knowledge, we are the first to learn label-label and context-label correlations together with an attention-based two-step framework, and the main contributions of this paper include:

1. We propose a basic global embedding strategy that represents context and all class labels in the same latent space by Transformer to generate token-level representations, which captures correlations and reduces the dependence on label description information.
2. We propose an effective and novel method, called CORE, which exploits COrrelation-guided REpresentation for multi-label text classification. CORE utilizes higherorder context-label correlations to guide attention processes and attempts to produce a semantic-aware representation.
3. Experimental results show that CORE achieves competitive performance against other state-of-the-art multilabel text classification approaches. We further provide a series of BERT-based methods and analyze the performance with macro-based and rank-based metrics. Results show that the utilization of label embedding and label correlations have a significant impact on the performance of our approach.

2 The CORE Approach

In this section, we first introduce the standard formal definition of multi-label text classification. Afterwards, the formulation of our method is illustrated. The technical details of CORE are detailed in three steps, including global embedding strategy, text representation learning, and predictive model induction.

2.1 ProblemFormulation

Given a training set S = {(Xi, Yi)}Ni of multi-label text classification data, where Xi ∈ X is the text sequence and Yi ⊆ Y is its corresponding labels, the task of MLTC is to learn a predictive function f. More specifically, an input text sequence X of length m is composed of word tokens: X = {x1, x2, ..., xm}, and Y = {y1, y2, ..., yl} is the label space with l labels. Different from the single-label classification where only one label is associated with Xi, the multilabel classification function f : X → 2Y assigns a set of possible class labels (Y , 0 ≤ |Y | ≤ l) for the unseen text. Here, yi is either regarded to be relevant (y ∈ Y ) or irrelevant (y /∈ Y ) for instance X. Note that we use ki ∈ {0, 1} to denote the categorical information of yi.

A typical text classification approach first preprocesses text data X for the model to obtain text representation X. Then, the classifier annotates the text representation with a set of proper labels Y . Intuitively, the approaches utilize only the information from the input text sequence. Our method extends the input by adding label information. Therefore, the new input sequence of CORE is overlaid with both text and labels, which is composed of all tokens like: {X; Y} = {x1, x2, ..., xm; y1, y2, ..., yl}, the number of labels is fixed to l in the data. The preprocessing is to obtain text representation C from context and labels. The aim of the predictive function f : C → 2Y is to minimize a loss function which ensures that the model predicts relevant and irrelevant labels for each training instance with minimal misclassification.

2.2 Global Embedding Strategy

We utilize BERT [Devlin et al., 2019], which outperforms state-of-the-art models on a wide range of NLP tasks, as the base encoder in the CORE framework. The basic architecture of BERT is a multi-layer bidirectional self-attention Transformer. For classification tasks, a special token [CLS] is put to the beginning of the text and the output vector of the token [CLS] is designed to correspond to the final text representation. Different from this operation, we unite the input text with all class labels, which are packed into a single sequence and separated by a special token [SEP].

Let {[CLS], x1, x2, ..., xm, [SEP], y1, y2, ..., yl, [SEP]} be the token sequence which input into Transformer module. Note that the input representation of each given token is constructed by summing the corresponding token, segment, and position embeddings. We simply use the same pre-trained model parameters as the official public model to initialize our model, and finetune all the parameters end-toend to obtain the contextualized token-level representations H, i.e., {h[CLS], hx1, ..., hxm, h[SEP], hy1, ..., hyl, h[SEP]}. As shown in Figure 1, the utilization of global embedding strategy guarantees that we consider both label correlations and context-label correlations in the same space at the beginning. Note that when label descriptions are unavailable, we represent each label with a new word token (unused token) to learn the hidden representation.

2.3 Text Representation Learning

To characterize the underlying structure of the contextualized representations, CORE works by constructing an attention vector (Hx,Hy) → -→α . Hx corresponds to the set of context sequence representations and Hy corresponds to the set of label sequence representations. Since the input text is often flexible, we fixed the length for ease of use, i.e., the excess is trimmed off and the deficient is padded.

Attention Discovery

A simple way to measure the deep context-label correlations is to multiply matrix Hx by matrix Hy:

where G ∈ Rm×l , note that Hx and Hy have been normalized by L2 norm.

We consider a further generalization of Eq. (1), which aims to strengthen the relative spatial information among consecutive tokens. In particular, for a text fragment of length 2r +1 centered at p, the local matrix block Gp-r;p+r in G measures the correlation in the word-label fragment pairs. Since the matrix G can be viewed as an extension in the spatial orientation, we use g(·) to denote a convolution layer with ReLU activation to learn the higher-order correlation matrix:

where M ∈ Rl×m, p ∈ {1, ...,m}. Here, W1 ∈ R2r+1 and b1 ∈ Rl represent weight matrix and bias vector respectively. We compress the matrix M into a vector of length m by selecting the maximum value and reduce the effect of the fluctuating value: -→α = Ω(M) (3)

Here, the length of -→α is m, softmax and hyperbolic tangent are executed sequentially in Ω(·). By learning the contextlabel correlation matrix M from Eq.(2), the attention vector -→α can be instantiated in a manageable range in Eq.(3).

Text Representation Generation

Given the attention vector -→α , the original contextualized representations of text sequence can be transformed into an enriched version. In CORE, the final text representation is generated by aggregation of word representations, weighted by attention vector: -→c = -→α · Hx (4)

Intuitively, the text representation uses higher-order contextlabel correlations to guide attention processes. The nonlinear interaction between context and labels has been adequately considered to improve the performance.

2.4 Predictive Model Induction

According to the objective function f : C → 2Y , the correlation-guided representation replaces the original text representation for multi-label prediction. We choose standard neural networks to annotate the correlation-guided representation with a set of relevant labels:

where W2 ∈ Rl×|-→c | and b2 ∈ Rl are parameters to be learned. Notice that, the sigmoid function allows to deal with non-exclusive labels, while the softmax function only deals with exclusive classes.

In CORE, binary cross-entropy losses are used to measure probability errors in multi-label classification tasks where each class is independent, rather than mutually exclusive:

In order to minimize the loss function, we train the model end-to-end with all above parameters.

3 Experimental Setup

In this section, the datasets, comparing algorithms, evaluation metrics and parameter settings are introduced.

3.1 Datasets

We use two datasets for MLTC: AAPD [Yang et al., 2018] and RCV1-V2 [Lewis et al., 2004]. Table 1 summarizes the detailed characteristics of the two datasets. Each dataset is divided into a training set, a validation set, and a test set, which are used as basic divisions in the performance experiments of each algorithm [Yang et al., 2018].

3.2 Comparing Algorithms

The performance of CORE is compared against the following multi-label algorithms:

• SGM [Yang et al., 2018] proposes the sequence-tosequence model with an attention mechanism to capture label correlations. Although label correlations are exploited, it ignores the use of label semantics to construct text representations.
• Seq2Set [Yang et al., 2019] utilizes deep reinforcement learning to improve the performance of seq2seq model, which reduces the dependency of the label order. Similar to SGM, it lacks the use of label information.
• LSAN [Xiao et al., 2019] makes use of content and label text to learn the label-specific text representation with the help of self-attention and label-attention mechanisms.

• LEAM [Wang et al., 2018] applies label embedding in text classification, which obtains each label's embedding by its corresponding text descriptions. Here, we provide a comparable baseline method LEAMw/BERT , which uses BERT to provide text encoding, but its labels are encoded independently.
• BERT [Devlin et al., 2019] is a recently proposed language representation model that generates contextualized word vectors. For multi-label text classification, it only uses text as input, no label information. As is shown in Figure 2, we propose two comparable baseline methods. BERTonelab is input only one label at a time, and divides l labels into l times to perform the binary classification task. While incorporating label embedding, no label correlations can be learned. BERTlabseq uses the output of the part of label sequence Hy as multiple contextual representations for multiple binary classification. It considers our global embedding strategy and

solves the problem as a sequence annotation task with an additional BiLSTM-CRF layer.

More information about other baselines can be found in Binary Relevance (BR) [Boutell et al., 2004], CNN [Kim, 2014], Classifier Chains (CC) [Read et al., 2011], Label Powerset (LP) [Tsoumakas and Katakis, 2007] and CNN-RNN [Chen et al., 2017].

3.3 EvaluationMetrics

Following the previous work [Yang et al., 2018; Zhang and Zhou, 2014], we adopt Hamming Loss and Micro-F1 as our main metrics, the micro-precision and micro-recall are also reported for reference. We further used Macro-F1 assuming equal label weights as the key metric, which provides a different analytical perspective. The macro-precision, macro-recall are also reported for reference. When the intermediate realvalued function is available, macro-AUC, ranking loss, and coverage are provided as rank-based metrics, which expect related labels to score higher than the unrelated labels. For each evaluation metric, "↓" indicates "the smaller the better", while "↑" indicates "the larger the better".

3.4 Experimental Setting

We implement our experiments in Tensorflow on NVIDIA Tesla P40. In the experiments, we fine-tuned models on the base-uncased versions of BERT for English texts. The batch size is 32, the learning rate is 5e-5 , and the window size of additional layer is 10. Based on WCard(S) and L(S) in Table 1, the maximum total input sequence length is 320. In addition, learning rate decay is added to the BERT training part, which starts with a large learning rate and then decays multiple times [Clark et al., 2019]. Note that all BERT-based models in this paper use learning rate decay technique to improve performance.

4 Experimental Results

We report the detailed experimental results of all comparing algorithms on two datasets in Table 2. The following observations can be made according to the results:

1) Our proposed CORE presents the best performance in terms of hamming loss and micro-F1. We perform a significant test among the comparing algorithms suggesting that performance is statistically significant (p < 0.05). On AAPD dataset, compared to the traditional deep learning model CNN which only considers text content, CORE decreases by 17.97% on hamming loss and improves by 12.95% on microF1. As for BERT, CORE continues to perform well, with a relative reduction of 6.25% on hamming loss and an improvement of 2.18% on micro-F1. On RCV1-V2 dataset, compared with Seq2Set which only uses label semantics for corrected predictions, CORE achieves a reduction of 5.48% on hamming loss and an improvement of 0.91% on micro-F1. It shows that modeling correlations with label semantics can lead to performance gains.

2) With our global embedding strategy, BERTlabseq has made a significant improvement in micro-recall compared to BERT. We argue that the potential correlations among labellabel and word-label can help capture more meaningful features. BERTonelab predicts labels one by one, but compared to BERT, it achieves a reduction of 3.00% micro-F1 score on AAPD dataset. LEAMw/BERT improves LEAM by Transformer, but its performance is rather slightly lower than BERT because the labels are encoded independently. It indicates that the lack of label correlations may lead to performance degradation.

3) Algorithms like CNN, BERT, LSAN, etc., are biased to predict positive examples as negative examples, resulting in fewer matches than the actual number of samples in each class. CORE ensures good micro-precision while improving micro-recall. We attribute this phenomenon to the effect of the attention mechanism. According to the above observations, it is noteworthy that no algorithm significantly outperforms CORE across all evaluation metrics.

To summarize, comparing the proposed CORE against the recent state-of-the-art models, our method significantly improved previous state-of-the-art results in the main metrics.

5 Further Analysis

In this section, several studies are used to argue intuitively that CORE has good capability to learn high-order corrections and generate correlation-guided representation with competitive convergence speed. Moreover, CORE can classify each class well, even if that class is low-frequency.

We provide the macro-based and rank-based metrics in Table 3 to quantify the prediction performance from different analytical perspectives. Our proposed CORE shows the best performance on macro-F1, which proves that our method effectively improves the performance of all classes. In addition, BERTlabseq, which we proposed to validate the global embedding strategy, has the best performance on rank-based metrics. The direct use of label sequence representations has a more primitive preservation of label-label correlation, which favors relevant labels to rank higher than irrelevant ones. However, this method has weak word-label correlation and is prone to misclassification.

Alternatively, we divide labels on AAPD into three groups according to their occurring frequency. Nearly 56% of labels appearing more than 60 times are high-frequency labels and form Group1. Labels appearing 15-60 times form Group2 (34%), and the remaining 10% of labels formGroup3. Figure 3 shows that all algorithms perform better on high-frequency labels (Group1) than on low-frequency labels (Group3), which is reasonable since there are more samples of highfrequency labels. More significantly, CORE improves macroF1 on Group2 and Group3 compared to other methods, and it is more robust to classify mid/low-frequency labels. These results demonstrate the superiority of our proposed models in predicting low-frequency labels.

The convergence speed of three BERT-based models are shown in Figure 4. Both CORE and BERTlabseq outperform BERT in terms of convergence speed. CORE converges significantly faster than BERT, which means that the performance of our proposed CORE can approach the optimal solution more efficiently by global embedding strategy and text representation learning.

In Figure 5, we visualize one example on the AAPD dataset. The dark orange means more important words. For the label "CV" and "CL", the selected informative words are videos, image, movie, descriptions, etc. For the label "CV" and "LG", the selected informative words are images, machine, etc. Benefiting from the interpretability of the attention mechanism, the text representation learning can correctly detect the key words with proper scores.

6 RelatedWork

Text representation plays a significant role in model performance. It is crucial to extract essential hand-crafted features for early models [Joachims, 1998], but features can be extracted automatically by DNNs. During the past decade, DNNs have been employed progressively in classification tasks by learning a set of nonlinear transformations that serve to map text directly to outputs, such as CNN [Kim, 2014]. Each word in the text is represented by a specific vector obtained through the word embedding technique [Joulin et al., 2017]. Recently, Transformer, which is proposed by [Vaswani et al., 2017], relies entirely on an attention mechanism to draw global dependencies between input and output. [Devlin et al., 2019] is an important turning point in the development of text classification task and it works by generating contextualized word vectors using Transformer. [Yang et al., 2018; Yang et al., 2019] use the sequence-to-sequence model that consists of an encoder and a decoder connected through an attention mechanism. [Pang et al., 2021] performs the fewshot learning for text classification. To further manipulate the geŶeratiŶg desĐriptioŶs for ǀideos has ŵaŶLJ appliĐatioŶs iŶĐludiŶg assistiŶg ďliŶd people aŶd huŵaŶ roďot iŶteraĐtioŶ the reĐeŶt adǀaŶĐes iŶ iŵage ĐaptioŶiŶg. · referenĐe laďels: ͞CV͟ aŶd ͞CL͟

Đoŵpared to ŵaĐhiŶes , huŵaŶs are edžtreŵelLJ good at ĐlassifLJiŶg iŵages iŶto Đategories , espeĐiallLJ ǁheŶ theLJ possess prior kŶoǁledge of the Đategories at haŶd if this prior iŶforŵatioŶ is Ŷot aǀailaďle. ǁe propose aŶ iŶteraĐtiǀe ŵaĐhiŶe teaĐhiŶg algorithŵ that eŶaďles a Đoŵputer to teaĐh ĐhalleŶgiŶg ǀisual ĐoŶĐepts to a huŵaŶ our adaptiǀe algorithŵ Đhooses

• referenĐe laďels: ͞CV͟ aŶd ͞LG͟

Figure 5: Visualization of attention mechanism on the AAPD dataset. The reference labels are displayed after abstracts. Note that "CV", "CL", and "LG" denote computer vision, computational language, and machine learning, respectively.

feature space, [Sun and Zhang, 2021] exploits distance metric to generate discriminative meta-level features.

Label embedding techniques are gaining attention and application in large-scale heterogeneous networks [Tang et al., 2015], multi-class learning [Joshi et al., 2017], and label distribution learning [Peng et al., 2018]. [Pappas and Henderson, 2019] propose a non-linear transformation to capture the relationships across labels, which performs significantly better on unseen labels. [Liu et al., 2017] is the first DNN-based multi-label embedding method that seeks a deep latent space to jointly embed the instances and labels. [Chang et al., 2020] considers the extreme multi-label text classification (XML) problem and performs label embedding via label text or positive instances.

7 Conclusions and FutureWork

In this paper, we present the CORE approach, which exploits correlation-guided representation for multi-label text classification. We first introduce the global embedding strategy which learns high-order corrections between context and all class labels in the same space. Then, the attention mechanism is used to highlight the most informative words in the text sequence. Extensive comparative studies clearly validate the superiority of our proposed CORE against state-of-the-art multi-label classification algorithms.

Our method treats all class labels as a label sequence, which means that our default labels are ordered. However, it can also be treated as an unordered set. On the other hand, one label is virtualized as one single token. If the label has descriptive text, there could be multiple tokens for semantic learning, which might be useful for XML problems. The above issues should be further explored in the future.