Guided Attention Network for Concept Extraction

1 Introduction

Concept extraction aims at extracting words or phrases describing a concept (e.g., logistic regression, hash function, and infinite set) from a given corpus (e.g., research papers and textbooks). It plays an essential role in constructing knowledge bases [Song et al., 2019], transforming unstructured text into structured information [Shang et al., 2018], and supporting downstream analytical tasks [Xiong et al., 2017; Huang et al., 2017; Liu et al., 2018].

Most previous concept extraction approaches operate in an extractive manner [Gelfand et al., 1998; Pan and Wang, 2017], which usually consists of two steps: 1) Selecting important text spans as candidate phrases based on statistical do not provide information about the human decision-making process when recognizing a concept. Similar to the way the topic and clue words guide our recognition process, we argue that they can also guide the model to make decisions. Specifically, we define "clue words" as a group of words that can help explain the recognition process of a particular concept in the same sentence. The clue words can be regarded as a form of expression of domain knowledge and they are concise, interpretable, and does not rely on much training data to generate (e.g., clue words such as "is a", "is called an" in textbooks, and "we introduce/propose/study/explore/describe" in scholarly documents).

∗ Contact Author

However, there are many challenges for concept extraction using structured information (title, topic, clue words). First, title and topic as global information play the leading role in concept extraction. The key concept phrases have close semantics to topic [Liu et al., 2010]. Thus, how to combine the global information to pay attention to the topic-related words according to the semantic relatedness plays a crucial role in the recognition process. Second, it is difficult to collect a complete set of clue words. Clue words have large numbers of linguistic variants, which makes it difficult to generalize them to unseen sentences that are semantically equivalent but having different word usage. For example, if "discuss" is not included in the collection of clue words but its synonym "talk about" is included. When we perform exact matching with clue words we have collected for the sentence "In the previous, we have discussed the hash function.", the "discussed" fails to match with its synonyms "talk about". Third, there is an implicit relation between clue words and concept. Humans can recognize concepts through accumulated experience, but it is difficult for the machine. How to model the relationship between clue words and concept is challenging.

To address these challenges, we propose GACEN based on Long-Short TermMemory (LSTM) network. In GACEN, to incorporate the topic information into the feature representation, we design a shared topic-based encoder to model the title and main body of the document with topic vectors at the document- and word-level separately. To solve the problem of variants of clue words and improve the generalization ability, we pre-train a soft matching module with neural networks to capture semantically similar words. Then, we design two attention modules, one of them is to gather the relevant global topic information for each context word according to the semantic relatedness based on topic enhanced representation, and the other aims to model the complex implicit relationship between clue words and concept with the semantic and position information of clue words. Finally, we aggregate information from two attention networks as input to Conditional Random Field (CRF) to model dependencies in the output. We crowd clue words for three well-studied datasets. Extensive experiments demonstrate the effectiveness of our model, especially when the labeled data is insufficient.

2 RelatedWork

Concept Extraction. Concept extraction has been studied intensively in previous works. Conventional extractive methods usually use the two-step strategy that first extracts the candidate phrases using rules (hand-crafted or syntactic pattern matching) and then ranks them based on supervised or unsupervised methods [Liu et al., 2020; Pan and Wang, 2017; Gelfand et al., 1998]. The main drawback is that these methods mainly use statistical features, which are unable to capture the semantics behind the text. Recently, Deep Neural Networks (DNN) based models are used to solve concept extraction problems. Yang et al. [2020] applied an encoderdecoder framework to extract the concept directly from the text. Zhu et al. [2018] and Lange et al. [2020] formulated concept extraction as a sequence labeling problem. Yang et al. [2020] applied BERT to clinical concept extraction. They all showed that deep learning methods can improve the performance over baseline models for this task.

Keyphrase Extraction. Keyphrase extraction aims to extract phrases that provide a concise summary of a document. Generally, keyphrase extraction techniques can be classified into two groups: supervised methods and unsupervised methods. Among them, in unsupervised methods, keyphrase extraction is formulated as a ranking problem using different techniques, such as graph-based ranking [Liu et al., 2010], clustering [Liu et al., 2009] and language modeling. In supervised methods, keyphrase extraction is treated as a classification problem: each phrase in the document is either a key phrase or not. Recently, many researchers have studied the neural network-based solutions [Zhang et al., 2016; Chen et al., 2019]. Keyphrase extraction can be seen as generating top concepts for each document. Compared with concept extraction, there is a big difference in quantity.

Named Entity Recognition. Both NER and concept extraction can be defined as sequence labeling problems. NER aims at locating and classifying named entities into predefined categories from free text. The early works applied CRF, Support Vector Machine (SVM), and perception models with hand-crafted features [Luo et al., 2015; Downey et al., 2007]. With the advanced performance of deep learning, research has been shifting towards deep neural networks (DNN), which requires little feature engineering. Chiu et al. [2016] presented a bidirectional LSTM-CNNs architecture that detects word and character level feature. Chiu et al. [2019] further extended it into LSTM-CNNs-CRF architecture, where the CRF model was added to optimize the output label sequence. NER seeks to locate and classify named entities mentioned in unstructured text. This is quite different from our goal.

3 ProblemFormulation

We assume that we have collected clue words (the collection process of clue words is disscussed in Section 5.1). We let c = {𝑐1, 𝑐2, · · · , 𝑐𝑘 } denote the clue words set we have collected, 𝑐𝑖 represents the 𝑖-th clue words in this set. Given a sentence x = {𝑥1, 𝑥2, · · · , 𝑥𝑛} with corresponding title t = {𝑡1, . . . 𝑡𝑙}, 𝑥𝑖 represents the 𝑖-th word in the sentence and 𝑡𝑖 represents the 𝑖-th word in the title. Our goal is to predict labels y = {𝑦1, 𝑦2, · · · , 𝑦𝑛} for x based on c, t. For 𝑥𝑖 in the sentence, its label 𝑦𝑖 ∈ {𝑆 -𝐶𝑂𝑁, 𝐵 -𝐶𝑂𝑁, 𝐼 -𝐶𝑂𝑁, 𝐸 -𝐶𝑂𝑁,𝑂} indicating current word is a Single concept word, the Beginning of a concept, the Middle part of the concept, the End of the concept, or Out of a concept, respectively.

4 Our ProposedModel

In this section, we present our framework GACEN for concept extraction. As shown in Figure 1, the GACEN model includes four components: Topic-based Encoder, Soft Matching module, Attention Layer (Topic-aware Attention, Position-aware Attention), and CRF layer. Specifically, the topic-based encoder and Topic-aware attention module are used to gather the relevant information from the topic and title to each context word in the document. Soft matching and Position-aware attention modules are used to match the sentence with clue words and model the positional and semantic relationship between clue words and other words in the same sentence. The CRF is used to model dependencies in the output.

4.1 Topic-based Encoder

As we all know, both documents and words can be represented by a mixture of semantic topics. A word will play different important roles in different topics of the document. The concept words in the document are the key to representing the topic of the document. Thus, the topic distribution of key concepts and the document should be similar [Liu et al., 2010]. To incorporate the topic distributions into the text representation, we first get the topic distributions zd of documents and the topic distributions zw of words using Latent Dirichlet Allocation (LDA) [Blei et al., 2003] model, where zd, zw ∈ R 𝑘 and 𝑘 is the hyper-parameter indicating the number of topics in the document.

Meanwhile, we consider the title and main body of the document separately. As shown in Figure 2, given the input sentence x = (𝑥1, 𝑥2, . . . , 𝑥𝑛) where 𝑛 is the length of the sentence, and Corresponding title t = (𝑡1, . . . 𝑡𝑙) where 𝑙 is the length of the title. We first map the words in sentence and title to word embedding E𝑠𝑒𝑛. = ( e𝑥1 , e𝑥2 . . . , e𝑥𝑛 ) , E𝑡𝑖𝑡𝑙𝑒 = ( e𝑡1 , . . . e𝑡𝑙 ) by looking up word embedding matrix W𝑒 ∈ R 𝑑𝑒 . Next, concatenate each word vector in title with topic vector of document zd to get T = (e𝑡1 ⊕ zd, . . . e𝑡𝑙 ⊕ zd) and concatenate word vector in main body with topic vector of words zw to get X = ( e𝑥1 ⊕ zw1 , e𝑥2 ⊕ zw2 . . . , e𝑥𝑛 ⊕ zwn ) . ⊕ can be arbitrary operations, where we take as concatenation. Finally, a Bi-LSTM is used to obtain hidden states of title and main body, respectively:

The final hidden representation of position 𝑖 is:

Similarly, we can obtain the contextualized word representations v𝑖 for each token t𝑖 of title.

4.2 Soft MatchingModule

The soft matching module is used to match the corresponding clue words for the unseen sentences and locate where the clue words appear. Previous work implements this operation by regular expressions matching, which has a poor generalization ability because all synonyms and variations in a regular expression must be explicitly specified. To improve the generalization ability, we pre-train a soft matching module to enable capturing semantically similar words. Given an input sentence 𝑋 = {𝑥1, 𝑥2, · · · , 𝑥𝑛} and a clue words query 𝑞 from clue words set. The soft matching module 𝑓𝑠 (𝑋, 𝑞) returns a sequence of similarity scores 𝑆 = {𝑠1, 𝑠2, · · · , 𝑠𝑛} between each token 𝑥𝑖 and the query 𝑞. Inspired by [Li et al., 2018; Wang et al., 2019], for token 𝑥𝑖, we first extract 𝑁𝑐 different levels contexts for each token by sliding windows of different sizes. For example, if the maximum window size is 2, the different levels contexts of token 𝑥𝑖 are [𝑐𝑖0, 𝑐𝑖1, 𝑐𝑖2] , where 𝑐𝑖0, 𝑐𝑖1, 𝑐𝑖2 are [𝑥𝑖] , [𝑥𝑖-1; 𝑥𝑖] and [𝑥𝑖; 𝑥𝑖+1], respectively. Each of the different contexts 𝑐𝑖 𝑗 is encoded to a fixedlength vector zc𝑖 𝑗 by a shared Bi-LSTM encoder. Also the clue words query 𝑞 is encoded to a fixed-length vector by the same shared encoder and then are summarized as 𝑧𝑞 by an attention layer. Finally, we calculate scores of each token in the sentence 𝑋 and query 𝑞 by aggregating similarity scores from different sliding windows:

where D is a trainable diagonal matrix, 𝑣 ∈ 𝑅 𝑁𝑐 is the trainable weight of each sliding window.

In order to learn the parameters of the soft matching model, we construct training data in the form of (𝑠𝑒𝑛𝑡𝑒𝑛𝑐𝑒, 𝑞𝑢𝑒𝑟𝑦, 𝑙𝑎𝑏𝑒𝑙). To build training data, we randomly select spans of consecutive words as queries in the training data. Each query is paired with the sentence it comes from. The training set is denoted as {𝑋𝑖, 𝑞𝑖, 𝑙𝑖} 𝑁 𝑖=1 ,where 𝑙𝑖 𝑗 takes the value of 0 or 1 (1 indicates query 𝑞𝑖 is extracted from 𝑥𝑖 𝑗 and 0 otherwise). The loss function is defined as the binary cross-entropy loss, as follows:

After training, given a sentence x, we can use it to match the sentence with clue words and return the positions of clue words in this sentence by comparing the similarity score with the threshold 𝜃.

4.3 Attention Layer

Topic-aware Attention. The topic-based encoder above only considers the local context of each word, while the key concepts should be relevant to the global information of the document [Liu et al., 2010] (i.e., the title of the document, the major topics of document). So it is necessary to pay more attention to the words related to the documents topics. Note that, the title can be seen as an explicit topic [Chen et al., 2019]. The topic-aware attention is engaged to aggregate the relevant information from the global information for each word within the context. As shown in Figure 2, we define a summary vector q𝑡 = v𝑡 as the query (i.e., the output state of the Bi-LSTM). Following the conventional attention method [Luong et al., 2015], we create a sequence of topicaware token representations, u𝑖:

where 𝑊1,𝑊2, 𝑣1 are trainable parameters for computing the attention scores of each token.

Position-aware Attention. Although modern sequence models such as LSTM networks have gating mechanisms to control the relative influence of each individual word to the next token representation, these mechanisms do not explicitly model the position and semantic relationship between clue words and concept words in the sentence. The module aims at enforcing local attention, and adapting the attention weights so as to put more emphasis on clue-words-surrounded context by incorporating word position embedding. Inspired by the position encoding vectors used in [Zeng et al., 2014], we define a position sequence relative to the clue words:

where 𝑠1, 𝑠2 are the starting and ending indices of the clue words, respectively, and 𝑝𝑖 can be viewed as the relative distance of token 𝑥𝑖 to the clue words.

We first locate where the clue words appear by soft matching module and assign a position number to each token in the sentence according to formula (8). Then, we obtain position embedding vectors p 𝑠 = [ p 𝑠 1 , . . . , p 𝑠 𝑛 ] using a shared position embedding matrix P. As shown in Figure 3, in order to explicitly model the position and semantic relationship between clue words and concept words, we average the hidden vectors of all tokens inside the clue words as the attention query q. Then for each hidden state h𝑖, we calculate an attention weight 𝛼𝑖 , then we create a sequence of position-based token representations, m𝑖 :

where W3,W4,W5 and v2 are learnable parameters of the network.

4.4 Representation-enhanced Sequence Tagging

After encoding the context into the topic-aware representation u𝑖 and position-aware representation m𝑖, we aggregate them as h ′ by the following formula:

where 𝜆 ∈ (0, 1) is the corresponding hyperparameter.

Finally, we concatenate the original token representation h𝑖 with the enhanced one h ′ 𝑖 as the input ([h𝑖; h ′ 𝑖 ]) to the final CRF tagger. Our learning objective is the same as conventional sequence tagging, which is to correctly predict the tag for each token.

5 Experiments

In this section, we first describe the datasets and discuss how to collect clue words, then extensive experiments are conducted to verify the effectiveness of our proposed model.

5.1 Datasets Description

We use three datasets to evaluate our model. The details are described as follows:

CSEN [Pan and Wang, 2017]: this dataset contains 690 video captions in Massive Open Online Courses (MOOCs) for Computer Science courses.

KP-20K [Chen et al., 2018]: KP20K consists of 567,830 high-quality scientific publications from various computer science domains. We randomly select 20,000 articles from KP20K to form the KP-20K. We have collected concept phrases related to the computer field and automatically annotated the concept phrases in each article.

MTB [Huang et al., 2019]: this dataset consists of mathematics textbooks for elementary, middle, and high schools.

The statistics of datasets are listed in Table 2. #Titles denotes the number of video titles, article titles, and section titles in each dataset, and #Token is the number of words in each dataset. The column #labeled presents the ground-truth number after deduplication for each dataset. #clue words denotes the size of the final clue words set. Note that the CSEN does not provide an explicit title. We use the first sentence of each lesson as the title, which summarizes the content of this lesson. On all datasets, we use 70% as a training set, 10% as a validation set, and 20% as a testing set.

Preparing Clue Words. Our clue words are written by three experts with corresponding background knowledge. The development of the clue words set is considered completed when the annotator observes a fixed amount of training data. It takes the annotator in total less than 10 hours to develop all the clue words. We take the intersection of them as the final clue words set.

5.2 Baseline Approaches and EvaluationMetrics

To investigate the effectiveness of our model, we compare the performance of our algorithm with several baselines, including TextRank [Mihalcea and Tarau, 2004], Topical PageRank (TPR) [Liu et al., 2010] and Positionrank [Florescu and Caragea, 2017]. TextRank, TPR and Positionrank are three state-of-the-art graph-based keyphrase extraction approaches. We also employ three supervised baseline methods. The details are as follows:

• Joint-layer RNN [Zhang et al., 2016] is a neural tagging method, which combines keywords and context information to perform the keyphrase extraction task.
• CopyRNN [Chen et al., 2018] is an RNN-based generative model for predicting keyphrases in scientific text. It is the first application of the encoder-decoder model to the phrase prediction task.
• BERT-CRF [Yang et al., 2020] is a BERT-based model for concept extraction in clinical data. We set up all the parameters following the optimal setting.

For these unsupervised and generation-based methods we keep the top 𝑘 of the extracted phrases, 𝑘 is determined by the actual number of concepts in the document. Similar to previous methods, we use the precision (Pr), recall (Re), F1score (F1) to evaluate the performance.

5.3 Implementation Details

In our experiments, we run all experiments on one Tesla V100 GPU and 16 Intel CPUs. All words embeddings 𝑊𝑒 are initialized as 300-dimensional vectors by word2vec [Goldberg and Levy, 2014]. The hidden state dimensions of the Bi-LSTM encoder are set to 200. All weight matrices are randomly initialized by a uniform distribution U(-0.1, 0.1). In order to compute topic distributions of words and documents, we use LDA implementation in the topic modeling toolkit [Rehurek and Sojka, 2010] to train the topic model. The topic numbers are set to 50, 100, and 50 in the CSEN, KP-20K, and MTB respectively. The aggregation parameter 𝜆 in the attention layer is set to 0.5. The model is optimized by Adam with batch size 10 and dropout rate=0.1.

Soft Matching Module Hyperparameters. The threshold 𝜃 is set to 0.75 and the maximum window size is set to 3. We train the soft matching module until the loss does not decrease for about 20 epochs.

5.4 Results and Analysis

Overall Performance. Table 3 shows the performance of concept extraction on three datasets. From the table, we find that the proposed model outperforms all baselines on the datasets, which indicates the robustness and effectiveness of GACEN. The overall performance of unsupervised methods is worse than supervised methods. In unsupervised methods, they all mainly use statistical information within the corpus and have a strong reliance on term frequency, which hampers their performances. Specifically, TPR performs better than TextRank, perhaps because TPR also leverages the topic information. But they both perform worse than Positionrank which incorporates information from all positions of a word's occurrences into a biased PageRank. As for the supervised methods, we note that our GACEN model achieves the best performance on all datasets with significant margins. For instance, it achieves close to 10% of improvement on the F1score of the CSEN dataset compared with the second-best method BERT-CRF, which demonstrates the effectiveness of using topic, title, and clue words.

Ablation Study. We also perform an ablation study to better understanding the contributions of the main parts of our model. In Table 3, GACEN-topic, GACEN-position represent removing topic-aware attention module, position-aware attention module in GACEN, respectively. The GACEN*REs is a model, which replaces the soft matching module in GACEN with Regular Expression matching. As shown in Table 3, GACEN consistently achieves the best result on all three datasets in terms of F1-score. After we remove the topic-aware attention module, the recall decreases dramatically compared to precision on three datasets, indicating that the global topic information plays an important role in improving the recall. After we remove the position-aware attention module, the precision has dropped significantly, but the recall achieves the best result on CSEN and KP-20K. Conversely, when we replace soft matching with Regular Expression matching, GACEN*REs achieves the best result on all three datasets in terms of precision, whereas the recall decreases dramatically. They all show that clue words play a major role in improving the precision of the models and GACEN can better balance recall and precision.

Learning Efficiency. We study the learning efficiency of the model. We compare the performance of GACEN with BiLSTM-CRF for different percentages of the training data. As shown Figure 4(a), We can see that by using 20% of the training data, The GACEN model shows comparable performance as the base model using 70% training data. The drastic improvement in the model performance reflects the importance of title and clue words in concept extraction and justifies the slightly additional cost incurred in collecting clue words.

Generalization Ability. We further study the effect of the number of clue words on the results. Figure 4(b) shows the experimental results on different percentages of clue words. As the percentage of clue words increases, the performance of both GACEN and GACEN*REs continues to improve. As the number of clue words increases, synonymous clue words are added, the performance gap between the two models is decreasing. The result shows that the soft matching module can better generalize clue words to unseen sentences, especially when there are few clue words.

5.5 Case Study

Figure 5 shows two examples illustrating that the topic-aware attention and position-aware attention scores help GACEN recognize concepts. Figure 5(a) shows topic-aware attention scores, the concept "logistic regression", which is a title phrase that has high attention scores. For the positionaware attention module, in Figure 5(b), we find that the concept "hash function" gets high attention scores and the word "discussed" in this sentence is matched with "talk about" by soft matching. These results not only support our argument that the global information and clue words enhanced model such as GACEN can effectively learn, but also demonstrate that they can provide reasonable interpretation, something that lacks in other neural models such as Joint-layer RNN and CopyRNN.

6 Conclusion

In this paper, we proposed a novel model GACEN for the concept extraction task, which explicitly considered the structured information in the raw textual data. The experiment results on three real-world datasets exhibited the effectiveness of our model for concept extraction, especially when the labeled training data is limited.

Acknowledgements

This research was partially supported by grants from the National Key Research and Development Program of China (No. 2018YFC0832101), the National Natural Science Foundation of China (Grants No. 61922073 and 71802068), and the Fundamental Research Funds for the Central Universities (Grants No. WK2150110021).