Incorporating Extra Knowledge to Enhance Word Embedding

1 Introduction

Word embedding, a process to automatically transform the words in a vocabulary into dense vectors of real numbers in a continuous embedding space, has gained much attention and popularity as the derived word vectors often encode important syntactic and semantic information that is useful for Natural Language Processing (NLP).

Word embedding was shown to boost the performance of many downstream NLP tasks such as named entity recognition [Fang et al., 2016], relation extraction [Wang et al., 2014], sentiment analysis [Tang et al., 2014], text classification [Liu et al., 2018] and question answering [Cheng et al., 2015]

Since traditional word embeddings mainly capture the semantic relatedness between co-occurred words in a predefined context, they suffer from certain limitations. For example, although it is relatively easy to infer semantic relatedness between words based on word embeddings (e.g., to infer that kitten and cat are related is easy), it is often much harder to derive specific relationship types based on word embedding (e.g., to know that a kitten is a juvenile cat is difficult). Thus, the semantic relations explicitly encoded in a knowledge base can be useful to enrich and enhance word embeddings. Moreover, the same word may have multiple senses. Traditional word embedding is unable to discriminate among the different meanings of a word. Moreover, due to data sparsity, traditional word embedding also has difficulty producing reliable embeddings for rare words.

To address these issues, a large body of recent research has emerged to incorporate extra morphological, syntactic, semantic and domain knowledge to enhance word embedding. This technique can be used to (1) improve the quality of the word embedding learned from text; (2) add domain knowledge to facilitate domain-specific NLP; (3) overcome data sparsity when the text in the target domain is small or the target words are rare. In this survey, we summarize the recent advances in this research area. We include representative work on enhancing both static word embedding (e.g., Word2Vec) where a fixed embedding vector is learned for each word and dynamic/contextual word embedding (e.g., BERT), where the embedding vector for each word varies with its context.

2 Overview

Given the space limit, we define the scope of this survey quite narrowly to include only methods on incorporating knowledge into neural network based word embedding, as they are frequently the most widely used methods with the stateof-the-art performance. We exclude models that learn embeddings for larger text units (e.g., learning embeddings for phrases or documents). In addition, we only focus on English and exclude cross-lingual word embedding models. All the papers in the survey are published in the last ten years at the top NLP and AI venues. Table 1 is an overview of these papers. In the following, we summarize each paper based on the source and the type of knowledge that is added to word embedding, the downstream applications that employ these embeddings. We also explain and categorize the typical methods for knowledge injection into word embedding in Section 5.

3 Knowledge Sources

In this section, we describe various knowledge resources used in these papers to enrich word embeddings.

Wikipedia. This is the largest multilingual encyclopedia consisting of numerous articles. There are 6,017,385 articles

written in English. Wikipedia articles are all linked or crossreferenced. Both the text descriptions in Wikipedia and the graph created from interlinked Wikipedia pages are used as extra knowledge to enhance word embedding.

WordNet 1 and BabelNet 2 . Wordnet is a lexical database encoding semantic relations between words. Words are grouped into sets of cognitive synonyms called synsets, each expressing a distinct concept. Synsets are interlinked via semantic relations such as hyperonymy, hyponymy, meronymy, troponyms and antonymy. For instance, the synset [plant, flora, plant life] is linked to the synset [organism] via a hypernym relation. It is also linked to the synset [phytoplankton] via a hyponym relation. WordNet version 2.1 contains 155327 words and 117597 synsets. BabelNet is created by integrating WordNet with Wikipedia and other resources and maintaining the structure of WordNet.

Freebase and Probase. Freebase is a large collection of structured data. It comprises important semantic knowledge, especially the information about entities and relations (e.g., entity categories and belonging-to or is-a relations). For example, (France, capital, Paris) represents a relation in Freebase where "capital" denotes the relation between the entity "France" and the entity "Paris". As of January 2014, Freebase contained around over 40 million entities and 2 billion relations. Similarly, Probase [Wu et al., 2012] is a probabilistic taxonomy that contains concept-instance pairs connected in a hierarchical structure. For example, (vegetable, potato) is concept-instance pair and (food → vegetable) represents a concept hierarchy. It contains 2.7 million concepts.

Domain Dictionary. Domain dictionaries such as UMLS (Unified Medical Language System) [Bodenreider, 2004] and MAEC (Malware Attribute Enumeration and Characterization) contain useful domain knowledge that may be missing from the general word embedding. UMLS is the largest biomedical dictionary, containing more than 100 biomedical vocabularies, 13 million concepts and thousands of relations. For example, the concept of fever with the UMLS concept identifier "C0015967", is represented in nearly one hundred component vocabularies. Disease-treatment, diseasefinding, disease-prevention are some of the relationship types in UMLS. MAEC has specified a set of standard malware action attributes for cybersecurity [Kirillov et al., 2011].

Knowledge from Other NLP Tools. Morfessor is a tool that automatically splits a word into roots, prefixes, and suffixes (e.g. recycle=re(prefix)+cycle(root), kindness=kind(root)+ness(suffix)). Latent Dirichlet allocation (LDA) is a generative model that is used to learn a set of topics (latent themes) from unlabeled documents. A word sense disambiguation tool can also be used to annotate each word token with a specific sense tag.

Other Knowledge Sources. PPDB containing paraphrase pairs of English, general English dictionaries such as Oxford, Cambridge and Collins dictionaries, human annotations about the properties of words or relations between words as well as social graphs are also used in the surveyed papers.

1 https://wordnet.princeton.edu/

2 https://babelnet.org/

4 Knowledge Type

Since the knowledge injecting methods are shaped by the type of knowledge that is added, in this section, we categorize the basic knowledge elements that can be incorporated into word embedding.

Textual Knowledge. This type of knowledge can be found in Wikipedia as well as the definitions of words in a general or domain-specific dictionary. Textual descriptions often contain words that are highly relevant to the target word. One frequently used method to incorporate textual knowledge in word embedding is to first identify relevant and informative keywords from the text[Roy et al., 2019; Tissier et al., 2017] and then adapt the word embedding objective so that the embeddings of the target word will be close to the embeddings of the identified keywords.

Morphological Knowledge. Information about the basic elements of a word such as syllables, roots, or affix (prefix and suffix) is found in morphological knowledge. Leveraging morphological knowledge can facilitate the identification of semantically related words (e.g. words with the same root). As a result, it allows the system to generalize and capture the meanings of rare, out of vocabulary (OOV) and morphologically complex words accurately. To incorporate morphological knowledge, each morpheme is considered as the basic unit of representation and a vector representation is first learned for each unique morpheme. Then word embeddings are learned by composing the embeddings of all the morphemes within a word.

Categorical Knowledge. Semantic and syntactic properties of words are encoded in categorical knowledge. The semantic category of a word may include its concept, semantic type or semantic role while the syntactic category may include its part of speech tag and syntactic role. To integrate categorical knowledge in word embedding, we can either add an objective to minimize the distance between the word vectors that belong to the same category [Xu et al., 2014; Zhou et al., 2015] or increase the probability of predicting the target category of a given word [Roy et al., 2019; Cheng et al., 2015].

Relational Knowledge. This type of knowledge encodes the relationships between words. It is often represented as a triplet: (head H, relation R, tail T). Relational knowledge is injected into embedding by adding a constraint that requires the representation vector of T should be close to the representation vector of H plus the representation vector of R [Xu et al., 2014]. It is also possible that the vector representation of R is not explicitly learned during embedding. In this case, the relation constraints are enforced by incorporating additional training objectives related to R. For example, if R is a synonym relation, the system can add an objective to force the distance between the H vector and the T vector to be small. In contrast, if R is an antonym relation, the system can add a training objective to force the embedding of H and T to stay far apart [Bian et al., 2014; Faruqui et al., 2015; Glavaˇs and Vuli´c, 2018; Liu et al., 2015].

Knowledge Graph. Knowledge graph provides a rich source of high quality, human-curated structured knowledge to enhance word embedding. Knowledge graphs can be leveraged in two ways: (1) joint training with both the word embedding objective (the embeddings of neighboring words are similar) and the graph embedding objective (the embeddings of neighboring nodes are similar) [Cao et al., 2017; Fang et al., 2016; Wang et al., 2014]; or (2) using a pretrained knowledge graph embedding. Pre-trained knowledge graph embedding can be encoded into word embedding either by changing the objective function of word embedding [Zhang et al., 2019] or via word-to-entity attention mechanism [Peters et al., 2019].

5 Methodology

We can categorize the main methods employed in these papers along four dimensions: 1) static or dynamic embedding, (2) joint optimization or post-processing, 3) solo or coupled embedding and 4) single or multi-sense word embedding

5.1 Static or Dynamic Embedding

Word embedding techniques can be categorized into two classes: (1) Static word embedding and (2) Dynamic/contextual word embedding. Static word embedding only learns a single context-independent representation for each word. This implies that the meaning of a word is the same in the entire text corpus, regardless of its context. The most commonly used static embedding method is Word2Vec [Mikolov et al., 2013]. It employs a feed-forward shallow neural network trained with unlabeled text corpora. Two word2vec models have been proposed: CBOW and skipgram. In the CBOWmodel, context words are used to predict a target word and in the Skipgram model, context words are predicted based on the target word. Many of the static embedding methods in the survey aimed at improving Word2Vec with extra Knowledge.

On the other hand, dynamic/contextual word embedding generates a word representation that is a function of its context (e.g., the entire sentence). This allows dynamic embeddings to capture more context-dependent aspects of a word. The most commonly used dynamic embedding methods are ELMo (Embeddings from Language Models) [Peters et al., 2018] and BERT(Bidirectional Encoder Representations from Transformers) [Devlin et al., 2019]. For example, BERT is designed to pre-train deep bidirectional representations from a massive amount of unlabeled text by jointly conditioning on both the left and right context in all layers. It uses a transformer and an attention mechanism to learn contextual word embedding. Many of the dynamic embedding methods in the survey aimed at improving BERT with extra knowledge.

There are several differences between static and dynamic embedding: (1) static embedding is frequently trained using shallow neural networks and dynamic embedding is trained using deep neural networks; (2) the output of a static model is the embedding vectors, not the neural network models themselves. In contrast, the output of dynamic word embedding is the trained neural network models themselves as well as the word vectors dynamically generated by the models based on the context; (3) dynamic embeddings are more computationally expensive to train than static embeddings.

So far, the vast majority of the papers in our survey focuses on improving static word embedding (20 out of 22). Recently the field has shifted toward learning context-sensitive embedding. Some initial efforts have emerged to inject extra knowledge to dynamic word embedding. [Peters et al., 2019; Zhang et al., 2019].

5.2 Joint Optimization or Post Processing

In joint optimization, word embedding is trained with both text and extra knowledge simultaneously. These methods frequently modify the training objective to include both the native word embedding objective and a new objective related to the knowledge. The methods can be further categorized into: (1) adding prior or regularization to the original distributed representation learning objective [Xu et al., 2014; Bian et al., 2014]; (2) extending the original distributed representation learning objective to learn additional embedding [Cheng et al., 2015; Mancini et al., 2017; Roy et al., 2019]; (3) adding rank or hierarchical structure constraints, [Liu et al., 2015; Liu et al., 2018]; (4) combining training objective with knowledge graph embedding objective [Fang et al., 2016; Wang et al., 2014]; and (5) augmenting the input text with extra knowledge [Jiang et al., 2018; Tissier et al., 2017]. These models are tied to the distributional objective and any change of the underlying distributional model induces a change in the entire joint model.

Post-processing-based methods integrate extra knowledge into pre-trained word embedding. Popular post-processing methods such as retrofitting [Faruqui et al., 2015] fine tune the original word embeddings so that they satisfy additional constraints generated from the extra knowledge. Recently, [Peters et al., 2019] proposed a general method to inject multiple knowledge bases into pre-trained models.

The biggest benefit of post-processing over joint optimization is that it can be applied to any pre-trained word embedding models without expensive retraining. Post-processing methods often only locally update word vectors involved in the external constraints, whereas vectors of the other words remain intact. In contrast, joint optimization propagates the influence of the external knowledge to all the words via the joint objective.

5.3 Solo or Coupled Embedding

Solo or coupled embedding specifies whether a system learns word embedding only or word embedding is trained simultaneously with the training of one or more additional embeddings (e.g., sense embedding or semantic type embedding). The majority of the research in our survey only trains word embedding (13 out of 22 use solo embedding).

Coupled embedding can be achieved by augmenting the words in the input text with extra knowledge (e.g., augment a word with its sense or concept annotation). The model learns the vectors for them in the same embedding space [Mancini et al., 2017; Cheng et al., 2015; Roy et al., 2019]. Coupled embedding is also frequently used to combine knowledge graph with word embedding. For example, [Wang et al., 2014] presented one of the initial attempts to encode words and entities in a knowledge graph into a unified vector space. [Fang et al., 2016; Cao et al., 2017] focused on solving the ambiguity of entity mentions that arises from directly integrating entity and knowledge in the same space. Alternatively, [Peters et al., 2019; Zhang et al., 2019] focused on utilizing pre-trained entity embedding to improve the quality of word embedding.

5.4 Single or Multi-SenseWord Embedding

Single sense embedding ignores word polysemy and conflates all the meanings or senses of a word into a single representation. This deficiency hampers the effectiveness of word embedding and hurts the performance of downstream tasks.

To alleviate this problem, [Jauhar et al., 2015] uses external knowledge from a sense inventory to generate static sensespecific word embeddings. [Mancini et al., 2017] learns both word and sense embedding in a unified vector space. Recently, given a word, dynamic/contextual embedding produces different embedding vectors for the same word in different contexts. The main difference between these two approaches is that (1) dynamic embedding can learn word polysemy from text on its own while static multi-sense embedding requires external knowledge to identify the multiple senses of a word (e.g., with the help of a word sense disambiguation tool); (2) static multi-sense embedding assigns a fixed number of predefined sense embeddings to each word while in dynamic embedding, the number of vectors that can be associated with a word is, in principle infinite. Empirically, multi-sense embedding has significantly better performance over single sense embedding [Jauhar et al., 2015; Mancini et al., 2017].

5.5 Methodology Discussion

Dynamic word embedding techniques like BERT outperform static embedding techniques on many NLP tasks. However, due to the high computational cost associated with jointly optimizing the objective of dynamic word embedding and that related to the new knowledge, it would be very difficult for people who do not have access to powerful computational resources to apply the joint optimization approach. Since joint optimization is a more principled way to add new knowledge than post-processing, we may have missed opportunities to explore better but more expensive joint optimization-based approaches to enhance dynamic word embedding. In contrast, since the required resources needed to train a static embedding is much lower, there has been a wide range of methods proposed to enhance static word embeddings.

6 EvaluationMethods

In this section, we present the typical evaluation methods for assessing the quality of word embedding. We categorize them into intrinsic or extrinsic evaluation.

Intrinsic Evaluation. This is a generic evaluation of the quality and coherence of a vector space, independently from their performance in downstream applications. Different properties can be intrinsically tested, with semantic similarity being traditionally viewed as the most straightforward measure of the quality of word embedding. Cosine similarity is the most widely used metric for computing the similarity between word embeddings. Based on this metric, semantically similar words would have higher cosine similarity. The evaluation datasets consist of word pairs with similarity and relatedness scores assigned by humans. Spearman's test is used to compute the correlations between the similarity scores computed by the models and those assigned by humans. The majority of the papers we surveyed evaluated their models using the word similarity task. The second most common intrinsic evaluation task is analogical reasoning. Given two pairs of words with the same relation (e.g. man:woman :: king:queen), a model is considered to have answered the analogy question correctly if the nearest representation to Embd("man") - Embd("woman") + Embd("king") is Embd("queen"). Among the papers we surveyed, [Xu et al., 2014; Bian et al., 2014; Wang et al., 2014; Cao et al., 2017] reported model performance using this task.

Extrinsic Evaluation. This type of evaluation aims at assessing the quality of word embedding with downstream tasks. In addition to intrinsic evaluation, extrinsic evaluation is necessary to understand the effectiveness of different embedding techniques in real-world applications. Some of the most common NLP tasks used for extrinsic evaluation include synonym selection [Liu et al., 2015; Jauhar et al., 2015; Ling et al., 2017], name entity recognition [Liu et al., 2015], named entity disambiguation [Fang et al., 2016; Cao et al., 2017], entity typing [Peters et al., 2019; Zhang et al., 2019] relation extraction [Peters et al., 2019; Zhang et al., 2019; Wang et al., 2014], text classification [Liu et al., 2018; Tissier et al., 2017], sentiment analysis [Faruqui et al., 2015; Tang et al., 2014], question retrieval [Zhou et al., 2015] and question answering [Cheng et al., 2015].

7 Limitations and Future Directions

Based on this survey, we have identified some limitations in the current research. We also point out a few future research directions aiming at addressing these issues.

Currently, there is a limited understanding of the impact of adding new knowledge to word embedding. For example, word embeddings trained to capture distributional semantic are well known to exhibit seemingly linear behavior (e.g. the embeddings of analogy "woman is to queen as man is to king" and "work is to works as teach is to teaches"). The compositionality of word embedding is also an important topic. When extra knowledge is added to word embeddings using various knowledge injection methods, it is unclear whether the process might disturb the original embedding space and hurt some of the desired properties. Important questions such as "whether the linear behavior between semantically and syntactically related words are preserved in the new embedding space after knowledge injection?" and "how injecting new knowledge may impact the compositionality of word embedding?" are left unanswered. To address this, we may want to focus more on theory-guided knowledge integration methods so that we have a clear understanding of the impact of these methods on the learned embeddings. This issue can also be solved if we can interpret the meaning of word embeddings. Since the learned embeddings are in a high dimensional latent space that cannot be intuitively understood by humans, novel embedding visualization techniques may shed light on how different types of knowledge and injection methods may impact the embedding space. We may also use simulations to train word embeddings with synthesized text to enhance our understanding of the impact of various methods.

There are also insufficient efforts on incorporating diverse types of Knowledge in word embedding. Much of the work in this survey focuses on adding linguistic knowledge (e.g., morphological, syntactic and semantic knowledge). There has not been many efforts on incorporating other types of knowledge such as First Order Logic (FOL). Since logic is a powerful way to represent human knowledge, it is important that the knowledge expressed as FOL can be systematically incorporated into word embedding.

In addition to logic, there has not been much research on incorporating corpus-level constraints into word embedding. Corpus-level constraints can be used to specify certain requirements (e.g., to ensure that the learned word embeddings are unbiased). Previous research has shown that word embedding trained on human-generated text exhibits human biases [Bolukbasi et al., 2016]. It would be useful if we can add fairness as a corpus-level constraint to regularize the embedding training. Although it is possible to de-bias word embedding using other methods such as vector projection, encoding fairness as corpus-level constraints will give us more flexibility to plug in different fairness metrics.

So far, there is also little attention paid to embedding efficiency. Learning word embedding, especially contextual embedding (e.g., BERT) requires massive amount of data and computational power. Adding extra knowledge (e.g., a large knowledge base) to train a joint embedding only exacerbates this problem. Due to its high data and infrastructure demand, there are few options for people without access to the required resources can do to explore different knowledge injection options. So far, most work on integrating knowledge into word embedding has focused on improving model accuracy. More investigations are needed to balance the trade-off between accuracy and efficiency.