An Adaptive Hierarchical Compositional Model for Phrase Embedding

1 Introduction

Word and phrase embedding aims at representing words and phrases in a continuous vector space where semantically similar words and phrases are embedded near each other. The embeddings can be used to compute semantic relatedness and feed machine learning systems as the input. It have been proven successful in boosting the generalization performance of many NLP tasks such as machine translation [Mikolov et al., 2013a], syntactic parsing [Socher et al., 2013a], and sentiment analysis [Socher et al., 2013b].

Due to its usefulness, there have been many phrase embedding methods developed in the NLP community, and these methods can be broadly classified into three types: (1) compositional phrase embedding, (2) non-compositional phrase embedding, and (3) distinguish compositional phases from non-compositional phrases. The compositional phrase embedding is based on Frege's Principle of Compositionality, stating that the meaning of a complex whole is a function only of the meanings of its syntactic part. For example, the meaning of phrase North America is associated with the meanings of both North and America. Therefore, a phrase embedding is computed by aggregating its individual words' embedding using various composition functions [Mitchell and Lapata, 2010; Socher et al., 2012; Yu and Dredze, 2015]. Such compositional phrase embedding methods are scalable to the number of phrases and could learn fine-grained semantic compositionality. However, they are limited by their inability to represent idiomatic phrases, where the meaning of an idiomatic phrase is not associated with its component words, e.g., new york has a different meaning with either new or york.

Another type of embedding methods treats a phrase as a single unit and assign a unique embedding for it [Mikolov et al., 2013b; Zhang et al., 2014]. Such phrase embeddings are commonly referred to as non-compositional embedding. Solely relying on non-compositional embedding will suffer from data-sparsity problem since they do not distinguish nonidiomatic phrases from idiomatic phrases. Thus the accuracy of word embeddings for non-idiomatic phrases or their component words are both decreased. In addition, compare with the compositional phrase embeddings, the non-compositional phrase embeddings require extra vectors for phrases which increase the model complexity significantly.

Recently, new techniques are developed to automatically distinguish compositional phases from non-compositional phrases and learn their embedding individually. For instance, Yazdani et al. [2015] used supervised models to detect noncompositional phrase. Hashimoto and Tsuruoka [2016] proposed a model that learn compositionality levels by logistic regression. However, they share a significant drawback that they only stick to either full-compositional embedding or non-compositional embedding, while ignore the fact that phrases could be hybrid-compositional, which means there exist both non-compositional and compositional parts in a phrase. For example, phrase new york city should be represented as new york ⊕ city, where the words connected with ' ' is a non-compositional unit and '⊕' denotes the composition. Such hybrid-compositionality has not been fully explored by existing methods, which prevents them from having a deeper insight into the semantic structure of long phrases.

In order to address this issue, we propose a novel hierarchical compositional model for phrase embedding. The goal of our approach is to provide a joint learning paradigm thereby adaptively weighting different compositions by making the first effort to consider hybrid-compositionality of phrase, while at the same time optimizing its embedding accordingly. In addition to learning phrase embeddings, our approach could also mine valuable phrases and infer their internal structures. In our model, the compositions of a phrase are represented by an implicit hierarchical structure, and phrase embedding are defined as a mixture of non-compositional and the hierarchically compositional embedding. The proportion between the two components are adaptively adjust by composability. In this way, the weight could be decoupled with a specified composition, thus lower the model complexity. In order to avoid unnecessary cost caused by different constituent partition, our model could joint learn model parameters and probabilistically reason about the optimal constituent partition. Our method is purely data-driven, where all the parameters are jointly learned in conjunction with the target task for learning phrase embedding. We evaluated our model on both similarity tasks and analogy tasks, the results demonstrate that our model outperforms state-of-the-art models.

2 Hierarchical Compositional Model

Given a phrase new york city, it could have four distinct compositions: new york city, new york ⊕ city, new ⊕ york city, and new ⊕ york ⊕ city. We hope to propose a "good" model to get the correct composition of new york ⊕ city and its resulting embedding in a purely unsupervised manner. To incorporate all the compositions, a straightforward solution is to use weighted sum model and represent phrase embedding as the weighted sum of all its compositions (see Eq. (1)), and learn the right composition by adjusting the weight g(·) of each composition.

where H is the set of all compositions of a phrase p, and vec(h) is the embedding under composition h ∈ H.

Since there are 2n-1 compositions for an n-gram phrase (i.e. a phrase with n words), the weighted sum model needs to keep O(|Pr| · 2n ) free parameters for a corpus with entire phrase set Pr. Such a high model complexity generally results in overfitting on training data.

To tackle this problem, we propose a novel hierarchical compositional (HC) model that can provide lower model complexity to learn better phrase embeddings. In our model, a phrase's embedding vec(p) is defined as a mixture of its non-compositional embedding e(p) and a hierarchically compositional embedding. The latter one is defined on the phrase's optimal constituent partition C ∗ p which forms an implicit hierarchical structure, e.g., new york ⊕ city can be described as a composition of new york and city, while new york is a non-composition of new and york. The proportion of the two embeddings is adaptively adjust by its composability (the level of compositionality) β(p). Formally, phrase embedding vec(p) is computed as follow:

where β(p) (0 ≤ β(p) ≤ 1) is a free parameter for a given phrase p to denote its composability, and β(p) = 0 if p is a word. A larger value of β(p) indicates p is of higher composability and vice versa. Any constituent c ∈ C ∗ p could be either a single word or a multi-words term (i.e. phrase). The constituent partition refers to the non-overlapping sequence partition of a phrase, e.g., {new york, city} is a constituent partition of phrase new york city, in which new york or city is a constituent. We will discuss how to infer the optimal constituent partition C ∗ p in Section 3.2 and how to learn parameters β(p) and e(p) in Section 3.3.

There are at least two reasons why our HC model is superior to the weighted sum model. Firstly, it could be decoupled with a specific composition since the weights of compositions are implicitly represented by aggregating the composability of phrase. Therefore, HC model could provide a high enough representation ability with only O(|Pr|) parameters, which greatly reduces model complexity. Secondly, the loss can be back-propagated from a phrase to its component words. It could improve both phrase and component word embeddings.

In this paper, we adopt widely used element-wise addition [Tian et al., 2017] as our composition function.

3 Phrase Embedding Based on Hierarchical Compositional Model

Our goal is to learn "good" model parameters (β(p) and e(p)) with the help of constituent partition to generate better phrase embeddings. In this section, we propose a Hierarchical Compositional based Phrase Embedding (HCPE) approach to joint learning model parameters and optimal constituent partition.

3.1 HCPE Framework

Obviously, constituent partition and phrase embedding could benefit from each other. A "good" constituent partition could construct a correct hierarchical structure to facilitate better phrase embedding. On the other hand, "good" word/subphrase embeddings could also facilitate generating better partition: a better partition is supposed to further improve the prediction of context (or center word) to get better embedding. Therefore, we propose an HCPE approach to joint learning phrase embeddings and constituent partitions.

Let S = p1, p2, ..., p|S| be a corpus, in which each pi could be either a single word or a phrase (|S| denotes the number of pi in S), V be the set of vocabulary (i.e., distinct words and phrases) in S, and vi ∈ V is a distinct word or phrase. Pr be the set of distinct phrases in S, and pri ∈ Pr is a distinct phrase. HCPE aims to maximize the joint probability Pθ (S, C):

where C = {Cpr1 ,Cpr2 , ...,Cpr|Pr| } is a set of latent variables in which each Cpri denotes a latent variable to denote the constituent partition of phrase pri, and θ = {e(v1), . . . , e(v|V |), β(pr1), . . . , β(pr|Pr|)} is the model parameter.

In this paper, we adopt commonly used SkipGram [Mikolov et al., 2013b] as our embedding architecture. Based on Skip-Gram 1 , we have:

where Cont(pi) is the set of context words/phrases of pi, Cpi ∈ {C 1 pi , C 2 pi , ..., C 2|pi|-1 pi }, in which C jpi denotes a specific constituent partition of pi, e.g., {new, york city}, {new york, city}, or {new, york, city} for phrase new york city. Note that, if pi is a single word, we have P (Cpi ) = 1 and P (u|pi, Cpi ) = P (u|pi).

With the form of Eqs. (3) and (4), maximizing the joint probability in Eq. (3) is actually a Hard-ExpectationMaximization (Hard-EM) problem [MacKay, 2003], which has been proved to get converged and achieve a local optimum by a coordinate ascent strategy [MacKay, 2003]. To infer latent variables C and learn model parameters θ, the following two steps are repeatedly performed:

• E-step (Expectation step): inferring the optimal constituent partition C ∗ pi of a phrase pi with a fixed θ.

• M-step (Maximization step): learning model parameters using C ∗ pi inferred by E-step.

3.2 E-step: Inferring Optimal Constituent Partition

Firstly, we propose a generative model to compute the probability of generating a constituent partition (i.e., P (Cpi )). Secondly, considering that the enumerating-based method is intractable to to infer optimal constituent partition, we propose a dynamic programming method to make the computation feasible.

Probability of Constituent Partition

Given an n-gram phrase p = w1 . . .wn, let Cp = {c1, . . . , cm} be any constituent partition of p, which can be separated by a set of split boundaries B = {b1, . . . , bm, bm+1}, where b1 = 1 and bm+1 = n + 1.

1 Other architectures like CBOW [Mikolov et al., 2013b], GloVe [Pennington et al., 2014], etc can be readily applied.

where P(ρ, ci, bi+1|bi) is the conditional probability of observing ci is the i-th constituent, and ρ is a phrase indicator.

Eq. (7) is derived from the following generative model: Firstly, we generate split boundary bi+1 with a probability of P (L) assuming the order-1 Markov property, where L is a random variable and P (L) is the length prior over the number of words in ci, P(L) can be computed by:

where α ≤ 1 is a length penalty factor. A smaller α rewards a longer constituent length which would make the hierarchical structure deeper and thus provide a more plentiful compositional diversity.

Secondly, we generate ci according to the probability

where f(ci) denotes the frequency of ci in the corpus. This probability is based on the frequent term have high probability to be a whole unit.

Thirdly, we generate the phrase indicator ρ of ci. If ci is a phrase in the vocabulary V (including all words and phrases) , it should be a strong evidence that the whole ci is a unit. Therefore, we exclude the word sequences which is not in V by:

Based on the above, P(ρ, ci, bi+1|bi) in Eq. (7) can be probabilistically factorized as:

Efficiently Inferring Optimal Constituent Partition

Taken softmax as the activation function, the joint probability can be represented as:

Finally, based on Eq. (12), E-step is to solve the following optimization problem:

Intuitively, a straight-forward method is to enumerate every possible constituent partition to find the best solution, which is intractable (O(2n ) complexity) in practice. Therefore, the challenge here is how to avoid enumerating constituent partitions to make computation efficient.

We propose a dynamic programming-based strategy to find the optimal constituent partition to maximize Eq. (13) while keep the computation tractable. It is based on the observation that if C ∗ [1,i] is the optimal constituent partition of phrase w1 . . . wi-1, then there must have a constituent c[k,i] ∈ C ∗ [1,i] (k < i), makes C[1,i] - {c[k,i]} be the optimal constituent partition of phrase w1 . . .wk-1. Therefore, we set up a vector M ∈ [0, 1]n+1 , in which each component Mi (i ∈ {1, . . . , n + 1}) stores the optimal probability of phrase w1 . . . wi-1. Initially, we have M1 = 1. The recursion function is given as follows:

Based on Eq. (14), we choose the appropriate k to make Mi maximal. The value stored in Mn+1 equals the maximal value of Eq. (13). Then the optimal constituent set can be easily fetched by back-tracking the vector fromMn+1. Since we need to fill an (n+1)-length vector, and the cost of computing each cell is O(n), the total time complexity is O(n2 ).

3.3 M-step: LearningModel Parameters

Now we show how to learn model parameters β(p) and e(p) based on the optimal constituent partition. We adopt stochastic gradient descent (SGD) to optimize the objective function using random initialization. Considering that the original softmax function is impractical because its huge compuational cost [Mikolov et al., 2013b], in practice, some approximated objective fucntion are used to speed up the processing. We use J to denote a given approximated objective function, e.g., Hierarchical Softmax, Negative Sampling [Mikolov et al., 2013b], etc. Let δ(l) p ∈ Rd×1 (d is the dimension of embedded vectors) be the partial derivative of J with respect to the variable vec(p) on l-th layer of the hierarchical structure. Notice that, if l = 1, we have δ(1) p = ∂J ∂vec(p) . The partial derivative of J w.r.t. any lower-layer constituent ci can be computed via its direct upper-layer phrase p's partial derivative according to the following equation:

In this way, the error term can be back-propagated via the hierarchical structure.

At each layer, the gradient of non-compositional embedding e(p) of phrase p can be computed by:

From Eqs. (15) and (16), we can see that β is used for adjusting the error term's proportion between the noncompositional embedding and compositional embeddings. If β is close to 0, the non-compositional embedding are mainly updated, and vice versa.

At each layer, the partial derivative of β(p) can be computed by the follow equation:

Eq. (17) shows that, β(p) is used for adjusting between non-compositional embedding e(p) and hierarchical compositional embedding vec(p). To be specific, if the error term δ(p) is more closer to compositional embedding than the non-compositional embedding, i.e., δ(p)·∑ci∈Cp vec(ci) > δ(p) · e(p), β(p) needs to be bigger accordingly, and vice versa. The intuition behind this is that: (i) if a phrase is noncompositional, its contexts is often different from the contexts of its constituents; and (ii) if a phrase is compositional, its contexts is similar with the contexts of its constituents, i.e., its constituents and the contexts co-occurs frequently.

From both human judgments and empirical results, β follows a U-sharp distribution indicating the composability of most of phrases are very close to either 0 or 1. Moreover, it should be cut-off when β reaches either 0 or 1 to avoid overflowing. Therefore, we need a large learning rate when β is near 0.5 , and gradually decrease the learning rate towards 0 or 1. For β(p), we use the following learning ratio ηp:

In our HC model, the error term are back-propagated recursively from the root phrase to each non-composable unit until every composability ratio β and non-compositional embedding e are updated, which helps to improve both phrase embeddings and constituents' embeddings.

3.4 Complexity Analysis

Model complexity: Besides the conventional embedding model (e.g., Skip-Gram, CBOW), our model requires Θ(|Pr|) parameters for a set of phrases Pr. Taken SkipGram as our embedding architecture, then the overall model complexity would be Θ(|V | · T + |Pr|), where |V | is the vocabulary size and T is the dimension of vector. Generally, |V | · T ≫ |Pr|, thus our model has almost the same model complexity with conventional embedding models.

Time complexity: For an n-gram phrase p, the cost of E-step is O(n2 ), and the cost of M-step is O(n). Since E-step and M-step are alternately performed to scan the whole corpus S, the total time complexity is O(n2 |S|). Considering that a phrase commonly keeps a relative short and fixed length (among 2-6), the n2 term can be regarded as a constant value ω, and thus the complexity is O(ω|S|).

4 Experimental Evaluation

In this section, we evaluated our HCPE model on similarity tasks and analogical reasoning tasks, and demonstrate its superiority over state-of-the-art models. Further, a qualitative analysis along with a case study show the capability of HCPE to identify the composability of phrase embeddings.

4.1 Training Corpora

We used the following three training corpora:

• Text8 2 contains the first billion characters extracted from English Wikipedia.
• GoogleNews 3 contains all Google news articles in 2012.
• Wiki 4 contains all English Wikipedia articles in 2015.

The detailed statistics are summarized in Table 1.

4.2 ComparedModels

To demonstrate the effectiveness of our model, we compared our model with the following state-of-the-art models:

• Word2vec [Mikolov et al., 2013b] is a two-layer neural network model to produce words or phrases embeddings. It take all phrases as non-compositional. Here we use its Skip-Gram with Negative Sampling model.
• SVO [Hashimoto and Tsuruoka, 2016] learns a scoring function to detect compositional and non-compositional in phrase embedding. We adopt this method as it represent the state-of-the-art method for jointly learning compositional and non-compositional phrase embedding.

Kartsaklis et al. [2014] reported that non-compositional embeddings are better than full-compositional embeddings in phrase similarity task. Therefore, we do not compare with a full-compositional method.

4.3 General Experimental Setting

Our HCPE approach adopts Skip-Gram architecture, as it represents the state-of-the-art embedding model.

For optimization, both state-of-the-art methods and our method used Negative Sampling (NEG) technique [Mikolov et al., 2013b] to speed up training, and all methods use the same vector dimension and default setting. To be specific, we set the number of negative examples to be 25, and iterations (number of epochs) to be 5. The initial learning rate of SkipGram model were set to 0.05. We set the dimension of vector d = 200, unless noted otherwise. We set context window length to be 10 and sub-sampling rate 1e - 5.

In order to properly initialize parameter β, we had tried several different strategies, including: (1) Randomly initializing β by a Uniform distribution U[0, 1]; (2) Randomly initializing β by a Gaussian distribution N(0.5, 1); (3) Initializing β by a fixed value, e.g., let β = 0.5 for all phrases; (4) Initializing β by their phrase's normalized PMI (NPMI), i.e., β = 1 - npmi. According to experimental tests, random Gaussian was the best initialization strategy (was 3.78% better than random Uniform, 5.75% better than the best fixed β (0.5), and was 7.85% better than NPMI, on phrase analogical reasoning tasks). Thus, we randomly initialized β by a Gaussian distribution N(0.5, 1).

2 https://cs.fit.edu/ mmahoney/compression/textdata.html

3 http://www.statmt.org/wmt14/training-monolingual-newscrawl/news.2012.en.shuffled.gz

4 https://dumps.wikimedia.org/

To validate the effectiveness of our learning rate of β(p), we compared with two conventional approaches: (1) fixed learning rate, i.e., ηp ∈ {0.025, 0.05, 0.1}; and (2) add a regularization term ||β(p)2 + (1 - β(p))2 ||on β(p). We tested them on three corpora, on average, our approach was 1.20% better than the best fixed learning rate (0.05), and was 8.48% better than regularization approach on phrase analogy tasks.

4.4 Similarity Tasks

In the similarity tasks, each model needs to compute semantic similarity (Cosine similarity) for any given word/phrase pair, and the correlation between the result and gold standard are reported as the model performance. We adopt two correlation measurements: Spearman correlation ρ and Pearson correlation γ. We tested our model on both phrase and word similarity tasks with the same trained vectors to demonstrate our model could effectively reduce the sparsity problem and could help not only phrase similarity but also word similarity.

Phrase similarity

For phrase similarity tasks, we adopt three test datasets: (1) TR9856 [Levy et al., 2015] contains human-labeled phraserelatedness values for 9856 phrase pairs. (2) PhraseSim [Mitchell and Lapata, 2010] contains human-rated similarity at the scale of 0 to 9 for 324 phrases pairs. (3) WP300 [Li et al., 2013] contains 200 phrase pairs and 100 word pairs with human-assigned similarity judgment.

Table 2 shows results of HCPE, Word2vec, and SVO trained on three corpora and tested using the above three test datasets. HCPE achieved a better performance in most cases. An interesting observation is that, on Text8 corpus, HCPE performed sightly better than others on TR9856 while little worse than SVO on PhraseSim dataset. The reason is that, Text8 is a very small corpus (only 17 million words), it is hard for parameter β in HCPE to get converged in such a small corpus, thus it inevitably involves error at the unconverged period. But once it got a large enough corpus (e.g., GoogleNews or Wiki ), HCPE performed predominately better than Word2vec and SVO.

We also investigated the influence of data size and vector dimension on phrase embeddings, as shown in Figure 1. For testing the influence of data size, we varied data size from 0.2 billlion to 2 billion by increasing the iteration on GoogleNews dataset, and reported the Spearman correlation on Phrasesim dataset. Figure 1(a) shows that, all models could gain better results with a larger data size. Compared with Word2vec, HCPE can quickly achieve a better result with a small amount of data size (less than 0.6 billion words), while Word2vec could achieve only 0.30 correlation. Compared with SVO, our method could gain a better result in any data size. Figure 1(b) shows the result when increasing the vector dimension. We can see that different dimension settings (from 50 to 250) do not cause significant changes for the three models, and our model is still the best among the three.

Word similarity

We adopt three test datasets: (1) SimLex-999 5 is a gold standard resource for the evaluation of models that learn the meaning of words. (2) Sim-353 6 contains 353 word pairs along with human-assigned similarity judgments. (3) WP130 [Li et al., 2013] contains 130 word/phrases pairs along with human-assigned similarity judgments.

Table 3 shows the results of word similarity tasks. As expected, our HCPE model achieved a better performance, since it could adaptively adjust between different compositions, and update constituent words' embeddings by backpropagating error from its phrase embedding. Thus, compared with non-compositional model Word2vec, and binary compositional model SVO, our model could greatly reduce the sparsity caused by phrases so that it could improve both phrase embeddings and word embeddings.

4.5 Analogical Reasoning Task

We adopt Google semantic analogy dataset introduced by Mikolov et al. [2013b] which contains 3218 phrase analogy questions among five relations. Here we use the precision@top-1 as measurement, which means for each question the model is only allowed to return its top-1 similar phrase as answer. Table 4 reports the results of the three phrase embedding models.

5 http://www.cl.cam.ac.uk/ fh295/simlex.html

6 http://www.cs.technion.ac.il/ gabr/resources/data/wordsim353/

Table 4 shows the performance of Word2vec is the worst among the three, especially on small corpus, indicating that non-compositional phrase embedding models are easier to suffer from sparsity problem. HCPE could achieve a better performance than SVO nearly in all settings. This demonstrates comprehensively incorporating hybrid-composition, full-composition and non-composition using our hierarchical compositional structure is superior to the binary compositional model and account for its better performance.

We also compared the performance of HCPE model with Word2vec and SVO in analogy tasks with various corpus sizes and vector dimensions. Figure 2(a) shows the precision when increasing the corpus size from 0.2 billion to 2 billion. We can see the that, for all the three models, the larger corpus is used, the better the performance will be. Compared with Word2vec and SVO, our model could achieve the best precision in all corpus size settings. To be specific, our model could quickly achieve much higher precision than Word2vec and SVO on both small and large corpora. Figure 2(b) shows the result on various vector dimensions on Text8 corpus. Intuitively, dimensionality directly relates to the number of parameters to learn, and that's why we demonstrate its impact of these algorithms for varying dimensionality. Our model performs significantly better than Word2vec in all dimensions but performs slightly worse than SVO on 100 vector dimension, but on the whole, our model performs better on other settings and achieves best precision (32.18%).

4.6 Phrase Composability Analysis

To investigate the quality of composability β, we randomly sampled 104 phrases and asked three knowledgeable people to rate on a scale of 0 - 10 to judge the degree of a phrase's composability, based on criteria that whether the phrase has a figurative meaning that is different from its literal meaning, or the phrase has a meaning that is not a simple composition of the meanings of its constituent words [Mikolov et al., 2013b]. Table 6 shows the proportions of different compositionalities.

We averaged the three human ratings as gold standard and computed the Spearman ρ and Pearson γ correlation between the gold standard and the result of our model. For comparison, we also lists the correlation of three fixed β settings β ∈ {0, 0.5, 1}. Table 7 shows the result of phrase composability analysis.

HCPE performs significantly better than the three fixed β settings, which demonstrate the effectiveness of our model.

4.7 Case Study

We also provide a case study based on the result of HCPE model on GoogleNews dataset. Table 5 lists 9 sampled phrases and classifies them into three groups (non-compositional, hybrid-compositional, and full-compositional) according to their compositionality. The weight of each composition is computed by multiplicating composability of generating this composition. For each phrase, we show its highest-weighted composition inferred by HCPE along with a human rated weight about it.

From the table we can see that, the highest-weighted composition of each phrase is in accordance with human intuition and the weight is very close to human judgment. The non-compositional phrase (e.g., salt lake city) or non-compositional constituent (e.g., new york) are correctly picked up by HCPE model. Furthermore, our model could flexibly adjust between different compositionality and adaptively learn the right composition. This demonstrate HCPE model is of high effectiveness and works well in practice.

5 RelatedWork

The importance of phrase embedding has led to many researches over the past few years. Existing works can be broadly classified into three categories according to their treatment of compositionality.

The first category represents a phrase's embedding as the aggregation of its individual word's embedding on top of various composition functions, such as vector addition/averaging [Tian et al., 2017], component-wise multiplication, and tensor products [Mitchell and Lapata, 2010]. There are complicated functions such as weighted linear compositional model [Yu and Dredze, 2015], ordered tensor products [Clark and Pulman, 2007; Coecke et al., 2010; Socher et al., 2013a], and recursive matrix-vector neural network [Baroni and Zamparelli, 2010; Socher et al., 2012].

The second category treats phrases as single units, and assigns a unique embedding for each phrase. Mikolov et al. [2013b] models phrase as a single unit. Levy and Goldberg [2014] and Yin et al. [2016] train non-compositional phrase embedding using dependency parsing. Yazdani et al. [2015] adopt a supervised model to detect noncompositional phrases. Katz and Giesbrecht [2006] adopt LSA to automatically identify non-compositional phrases. Albeit non-compositional embeddings are preferable when dealing with idiomatic phrases, it may suffer from datasparsity problem.

The third category tries to distinguish compositional and non-compositional phrase. Farahmand et al. [2015] provide compositionality dataset for compound nouns. McCarthy et al. [2007] and Venkatapathy and Joshi [2005] use distributional features of corpus to detect compositionality. Kiela and Clark [2013] detect the compositionality using word embeddings. Hashimoto and Tsuruoka [2016] learns compositionality by a scoring function. Wang et al. [2017] propose a supervised model to learn sentence-level embedding. In their model, a sentence is always compositional, while a constituent's embedding could be the max-pooling of its compositional and non-compositional embeddings.

Existing models only stick to either full-composition or non-composition, while ignore the hybrid-composition of phrases. By contrast, our model could adaptively learn phrase compositionality and updating its embedding accordingly.

6 Conclusion

In this paper, we proposed a HCPE method to learn phrase embedding. Compared with the best state-of-the-art method SVO, on average, our method was 5.5% better in analogical reasoning tasks and was 1.8% better in phrase similarity task.

Acknowledgments

The work was partially supported by the National Natural Science Foundation of China (Nos. U1736104, 61572122, 61532021), CSC scholarship No. 201606080078. Wei Wang was partially supported by D2DCRC DC25002 and DC25003, and ARC DP 170103710 and 180103411. Wei Cui was supported by Natural Science Foundation of Shandong Province No. ZR201709250069. Xiaochun Yang is the corresponding author of this work.