Adaptive Semantic Compositionality for Sentence Modelling

1 Introduction

Currently, there has been a surge of interest in learning distributed representations for sentences with neural models. Most of related work focuses on the question that how to obtain a desirable sentence vector given the vector representations of words in the sentence. Existing approaches take a compositional function with different forms to compose word vectors recursively until obtaining a sentence representation. Typically, these compositional functions involve recurrent neural networks with long short-term memory [Hochreiter and Schmidhuber, 1997], convolutional neural networks [Kalchbrenner et al., 2014], and recursive neural networks [Socher et al., 2013].

However, a potential weakness of these models is that not all the phrases are compositional. For example, most of the idioms are non-compositional, whose meanings are hard to be predicted from the meaning of their constituents. Besides, it may harm the performance of the model to assume that all the phrases are compositional. The words in idioms are uninformative or even can act as noisy samples. For example, for the sentence "She will go bananas about your behaviour", the phrase "go bananas" means

∗Corresponding author.

"get very angry". We can hardly compose this meaning according to the words "go" and "bananas".

Based on this observation, we propose an adaptive semantic compositional model for sentence modeling as shown in 1, which can learn how to obtain the representation of a phrase: compositional or non-compositional way. In compositional way, the representation of a phrase is composed of its containing words or sub-phrases. In non-compositional way, the representation of a phrase is learned as a whole unit, which can be pre-trained from large scale unlabeled data. To do this, we adopt tree-structured Long Short-term Memory Network to recursively model a sentence over its syntactic tree, and introduce a parameterized controller, which can adaptively determine proper way (compositional/non-compositional way) to model a non-leaf node.

Additionally, considering that the compositionality of some phrases are context-dependent (e.g. "around the corner"), we propose to use a hybrid mechanism to learn context-sensitive phrase representations, which can disambiguates the meanings of the phrases.

We evaluate our models on five datasets of sentiment classification. The experimental results show the effectiveness of our proposed models. Furthermore, we present an elaborate qualitative analysis of our models, giving an intuitive understanding how our model worked.

The contributions of this paper can be summarized as follows.

1. By introducing a compositional switch and a hybrid mechanism, we grow the capacity of syntax-based neural sentence models, allowing it to model the noncompositional phrases when learning sentence represen-

tations.

1. We propose two kinds of models to encode noncompositional phrases, one of which can address the problem of idiomatic variations.
2. Beyond quantitative measurement, we carefully perform qualitative analysis, and demonstrate why and how the idea works.

2 Neural Models for Sentence Modelling

The primary role of neural sentence modelling is to represent the variable-length sentence as a fixed-length vector. These models generally consist of a projection layer that maps words, sub-word units or n-grams to vector representations, and then compose them with different forms of compositional functions. Most of these models for distributed representations of sentences can be classified into three categories.

Sequencemodels Sequencemodels construct the representation of sentences based on the recurrent neural network (RNN) [Mikolov et al., 2010] or the gated versions of RNN [Sutskever et al., 2014; Liu et al., 2015; 2016a]. Sequence models are sensitive to word order, but they have a bias towards the latest input words.

Convolutional models Convolutional neural network (CNN) is also used to model sentences [Collobert et al., 2011; Kalchbrenner et al., 2014; Hu et al., 2014]. It takes as input the embeddings of words in the sentence aligned sequentially, and summarizes the meaning of a sentence through layers of convolution and pooling, until reaching a fixed length vectorial representation in the final layer.

Syntax-based models Syntax-based models compose the sentence representations following a given tree structure (i.e. Constituency tree). More specifically, the model computes parent vectors in a bottom up fashion using different types of compositional functions [Socher et al., 2013; Tai et al., 2015; Liu et al., 2016b; 2017]. Lastly, The vector computed at the top node gives a representation for the sentence.

3 Syntax-based Long Short-termMemory Network

Syntax-based sentence models can be equipped with various kinds of compositional functions, such as neural tensor layer [Socher et al., 2013] and tree-structured LSTM (T-LSTM) [Tai et al., 2015]. Here, we introduce the latter model due to its superior performance in representing sentence meaning.

T-LSTM is a generalization of LSTMs to tree-structured network topologies. More formally, given a binary constituency tree T induced by a sentence, there are two child nodes for each non-leaf node. We refer to hj and cj as the hidden state and memory cell of node j. The transition equations of each node j are as follows:

where xj denotes the input vector and is non-zero if and only if it is a leaf node. The superscript l and r represent the left child and right child respectively. σ represents the logistic sigmoid function and ⊙ denotes element-wise multiplication. TA,b is an affine transformation which depends on parameters of the network A and b.

4 Adaptive Semantic Compositional Neural Network

Existing neural sentence models are failure to understand non-compositional phrases in a sentence, which significantly affects one's understanding of the sentence.

To address this problem, we propose an adaptive semantic compositional neural network, which integrates a compositional switch into syntax-based neural sentence models and can adaptively learn the compositional and noncompositional representation of phrases over a syntax tree recursively thereby obtaining a sentence representation.

Specifically, the model consists of three parts: compositional component, non-compositional representation, and semantic compositional switch. Given a node xj from a binary constituency tree, we refer its two children as xlj and xrj . And the hidden states of these three nodes are referred to as hj, hlj , hrj respectively.

4.1 Compositional Encoder

The function of this module is to compose any two words or phrases with a parametric composition function. More formally, given two hidden states of the children, hlj and hrj , the hidden state of their parent can be computed as:

where f(, ) represents the composition function, and particularly, here we use tree-structured LSTM (T-LSTM) [Tai et al., 2015] unit as described in Eq. (1-3).

4.2 Non-compositional Encoder

Different with previous models, apart from compositional function, each phrase is assigned an extra vector hn to represent the idiomatic meaning. To obtain hn , we propose two models as follows.

Retrieval Model (RM) In this model, the noncompositional phrases are directly retrieved from an external memory, which is constructed by a learnable matrix M.

where k denotes the index of the corresponding phrase.

Retrieval model provides an efficient way to generate idiomatic meanings for non-compositional phrases, which can benefit from the knowledge of external data. Apparently, one limitation of this model is that RM requires a large matrix and suffers from the problem of OOVs. To address this problem, we propose another representing mechanism.

Compositional Model (CM) Nunberg et al. [1994] proposes a view that idiomatic phrases can still be composed in a new space. Inspired by it, we compose non-compositional phrases with a new compositional function.

where W ∈ Rd×2d is a learnable compositional matrix, b is the bias vector.

Model Comparison For the compositional encoder and non-compositional encoder, as shown in Figure 2, they both share an embedding layer, while have different parameters to learn.

For two different non-compositional encoders, CM generates the representation of non-compositional phrases online, saving a lot parameters therefore avoiding the problem of overfitting. Besides, in this compositional manner, the model can handle a lot of variation in terms of morphology, lexicon, and syntax. For example, RM can model the idiom "go bananas" while "went bananas" is an unknown phrase for RM. By contrast, the case is more simple for CM since it knows "went" and "go" are similar (CM shares embedding layer with compositional encoder.).

4.3 Semantic Compositional Switch

A major element of our model is the introduction of a parameterized semantic compositional switch, which outputs a scalar α to determine whether a phrase is compositional or non-compostional. When the compositional switch is turned on at one node of a tree (i.e., when α equals 1), the model considers this phrase as non-compositional signifying that the meaning of the phrase is non-literal. Therefore, at each node of the tree, the model will compute the value of the switch. Here, we use a single-layer multilayer perceptron to compute α:

where Ws ∈ Rd×3d , vs ∈ Rd are the weight matrices. Finally, the hidden state of hj can be obtained as follows:

4.4 Context-dependent Phrase Composition

We consider in some cases the same phrase can act as both compositional and non-compositional in different contexts. For example, the phrase "around the corner" in the sentence "My brother played with friends around the corner" is compositional while in the sentence "The Spring Festival is just around the corner" is non-compositional.

In order to enable our model to disambiguate the meanings of phrases, we can easily extend above proposed model to achieve context-dependent phrase composition. Inspired by the work tracking LSTM proposed by Bowman et al. [2016], we choose a simple way with a little modification of Eq. 1. Specifically, we replace the word vector x with hidden state s, which is a hidden state emitted by a sequence-based LSTM.

where t = 1, 2, · · · , n and n represents the length of the input sequence.

5 Experiment

5.1 Training

To evaluation our models, we choose the task of sentiment classification and use five datasets. More formally, given a sentence X and its label l. The output ˆl of neural network is the probabilities of the different classes. The parameters of the network are trained to minimise the cross-entropy of the predicted and true label distributions.

where l is one-hot representation of the ground-truth label l;ˆl is predicted probabilities of labels; C is the class number.

To minimize the objective, we use stochastic gradient descent with the diagonal variant of AdaGrad [Duchi et al., 2011].

5.2 Competitor Methods

To make a comprehensive comparison, we choose several typical and powerful models as our baselines. The settings of some major competitor methods and our models are listed as follows:

• BiLSTM: Bi-directional LSTM.

• T-LSTM: LSTM over tree structure.
• HT-LSTM: A hybrid of tree and sequence LSTM, with the ability to learn context-dependent phrase composition.
• AdaHT-LSTM-RM: HT-LSTM with an adaptive compositional switch and RM non-compositional encoder.
• AdaHT-LSTM-CM: HT-LSTM with an adaptive compositional switch and CM non-compositional encoder.

Initialization and Hyperparameters In all of our experiments, the word and phrase embeddings are trained using word2vec [Mikolov et al., 2013] on the Wikipedia corpus (1B words). The other parameters are initialized by randomly sampling from uniform distribution in [-0.1, 0.1].

Generally, the longer the phrases are, the lower likelihood it is non-compositional. To reduce the computational cost, we only provide the additional option of non-compositional representation to the phrase containing not more than L words. L is a hyper-parameter, and is tuned on the development set.

For each task, we take the hyperparameters which achieve the best performance on the development set via an small grid search over combinations of the initial learning rate [0.1, 0.01, 0.001], l2 regularization [0.0, 5E-5, 1E-5] and the values of L [3, 4, 5, 6]. The final hyper-parameters are as follows. For the idiom-enriched dataset, the size of pretrained embedding is 300 while the size of hidden state is 200. The initial learning rate is 0.01. The regularization weight of the parameters is 10-5 . For the other datasets, the sizes of pre-trained embeddings and hidden states are both set as 300. The initial learning rate is 0.1. The regularization weight of the parameters is 0. Besides, the value of L is set to 4 for all datasets.

Data Preparation For all the sentences from the five datasets, we parse them with binarized constituency parser [Klein and Manning, 2003] to obtain the trees for our and some competitor models.

5.3 Experiment I: Idiom-enriched Sentiment Classification

To test the ability to model non-compositional phrases of our models, we carefully select an idiom-enriched sentiment classification dataset. Specifically, we choose this dataset due to the following reasons:

• Most of idioms are non-compositional phrases.
• Idioms typically imply an affective stance toward something rather than a neutral one. [Williams et al., 2015]
• the error analysis of sentiment classification results reveals that a large number of errors occur when idioms are used to express sentiment [Balahur et al., 2013].

We use the dataset proposed by [Williams et al., 2015]. In their work, they first collected a set of 580 idioms that are relevant to sentiment analysis, and then they assembled a corpus of 2521 sentences that contain an expression which can be matched to an idiom. In most cases, this expression will have a figurative meaning associated with an idiom, but in some cases it will convey a literal meaning. Moreover, each sentence was annotated with a sentiment label "positive","negative" or "other" to create a gold standard .

To better understand this task, we give some examples of this dataset as shown in Table 1.

Results Table 3 shows the evaluation results on idiomenriched sentiment classification. The classification performance was evaluated in terms of three measures: precision (Prec.), recall (Rec.) and F-measure (F). All the measures are computed by averaging metrics of each class and weighted by the number of true instances for each class.

From experimental results, we have several findings.

• AdaHT-LSTM consistently outperforms all the baseline with a significant margin and CM encoder shows a better performance than RM. We attribute the success of RM to its power in modeling variations of non-compositional phrases.
• Both sequenced-based and tree-based models perform better than NBOW, which indicates the importance of word orders. Additionally, T-LSTM outperforms LSTM with just a small margin.
• Compared with T-LSTM, HT-LSTM achieves a better performance, which shows the effectiveness of introducing context-dependent phrase composition.

Non-compositionality In order to understand where the performance improvement comes from, we analyze some cases and design an experiment to compare the output of HTLSTM and adptive HT-LSTM at each node of their corresponding trees.

More specifically, we randomly sample one sentence 1 from the development set, and the dynamical changes of the predicted sentiment scores over trees are shown in Figure 3.

The sentence "His performance is at fever pitch" has a positive sentiment. T-LSTM ignores the idiom "at fever pitch" and thereby consider the label of the sentence is "Other". By contrast, adaptive HT-LSTM captures this informative pattern and regards the phrase "at fever pitch" as non-compositional therefore making a correct prediction.

Disambiguation With the help of context-dependent LSTM, we find our model can perform disambiguation of those phrases which both have literal and non-literal meanings. As shown in Table 2, the phrase "wooden spoon" can both refer to "a kinds of spoon " and "the prize to be given to an individual or team which has come last in a competition".

Based on the different contexts, our model gives different compositional scores towards these two phrases thereby revealing its senses which are considered as indicators of sentiment. Another example also illustrates this case.

5.4 Experiment II: Large Datasets for Sentiment Classification

To make an extensive evaluation, we also choose several movie review datasets for sentiment classification, which not only have more training data, but contain many sentimentoriented idioms [Williams et al., 2015]. The detailed statistics about these four datasets are listed in Table 4. Each dataset is briefly described as follows.

• MR The movie reviews with two classes [Pang and Lee, 2005].

1 The pages of the paper is so limited that we can not give more examples.

• SST-1 The movie reviews with five classes (negative, somewhat negative, neutral, somewhat positive, positive) in the Stanford Sentiment Treebank 2 [Socher et al., 2013].
• SST-2 The movie reviews with binary classes. It is also from the Stanford Sentiment Treebank.
• SUBJ Subjectivity data set where the goal is to classify each instance (snippet) as being subjective or objective. [Pang and Lee, 2004]

Results Table 6 shows the classification accuracies of our proposed models compared with the competitor models. AdaHT-LSTM-CM consistently outperforms RAE, MVRNN, RTNT, T-LSTM and HT-LSTM by a large margin while achieving comparable results to the state-of-the-art. With the help of CM encoder, the performances on all four datasets are improved, indicating the effectiveness of it. Moreover, HT-LSTM performs better than T-LSTM, which shows the effectiveness of context-dependent phrase composition again.

Analysis of Non-compositional Phrases To get a better intuition of the behaviour of compositional switch, we examined the non-compositional scores of each phrases and look at what kinds of phrase can be assigned a higher score by our model. Specifically, we selected all the phrases whose noncompositional score are larger than 0.9 from all the five development sets. Surprisingly, we observe that most of those phrases with higher non-compositional scores are indeed noncompositional. Besides those phrases can be roughly summed up into six types: Proper Noun, Light-verb Phrases, Phrasal Verbs, Noun Phrases, Adverb Phrase and Idioms. We randomly sample some phrases and list all six categories as shown in Table 5.

We can see the model can not only pick up the proper noun such as: the name of movie "Star Wars saga" and the name of people "Barry Skolnick", but can identify the phrases with metaphorical and expanded meanings such as "short cuts" and "weak at the knee". Most of Table 6: Accuracies of our models on four datasets against stateof-the-art neural models. NBOW: Sums up the word vectors and applies a non-linearity followed by a softmax classification layer. RAE: Recursive Autoencoders with pre-trained word vectors from Wikipedia [Socher et al., 2011]. MV-RNN: Matrix-Vector Recursive Neural Network with parse trees [Socher et al., 2013]. RNTN: Recursive Neural Tensor Network with tensor-based feature function and parse trees [Socher et al., 2013]. DCNN: Dynamic Convolutional Neural Network with dynamic k-max pooling [Kalchbrenner et al., 2014; Denil et al., 2014]. CNN-multichannel: Convolutional Neural Network [Kim, 2014].

2 http://nlp.stanford.edu/sentiment.

these phrases either imply an affective stance toward something: such as "go bananas", "thumbs down", or are critical to the understanding of sentences such as the "Lightverb Phrases" and "Phrasal Verbs".

6 RelatedWork

There have been many studies, which focused on exploring the compositionality of various types of phrases. Kartsaklis et al. [2012] have discussed the advantages and disadvantages of using compositional or non-compositional embeddings. More recently, Hashimoto and Tsuruoka [2016] proposed to use compositionality scores to adaptively learn the phrase embeddings. Different with these models, we focus on sentence modelling and integrate compositional switch structure into neural sentence models, which can naturally encode the noncompositional phrases thereby obtaining more desirable sentence vectors. Furthermore, compared with above models, the composition of phrases in our models are context-sensitive and therefore can disambiguates the meanings of phrases.

Another thread of work is sentence modelling with various kinds of composition functions. Socher et al. [2010] proposed a context-sensitive recursive neural networks, which can model the context-dependent compositonality of phrases. Dong et al. [2014] proposed an adaptive recursive neural network, which can model the adaptive sentiment propagations as distributions over multiple composition functions. Tai et al. [2015] proposed tree-based LSTM, which builds on recursive neural networks. However, these models ignore the non-compositionality of phrases when modelling sentence. By contrast, we grow the capacity of syntax-based neural sentences model, allowing it can model the non-compositional phrases when learning sentence representation.

7 Conclusion

In this paper, we introduce a context-dependent controller in recursively neural sentence modelling, which adaptively determines whether the representation of a phrase should be composed of its containing words or subphrases. Experiments on five sentiment classification datasets demonstrate the efficacy of our proposed model and its superiority to competitor models. Furthermore, we have made an elaborate experiment design and case analysis to evaluate the effectiveness of our proposed models.

The proposed model approaches one step towards understanding collocation (such as idioms) in a sentence. In future, we want to investigate how to jointly identify and understand the collocations appeared in the sentences.

Acknowledgments

We would like to thank the anonymous reviewers for their valuable comments and thank Kaiyu Qian for useful discussions. This work was partially funded by National Natural Science Foundation of China (No. 61532011 and 61672162), Shanghai Municipal Science and Technology Commission (No. 16JC1420401).