Deep Mask Memory Network with Semantic Dependency and Context Moment for Aspect Level Sentiment Classification

1 Introduction

Aspect level sentiment classification, which aims to identify the sentiment polarity (i.e., negative, neutral, or positive) of each aspect term in a sentence, has received much attention these years [Chen et al., 2017; Liu et al., 2018; Li et al., 2018]. Different from the sentiment analysis for a sentence, aspect level sentiment classification conducts a finer sentiment analysis. For example, given the sentence "The price is very cheap, but the service is very poor", the sentiment polarity of the sentence is positive if the aspect is "price" while the sentiment polarity is negative if the aspect is "service". It's obvious that the same sentence may have different sentiment polarity for different opinion aspects.

∗The corresponding author is Meng Yang.

Early methods for aspect-based sentiment analysis mostly adopted supervised learning approaches with hand-crafted features [Rao and Ravichandran, 2009; Jiang et al., 2011; Perez-Rosas et al., 2012; Kiritchenko et al., 2014; Vo and Zhang, 2015]. However, the performance of these models is highly dependent on the quality of features and feature engineering is labor intensive.

In recent years, deep neural networks have been employed to learn useful features from aspects and contexts automatically, achieving promising results on the aspect level sentiment classification task. [Dong et al., 2014] learned the aspect-aware representation via semantic composition over dependency tree. [Vo and Zhang, 2015] developed a lexiconenhanced network which used sentiment lexicon and neural pooling functions. The popular convolutional neural networks (CNNs) and long short-term memory (LSTM) have also been applied in this task. [Xue and Li, 2018] proposed a simpler and easily-parallelized model based on CNNs and gating mechanisms; [Li et al., 2018] applied a transformation network (TNet) which employed CNNs to extract salient features and a component to generate target-specific representations; [Tang et al., 2015] used two target-Dependent LSTM networks to separately represent text and targets, and classified the sentiment of the desired aspect by adopting a feedforward network. However, these methods capture context information in an implicit way, and are incapable of explicitly exhibiting important context clues of an aspect.

Recently, attention mechanism and memory network have begun to be used in natural language processing tasks [Bahdanau et al., 2014; Weston et al., 2014; Sukhbaatar et al., 2015]. Attention mechanism can capture important parts of a sentence [Wang et al., 2016; Ma et al., 2017; Liu et al., 2018], while memory network can store context information [Tang et al., 2016; Chen et al., 2017]. For instance, [Liu and Zhang, 2017] applied a bidirectional LSTM network with attention and gates (BILSTM-ATT-G) to measure the importance of context; [Ma et al., 2018] promoted the LSTM network with attention mechanism and common-sense knowledge; [Tay et al., 2017] enabled rich dyadic interaction between aspect and document; [Fan et al., 2018] proposed a convolutional memory network (Conv-Memnet) with an attention mechanism. Moreover, attention mechanism or bidirectional long shortterm memory (BiLSTM) has also been proposed to consider the influence of other aspects in the same sentence [Majumder et al., 2018; Hazarika et al., 2018].

Although improved performance has been achieved by the above methods, semantic information between context and aspect and the relation between aspects in the same sentence are not well exploited. To solve the above issues, we propose a novel framework of deep mask memory network with semantic dependency and context moment (DMMN-SDCM) for aspect sentiment classification. In the proposed DMMNSDCM, the semantic parsing information is integrated into deep memory network to guide the attention mechanism and inter-aspect learning network, which selects different parts of the context memory for different computational layers and utilizes the information provided by the nearby aspects in the sentence, respectively. To introduce the whole sentiment information, we jointly conduct a context moment learning task to explore the sentiment distribution of the entire sentence, which is able to provide a background for the desired aspect. The experiments on SemEval 2014 Datasets clearly show that our model achieves a state-of-the-art performance. The major contributions of this paper are three-fold:

1. Integrating semantic parsing information instead of location information into deep memory network to guide the attention mechanism;
2. Integrating semantic parsing information into interaspect modeling for utilizing the information provided by the nearby aspects in a better way;
3. Designing an auxiliary task to learn the sentiment distribution of the entire sentence, which can provide a background for the sentiment analysis of the desired aspect;

2 RelatedWork

The deep learning models with attention mechanism and memory network has been applied to the aspect-level sentiment classification, with promising performance reported.

Attention mechanism has been widely applied in sentiment classification. [Wang et al., 2016] proposed the attentionbased LSTM with aspect embedding (ATAE-LSTM), which firstly applied attention mechanism to aspect level sentiment classification by simply concatenating the aspect vector into the sentence hidden representations and achieved a good performance. [Ma et al., 2017] developed an interactive attention network (IAN) to learn the representations of aspects and contexts, respectively. [Liu et al., 2018] applied sentencelevel content attention mechanism and learned the weights of the memory with two gated recurrent unit (GRU) networks.

Memory network is also used to capture the context information. [Tang et al., 2016] first used deep memory network (MemNet), in which attention mechanism with multiple computational layers is applied over an external memory and location attention is integrated into content location. [Chen et al., 2017] proposed a recurrent attention network which used a BiLSTM model to generate the memory and combined all attention results in a nonlinear way by using a GRU model.

To utilize the information of inter-aspect relation, some effective inter-aspect relation modeling methods have been proposed. [Majumder et al., 2018] applied attention mechanism to model the relation between aspects while [Hazarika et al., 2018] applied a BiLSTM network.

Although deep memory network and inter-aspect modeling have achieved promising results on aspect level sentiment classification, there are still some problems unsolved.

Firstly, the semantic information of aspect in the sentence can't be well exploited. The deep memory network approaches [Tang et al., 2016; Chen et al., 2017], which generate location information according to the absolute distance from each context word to aspect, can't capture important related words in semantic analysis. For example, given the sentence "Food is good but service is bad" and the aspect word "service", the context word "good" is considered to be as important as the context word "bad" by the traditional methods.

Secondly, the global information of the sentence, including the sentiment polarity of the entire sentence and the relation between aspects, isn't well exploited in previous methods. For example, given the sentence "service and food are good.", it's easy to know the sentiment results of "service" and "food" if we know the sentiment of the sentence is positive and the relation between the two aspects is coordination.

3 Model

To address the issues above, we proposed a novel model for aspect level sentiment classification, deep mask memory network with semantic dependency and context moment (DMMN-SDCM). Based on deep memory network, we introduce the information of semantic parsing to guide the attention mechanism and effectively learn the information provided by other aspects in the same sentence. Meanwhile the context moment embedded in the sentiment of entire sentence is designed to provide a background for the desired aspect. In the following, we will first give an overview of the proposed DMMN-SDCM, and then detail the semantic-dependencymask attention, the inter-aspect semantic modeling and the context-moment sentiment learning. In our paper, the operation of (x, y) and (x, y, z) are used to denote the concatenation of two vectors (e.g., x and y) and three vectors (e.g., x, y, and z), respectively.

3.1 Overview of DMMN-SDCM

The architecture of DMMN-SDCM is shown in Fig. 1, which is composed of six modules: embedding module, memory building module, semantic-dependency-mask attention module, inter-aspect semantic modeling module, context-moment sentiment learning module, and classification module.

In our model, as the task setting of aspect level sentiment classification [Li et al., 2018], the aspects appearing in the sentence have been extracted and identifying the sentiment of the given aspects is the focus. To avoid aspect information from interfering with context modeling, we replace each aspect, which may be word or a phrase, with a placeholder.

The embedding module uses a word embedding lookup table, which can be pre-trained by unsupervised methods like Word2Vec [Mikolov et al., 2013], Glove [Pennington et al., 2014] or FastText [Joulin et al., 2016; Bojanowski et al., 2016], to convert most words to theirs word vectors, then initialize other out-of-vocabulary words by sampling from the uniform distribution.

To build an external memory, the memory building module applys a BiLSTM network, which can capture the relationship between words in a sentence. After modeling, the final memory generated is {m∗1 , . . . ,m∗i , . . . ,m∗n }, where n stands for the length of sentence andm∗i = (-→ hi, ←- hi), with -→ hi and ←- hi denoting the hidden states generated by the forward and the backward LSTMs of BiLSTM, respectively.

The semantic-dependency-mask attention module extracts semantic parsing information, and then selects different slices of memory dynamically for different computational layers to guide the attention mechanism. The output of the module for the desired aspect is denoted as vsd.

To utilize the information of other aspects in the same sentence, the inter-aspect semantic modeling module and the context-moment sentiment learning module are designed. The former module uses semantic dependency information and attention mechanism to get the useful information of other aspects, while the latter module applies a context moment learning task to learn the relation between all aspects in the entire sentence. The outputs of the two modules are denoted as vim and vcm.

With the last three modules, we can get the sentiment representation of the desired aspect:

The classification module adopt a feed-forward network to project v into the space of the targeted C classes.:

where y = [y1, · · · , yi, · · · , yC] ∈ RC is the estimated probability, and Wc and bc are weight matrix and bias, respectively. The model is finally trained by minimizing the sum of the cross entropy, the L2 regularization term and the context moment learning loss:

where S is the sentence set of the training data and T is the aspect set of the sentence s. gt = [gt1 , · · · , gt i , · · · , gtC ] ∈ RC is the ground truth for aspect t, which is represented by a one-hot vector where the element for the correct polarity is 1. yt = [yt1 , · · · , yt i , · · · , ytC ] ∈ RC is the estimated probability for aspect t. λ is the weight of L2 regularization term, and λm is the weight of the context moment learning loss. The term of lm(s) is the context moment learning loss for the sentence s, which will be detailed in the following section.

3.2 Semantic-Dependency-Mask Attention

In this section, we present the semantic-dependency-mask attention, as shown in the right part of Fig. 2. To guide the attention mechanism, the module masks the context memory for different computational hops according to the semantic dependency parsing tree. At each computational hop, the module applies attention mechanism between the representation of aspect and every memory unit. The results of multiple attention layers are combined in a nonlinear way.

We use a open-source library spaCy 1 , whose parsing model is a blend of recent methods, to generate the dependency parsing tree. In a dependency tree, the distances from context word to different aspects are different, so that we can get different semantic parsing information for different aspects.

Depending on the distance between each context words and the aspect, we can extract semantic parsing information from the dependency tree of the sentence. For different layers in deep memory network, the model selects different parts of memory depending on semantic parsing information. At l-th layer, each memory unit masked with semantic dependency parsing information can be represented as follows:

where m∗i stands for the original memory unit generated by the BiLSTM, and dist(wi,wt) is the length of the path from the aspect word wt to the context word wi in the semantic dependency tree of the sentence. Therefore, the final masked memory generated at l-th layer is {ml1 , . . . ,mli , . . . ,mln }.

With masked memory, we can get the normalized attention score αli of each memory unit at l-th layer mli as follows:

where W lAL and blAL denote weight matrices and biases of the l-th layer, respectively, and mli and vl-1 sd stand for the memory unit and the output from the previous layer, respectively. Here vt denotes the aspect representation, which is generated by adopting a BiLSTM layer and an attention layer.

And the attention output iAL l is calculated as:

To generate the final representation of the l-th layer, we adopt a linear layer as well as a transform layer, which introduces a gating function to control the passed proportions of the transformed features and the input features [Srivastava et al., 2015]:

where Wo and bo are weight matrices and biases of the linear layer, respectively. And the gate T as follows:

where σ stands for the sigmoid activation function, andWt and bt are weight matrices and biases, respectively.

Finally, the output of the last layer will be taken as the output of the semantic-dependency-mask attention module vsd.

1 https://spacy.io/

Semantic-dependency-mask Attention

3.3 Inter-Aspect Semantic Modeling

To model the inter-aspect relation, we propose a novel framework which integrates semantic dependency information into attention mechanism (shown in the left part of Fig. 2). After applying the semantic mask attention, we can get p aspectaware representations for all p aspects in the sentence. Then we stack the representations as the aspect memory according to the the order of occurrence of their corresponding aspectterms. The aspect memory can be represented as follows:

For different aspects in the same sentence, their original aspect memories are same. To introduce the information of the desired aspect to the aspect memory, we concatenate every memory unit and the representation of the desired aspect vsdt , and weight the memory according to the semantic dependency information.

where v′sdi = λi ∗ (vsdi , vsdt ) and λi is the semantic dependency weight, which is calculated as follows:

where dist(wi,wt) is the semantic distance from the current aspect wi to the desired aspect wt and depth stands for the depth of the dependency tree.

Then we apply the attention mechanism and get the final representation vim.

where Wim and bim are weight matrices and biases.

3.4 Context-Moment Sentiment Learning

For a sentence with multiple aspects, there are generally some sentiment relations between aspects, like comparisons and coordinations. It will be helpful to identify the aspect sentiment polarity if the relation between aspects is introduced to the model, since these kind of relations can provide the sentence sentiment background for sentiment classification task. In this section, we propose a context moment learning task to capture the distribution of the sentence sentiment, and design a context-moment sentiment learning module, as as shown in Fig. 3.

where E is the expectation operator, x is the random variable of the sentiment distribution, µ is the mean of the distribution, and i stands for the rank of the moment. It's easy to know µi ∈ [-1, 1] if the rank is odd while µi ∈ [0, 1] if even. Therefore we normalize all the moments to [0, 1].

To illustrate the context moment descriptors clearer, we give some examples shown in Table. 1. From the table, we observe that the greater the mean, the more positive the overall sentiment of the sentence. In addition, the closer the variance is to 1, the closer the relation between aspects is to comparison, while the closer the variance is to 0, the closer the relation is to coordination. Therefore, the context moment can describe the overall sentiment of the sentence and the relation between aspects.

To learn the estimation of context moments like mean and variance, the module applies an attention layer and a twolayer feed-forward network. And then a context-moment loss for a sentence s is defined to help the optimization of aspect sentiment classification task:

where k stands for the greatest rank of the moment used, µi is the value of the ith moment while µ′i is the estimate value.

To portray the sentiment distribution of the overall sentence, we use the first moment µ1 and the second moment µ2 as the learning goal of context-moment learning. To learn these moments, we firstly apply attention mechanism to get the sentence representation v1s and v2s for the first and second moment, respectively, then use a two-layer fully connected network can get the estimates of moments. For example, we can get the mean representation of the sentence:

Then the estimation of the first moment is obtained:

where vglobal is calculated by taking the average of the all p representations of the aspects generated by the semantic mask attention module:

By the same way, we can get the variance representation vvariance and the estimate of the second moment µ′2 .

Finally, the module will generate the corresponding representations of the sentence for each moment, which are embedded to the classification module for the analysis of the aspect sentiment. Specifically, The mean representation and the variance representation will be concatenated as the final output of the module:vcm = (vmean, vvariance) (19)

4 Experiment

4.1 Experimental Setup

To verify the effectiveness of DMMN-SDCM, we do experiments on SemEval 2014 Task 4 [Pontiki et al., 2014]. The SemEval 2014 reviews dataset consists of Laptop dataset and Restaurant dataset. The reviews are labeled with four sentiment polarities: positive, neutral, negative and conflict. We remove conflict category as the number of conflict samples is very small and make the dataset extremely unbalanced. The details of the experimental datasets are shown in Table. 2

The embedding table used in experiments is pre-trained by Glove [Pennington et al., 2014]. The out-of-vocabulary words are initialized by sampling from the uniform distribution U(-0.25, 0.25). The dimension of word vectors is 300.

We employ Adam Optimizer [Kingma and Ba, 2014] to train our model. The initial learning rate is 0.01, the weight of L2 regularization term is 0.0001, the weight of context moment loss is 1.5 and the dropout rate is 0.5. The dimension of LSTM hidden states and output representation is 50. The evaluation metrics are Accuracy and Macro-F1.

4.2 Comparison with Baselines

To evaluate the performance of DMMN-SDCM, we compare our model to some baseline methods: TD-LSTM [Tang et al., 2015], ATAE-LSTM [Wang et al., 2016], MemNet [Tang et al., 2016], IAN [Ma et al., 2017], BILSTM-ATT-G [Liu and Zhang, 2017], RAM [Chen et al., 2017], Conv-Memnet [Fan et al., 2018] and TNet [Li et al., 2018]. The results of performance comparison is shown in Table. 3.

From the experimental results, we can observe that DMMN-SDCM outperforms all baselines in both Restaurant reviews and Laptop reviews. Our model has the best performance through integrating semantic dependency information and context moment information into deep memory network. Semantic dependency information can not only provide dynamic and reasonable memory for each attention layer, but also guide the inter-aspect attention mechanism to capture the important information of other aspects. Moreover, the context-moment learning captures the sentence sentiment distribution, which is helpful for aspect sentiment classification.

4.3 Effects of EachModule

To examine the effectiveness of each module, we conduct extensive experiments and the results are shown in Table. 4.

From Table. 4, integrating semantic dependency information into deep memory is effective since the model which only contains the mask attention module outperforms RAM [Chen et al., 2017], which applies position-weighting attention. Moreover, the two modules, context moment learning and inter-aspect modeling, improve the performance of the model too, since both of them utilize the relation between aspects in the sentence. With the three creative modules, our model outperforms all the baselines.

4.4 Effects of Inter-Aspect Modeling

To verify if semantic-dependency weighting method works, we conduct experiments and the results are shown in Table. 5.

We can observe that weighting strategy can get improved performance whatever method the model uses to model interaspect relation. In addition, we can see that attention mechanism [Majumder et al., 2018] is more effective than BiLSTM

Table 6: Effects of moment ranks on the Laptop dataset. "Mean", "Variance" and "Skewness" are the first, second and third moments.

[Hazarika et al., 2018] for modeling inter-aspect relation.

4.5 Effects of Moment Ranks

Introducing different context moments to the model may have different effects. We evaluate our model with using different context moments, and the results are shown in Table. 6.

From the table, we can see that either using the first moment only or using the second moment only can't achieve good performance, for the reason that insufficient information is obtained. In addition, introducing the higher-rank moment isn't effective too, as it may bring interference information as the aspect sentiment distribution is not complex enough.

As the experimental results show, using the first and the second moment can perform best. The mean representation can model the sentiment of the whole sentence while the variance representation can model the inter-aspect relation.

5 Conclusion

In this paper, we design a deep mask memory network with semantic dependency and context moment (DMMN-SDCM) which integrates semantic parsing information and context moment learning into deep memory network for the first time. With the semantic dependency, we proposed a more discriminative attention scheme, which effectively selects different parts of the context memory for different computational layers, and presented an effective method to model the interaspect relation. The sentiment distribution of the entire sentence is also encoded by using a context moment for the first time, which guides the deep memory network to learn an effective feature. We have conducted extensive experiments on SemEval 2014 review datasets, and the experiment results clearly show that our model performance is state-of-the-art.

Acknowledgments

This work is partially supported by the National Natural Science Foundation of China (Grant no.61772568), the Key Areas Research and Development Program of Guangdong (Grant no.2018B010109007), the Guangzhou Science and Technology Program (Grant no. 201804010288), and the Fundamental Research Funds for the Central Universities (Grant no.18lgzd15).