Aspect Term Extraction with History Attention and Selective Transformation

1 Introduction

Aspect-Based Sentiment Analysis (ABSA) involves detecting opinion targets and locating opinion indicators in sentences in product review texts [Liu, 2012]. The first sub-task, called Aspect Term Extraction (ATE), is to identify the phrases targeted by opinion indicators in review sentences. For example, in the sentence "I love the operating system and preloaded software", the words "operating system" and "preloaded software" should be extracted as aspect terms, and the sentiment on them is conveyed by the opinion word "love". According to the task definition, for a term/phrase being regarded as an aspect, it should co-occur with some "opinion words" that indicate a sentiment polarity on it [Pontiki et al., 2014].

Many researchers formulated ATE as a sequence labeling problem or a token-level classification problem. Traditional sequence models such as Conditional Random Fields (CRFs) [Chernyshevich, 2014; Toh and Wang, 2014; Toh and Su, 2016; Yin et al., 2016], Long Short-Term Memory Networks (LSTMs) [Liu et al., 2015] and classification models such as Support Vector Machine (SVM) [Manek et al., 2016] have been applied to tackle the ATE task, and achieved reasonable performance. One drawback of these existing works is that they do not exploit the fact that, according to the task definition, aspect terms should co-occur with opinionindicating words. Thus, the above methods tend to output false positives on those frequently used aspect terms in nonopinionated sentences, e.g., the word "restaurant" in "the restaurant was packed at first, so we waited for 20 minutes", which should not be extracted because the sentence does not convey any opinion on it.

∗The work was done when Xin Li was an intern at Tencent AI Lab. The project is substantially supported by a grant from the Research Grant Council of the Hong Kong Special Administrative Region, China (Project Code: 14203414).

1 Codes are available at https://github.com/lixin4ever/HAST.

There are a few works that consider opinion terms when tackling the ATE task. [Wang et al., 2016] proposed Recursive Neural Conditional Random Fields (RNCRF) to explicitly extract aspects and opinions in a single framework. Aspect-opinion relation is modeled via joint extraction and dependency-based representation learning. One assumption of RNCRF is that dependency parsing will capture the relation between aspect terms and opinion words in the same sentence so that the joint extraction can benefit. Such assumption is usually valid for simple sentences, but rather fragile for some complicated structures, such as clauses and parenthesis. Moreover, RNCRF suffers from errors of dependency parsing because its network construction hinges on the dependency tree of inputs. CMLA [Wang et al., 2017] models aspect-opinion relation without using syntactic information. Instead, it enables the two tasks to share information via attention mechanism. For example, it exploits the global opinion information by directly computing the association score between the aspect prototype and individual opinion hidden representations and then performing weighted aggregation. However, such aggregation may introduce noise. To some extent, this drawback is inherited from the attention mechanism, as also observed in machine translation [Luong et al., 2015] and image captioning [Xu et al., 2015].

To make better use of opinion information to assist aspect term extraction, we distill the opinion information of the whole input sentence into opinion summary 2 , and such distillation is conditioned on a particular current token for aspect prediction. Then, the opinion summary is employed as part of features for the current aspect prediction. Taking the sentence "the restaurant is cute but not upscale" as an example, when our model performs the prediction for the word "restaurant", it first generates an opinion summary of the entire sentence conditioned on "restaurant". Due to the strong correlation between "restaurant' and "upscale" (an opinion word), the opinion summary will convey more information of "upscale" so that it will help predict "restaurant" as an aspect with high probability. Note that the opinion summary is built on the initial opinion features coming from an auxiliary opinion detection task, and such initial features already distinguish opinion words to some extent. Moreover, we propose a novel transformation network that helps strengthen the favorable correlations, e.g. between "restaurant' and "upscale", so that the produced opinion summary involves less noise.

2 Technically, opinion summary is the linear combination of the opinion representations generated from LSTM.

Besides the opinion summary, another useful clue we explore is the aspect prediction history due to the inspiration of two observations: (1) In sequential labeling, the predictions at the previous time steps are useful clues for reducing the error space of the current prediction. For example, in the B-I-O tagging (refer to Section 2.1), if the previous prediction is "O", then the current prediction cannot be "I"; (2) It is observed that some sentences contain multiple aspect terms. For example, "Apple is unmatched in product quality, aesthetics, craftmanship, and customer service" has a coordinate structure of aspects. Under this structure, the previously predicted commonly-used aspect terms (e.g., "product quality") can guide the model to find the infrequent aspect terms (e.g., "craftmanship"). To capture the above clues, our model distills the information of the previous aspect detection for making a better prediction on the current state.

Concretely, we propose a framework for more accurate aspect term extraction by exploiting the opinion summary and the aspect detection history. Firstly, we employ two standard Long-Short Term Memory Networks (LSTMs) for building the initial aspect and opinion representations recording the sequential information. To encode the historical information into the initial aspect representations at each time step, we propose truncated history attention to distill useful features from the most recent aspect predictions and generate the history-aware aspect representations. We also design a selective transformation network to obtain the opinion summary at each time step. Specifically, we apply the aspect information to transform the initial opinion representations and apply attention over the transformed representations to generate the opinion summary. Experimental results show that our framework can outperform state-of-the-art methods.

2 The ProposedModel

2.1 The ATE Task

Given a sequence X = {x1, ..., xT} of T words, the ATE task can be formulated as a token/word level sequence labeling problem to predict an aspect label sequence Y = {y1, ..., yT }, where each yi comes from a finite label set Y = {B, I,O} which describes the possible aspect labels. As shown in the example below:

B, I, and O denote beginning of, inside and outside of the aspect span respectively. Note that in commonly-used datasets such as [Pontiki et al., 2016], the gold standard opinions are usually not annotated.

2.2 Model Description

As shown in Figure 1, our model contains two key components, namely Truncated History-Attention (THA) and Selective Transformation Network (STN), for capturing aspect detection history and opinion summary respectively. THA and STN are built on two LSTMs that generate the initial word representations for the primary ATE task and the auxiliary opinion detection task respectively. THA is designed to integrate the information of aspect detection history into the current aspect feature to generate a new history-aware aspect representation. STN first calculates a new opinion representation conditioned on the current aspect candidate. Then, we employ a bi-linear attention network to calculate the opinion summary as the weighted sum of the new opinion representations, according to their associations with the current aspect representation. Finally, the history-aware aspect representation and the opinion summary are concatenated as features for aspect prediction of the current time step.

Building Memory

As Recurrent Neural Networks can record the sequential information [Graves, 2012], we employ two vanilla LSTMs to build the initial token-level contextualized representations for sequence labeling of the ATE task and the auxiliary opinion word detection task respectively. For simplicity, let LSTMT (xt) denote an LSTM unit where T ∈ {A, O} is the task indicator. In the following sections, without specification, the symbols with superscript A and O are the notations used in the ATE task and the opinion detection task respectively. We use Bi-Directional LSTM to generate the initial token-level representations hTt ∈ R2dimTh (dimTh is the dimension of hidden states):

Capturing Aspect History

In principle, RNN can memorize the entire history of the predictions [Graves, 2012], but there is no mechanism to exploit the relation between previous predictions and the current prediction. As discussed above, such relation could be useful because of two reasons: (1) reducing the model's error space in predicting the current label by considering the definition of B-I-O schema, (2) improving the prediction accuracy for multiple aspects in one coordinate structure.

We propose a Truncated History-Attention (THA) component (the THA block in Figure 1) to explicitly model the aspect-aspect relation. Specifically, THA caches the most recent N A hidden states. At the current prediction time step t, THA calculates the normalized importance score sti of each cached state hAi (i ∈ [t -N A, t -1]) as follows:

˜hAi denotes the previous history-aware aspect representation (refer to Eq. 5). v ∈ R2dimAh can be learned during training. W1,2,3 ∈ R2dimAh ×2dimAh are parameters associated with previous aspect representations, current aspect representation and previous history-aware aspect representations respectively. Then, the aspect history ˆhAt is obtained as follows:

To benefit from the previous aspect detection, we consolidate the hidden aspect representation with the distilled aspect history to generate features for the current prediction. Specifically, we adopt a way similar to the residual block [He et al., 2016], which is shown to be useful in refining word-level features in Machine Translation [Wu et al., 2016] and Part-OfSpeech tagging [Bjerva et al., 2016], to calculate the historyaware aspect representations ˜hAt at the time step t:

where ReLU is the relu activation function.

Capturing Opinion Summary

Previous works show that modeling aspect-opinion association is helpful to improve the accuracy of ATE, as exemplified in employing attention mechanism for calculating the opinion information [Wang et al., 2017; Li and Lam, 2017]. MIN [Li and Lam, 2017] focuses on a few surrounding opinion representations and computes their importance scores according to the proximity and the opinion salience derived from a given opinion lexicon. However, it is unable to capture the longrange association between aspects and opinions. Besides, the association is not strong because only the distance information is modeled. Although CMLA [Wang et al., 2017] can exploit global opinion information for aspect extraction, it may suffer from the noise brought in by attention-based feature aggregation. Taking the aspect term "fish" in "Furthermore, while the fish is unquestionably fresh, rolls tend to be inexplicably bland." as an example, it might be enough to tell "fish" is an aspect given the appearance of the strongly related opinion "fresh". However, CMLA employs conventional attention and does not have a mechanism to suppress the noise caused by other terms such as "rolls". Dependency parsing seems to be a good solution for finding the most related opinion and indeed it was utilized in [Wang et al., 2016], but the parser is prone to generating mistakes when processing the informal online reviews, as discussed in [Li and Lam, 2017].

To make use of opinion information and suppress the possible noise, we propose a novel Selective Transformation Network (STN) (the STN block in Figure 1), and insert it before attending to global opinion features so that more important features with respect to a given aspect candidate will be highlighted. Specifically, STN first calculates a new opinion representation ˆhOi,t given the current aspect feature ˜hAt as follows:

where W4 and W5 ∈ R2dimOh ×2dimOh are parameters for history-aware aspect representations and opinion representations respectively. They map ˜hAt and hOi to the same subspace. Here the aspect feature ˜hAt acts as a "filter" to keep more important opinion features. Equation 6 also introduces a residual block to obtain a better opinion representation ˆhOi,t , which is conditioned on the current aspect feature ˜hAt .

For distilling the global opinion summary, we introduce a bi-linear term to calculate the association score between ˜hAt and each ˆhOi,t :

whereWbi and bbi are parameters of the Bi-Linear Attention layer. The improved opinion summary ˆhOt at the time t is obtained via the weighted sum of the opinion representations:

Finally, we concatenate the opinion summary ˆhOt and the history-aware aspect representation ˜hAt and feed it into the top-most fully-connected (FC) layer for aspect prediction:

Note that our framework actually performs a multi-task learning, i.e. predicting both aspects and opinions. We regard the initial token-level representations hOi as the features for opinion prediction:

WT f and bTf are parameters of the FC layers.

2.3 Joint Training

All the components in the proposed framework are differentiable. Thus, our framework can be efficiently trained with gradient methods. We use the token-level cross-entropy error between the predicted distribution P(yT t |xt) (T ∈ {A,O}) and the gold distribution P(yT ,g t |xt) as the loss function:

Then, the losses from both tasks are combined to form the training objective of the entire model:

where LA and LO represent the loss functions for aspect and opinion extractions respectively.

3 Experiment

3.1 Datasets

To evaluate the effectiveness of the proposed framework for the ATE task, we conduct experiments over four benchmark datasets from the SemEval ABSA challenge [Pontiki et al., 2014; Pontiki et al., 2015; Pontiki et al., 2016]. Table 1 shows their statistics. D1 (SemEval 2014) contains reviews of the laptop domain and those of D2 (SemEval 2014), D3 (SemEval 2015) and D4 (SemEval 2016) are for the restaurant domain. In these datasets, aspect terms have been labeled by the task organizer.

Gold standard annotations for opinion words are not provided. Thus, we choose words with strong subjectivity from MPQA 3 to provide the distant supervision [Mintz et al., 2009]. To compare with the best SemEval systems and the current state-of-the-art methods, we use the standard traintest split in SemEval challenge as shown in Table 1.

3 http://mpqa.cs.pitt.edu/

3.2 Comparisons

We compare our framework with the following methods:

• CRF-1: Conditional Random Fields with basic feature templates 4 .
• CRF-2: Conditional Random Fields with basic feature templates and word embeddings.
• Semi-CRF: First-order Semi-Markov Conditional Random Fields [Sarawagi et al., 2004] and the feature templates in Cuong et al. [2014] are adopted.
• LSTM: Vanilla bi-directional LSTM with pre-trained word embeddings.
• IHS RD [Chernyshevich, 2014], DLIREC [Toh and Wang, 2014], EliXa [San Vicente et al., 2015], NLANGP [Toh and Su, 2016]: The winning systems in the ATE subtask in SemEval ABSA challenge [Pontiki et al., 2014; Pontiki et al., 2015; Pontiki et al., 2016].
• WDEmb [Yin et al., 2016]: Enhanced CRF with word embeddings, dependency path embeddings and linear context embeddings.
• MIN [Li and Lam, 2017]: MIN consists of three LSTMs. Two LSTMs are employed to model the memory interactions between ATE and opinion detection. The last one is a vanilla LSTM used to predict the subjectivity of the sentence as additional guidance.
• RNCRF [Wang et al., 2016]: CRF with high-level representations learned from Dependency Tree based Recursive Neural Network.
• CMLA [Wang et al., 2017]: CMLA is a multi-layer architecture where each layer consists of two coupled GRUs to model the relation between aspect terms and opinion words.

To clarify, our framework aims at extracting aspect terms where the opinion information is employed as auxiliary, while RNCRF and CMLA perform joint extraction of aspects and opinions. Nevertheless, the comparison between our framework and RNCRF/CMLA is still fair, because we do not use manually annotated opinions as used by RNCRF and CMLA, instead, we employ an existing opinion lexicon to provide weak opinion supervision.

3.3 Settings

We pre-processed each dataset by lowercasing all words and replace all punctuations with PUNCT. We use pre-trained GloVe 840B vectors 5 [Pennington et al., 2014] to initialize the word embeddings and the dimension (i.e., dimw) is 300. For out-of-vocabulary words, we randomly sample their embeddings from the uniform distribution U(-0.25, 0.25) as done in [Kim, 2014]. All of the weight matrices except those in LSTMs are initialized from the uniform distribution U(-0.2, 0.2). For the initialization of the matrices in LSTMs, we adopt Glorot Uniform strategy [Glorot and Bengio, 2010]. Besides, all biases are initialized as 0's.

4 http://sklearn-crfsuite.readthedocs.io/en/latest/

The model is trained with SGD. We apply dropout over the ultimate aspect/opinion features and the input word embeddings of LSTMs. The dropout rates are empirically set as 0.5. With 5-fold cross-validation on the training data of D2, other hyper-parameters are set as follows: dimAh = 100, dimOh = 30; the number of cached historical aspect representations N A is 5; the learning rate of SGD is 0.07.

3.4 Main Results

As shown in Table 2, the proposed framework consistently obtains the best scores on all of the four datasets. Compared with the winning systems of SemEval ABSA, our framework achieves 5.0%, 1.6%, 1.4%, 1.3% absolute gains on D1, D2, D3 and D4 respectively.

Our framework can outperform RNCRF, a state-of-the-art model based on dependency parsing, on all datasets. We also notice that RNCRF does not perform well on D3 and D4 (3.7% and 3.9% inferior than ours). We find that D3 and D4 contain many informal reviews, thus RNCRF's performance degradation is probably due to the errors from the dependency parser when processing such informal texts.

CMLA and MIN do not rely on dependency parsing, instead, they employ attention mechanism to distill opinion information to help aspect extraction. Our framework consistently performs better than them. The gains presumably come from two perspectives: (1) In our model, the opinion summary is exploited after performing the selective transformation conditioned on the current aspect features, thus the summary can to some extent avoid the noise due to directly applying conventional attention. (2) Our model can discover some uncommon aspects under the guidance of some commonlyused aspects in coordinate structures by the history attention.

5 https://nlp.stanford.edu/projects/glove/

CRF with basic feature template is not strong, therefore, we add CRF-2 as another baseline. As shown in Table 2, CRF-2 with word embeddings achieves much better results than CRF-1 on all datasets. WDEmb, which is also an enhanced CRF-based method using additional dependency context embeddings, obtains superior performances than CRF-2. Therefore, the above comparison shows that word embeddings are useful and the embeddings incorporating structure information can further improve the performance.

3.5 Ablation Study

To further investigate the efficacy of the key components in our framework, namely, THA and STN, we perform ablation study as shown in the second block of Table 2. The results show that each of THA and STN is helpful for improving the performance, and the contribution of STN is slightly larger than THA. "OURS w/o THA & STN" only keeps the basic bi-linear attention. Although it performs not bad, it is still less competitive compared with the strongest baseline (i.e., CMLA), suggesting that only using attention mechanism to distill opinion summary is not enough. After inserting the STN component before the bi-linear attention, i.e. "OURS w/o THA", we get about 1% absolute gains on each dataset, and then the performance is comparable to CMLA. By adding THA, i.e. "OURS", the performance is further improved, and all state-of-the-art methods are surpassed.

3.6 Attention Visualization and Case Study

In Figure 2, we visualize the opinion attention scores of the words in two example sentences with the candidate aspects "maitre-D" and "bathroom". The scores in Figures 2a and 2c show that our full model captures the related opinion words very accurately with significantly larger scores, i.e. "incredibly", "unwelcoming" and "arrogant" for "maitreD", and "unfriendly" and "filthy" for "bathroom". "OURS w/o STN" directly applies attention over the opinion hidden states hOi 's, similar to what CMLA does. As shown in Figure 2b, it captures some unrelated opinion words (e.g. "fine") and even some non-opinionated words. As a result, it brings in some noise into the global opinion summary, and consequently the final prediction accuracy will be affected. This example demonstrates that the proposed STN works pretty well to help attend to more related opinion words given a particular aspect.

Some predictions of our model and those of LSTM and OURS w/o THA & STN are given in Table 3. The models incorporating attention-based opinion summary (i.e., OURS and OURS w/o THA & STN) can better determine if the commonly-used nouns are aspect terms or not (e.g. "device" in the first input), since they make decisions based on the global opinion information. Besides, they are able to extract some infrequent or even misspelled aspect terms (e.g.

"survice" in the second input) based on the indicative clues provided by opinion words. For the last three cases, having aspects in coordinate structures (i.e. the third and the fourth) or long aspects (i.e. the fifth), our model can give precise predictions owing to the previous detection clues captured by THA. Without using these clues, the baseline models fail.

4 RelatedWork

Some initial works [Hu and Liu, 2004] developed a bootstrapping framework for tackling Aspect Term Extraction (ATE) based on the observation that opinion words are usually located around the aspects. [Popescu and Etzioni, 2005] and [Qiu et al., 2011] performed co-extraction of aspect terms and opinion words based on sophisticated syntactic patterns. However, relying on syntactic patterns suffers from parsing errors when processing informal online reviews. To avoid this drawback, [Liu et al., 2012; Liu et al., 2013] employed word-based translation models. Specifically, these models formulated the ATE task as a monolingual word alignment process and aspect-opinion relation is captured by alignment links rather than word dependencies. The ATE task can also be formulated as a token-level sequence labeling problem. The winning systems [Chernyshevich, 2014; San Vicente et al., 2015; Toh and Su, 2016] of SemEval ABSA challenges employed traditional sequence models, such as Conditional Random Fields (CRFs) and Maximum Entropy (ME), to detect aspects. Besides heavy feature engineering, they also ignored the consideration of opinions.

Recently, neural network based models, such as LSTMbased [Liu et al., 2015] and CNN-based [Poria et al., 2016] methods, become the mainstream approach. Later on, some neural models jointly extracting aspect and opinion were proposed. [Wang et al., 2016] performs the two task in a single Tree-Based Recursive Neural Network. Their network structure depends on dependency parsing, which is prone to error on informal reviews. CMLA [Wang et al., 2017] consists of multiple attention layers on top of standard GRUs to extract the aspects and opinion words. Similarly, MIN [Li and Lam, 2017] employs multiple LSTMs to interactively perform aspect term extraction and opinion word extraction in a multitask learning framework. Our framework is different from them in two perspectives: (1) It filters the opinion summary by incorporating the aspect features at each time step into the original opinion representations; (2) It exploits history information of aspect detection to capture the coordinate structures and previous aspect features.

5 Concluding Discussions

For more accurate aspect term extraction, we explored two important types of information, namely aspect detection history, and opinion summary. We design two components, i.e. truncated history attention, and selective transformation network. Experimental results show that our model dominates those joint extraction works such as RNCRF and CMLA on the performance of ATE. It suggests that the joint extraction sacrifices the accuracy of aspect prediction, although the ground-truth opinion words were annotated by these authors.

Moreover, one should notice that those joint extraction methods do not care about the correspondence between the extracted aspect terms and opinion words. Therefore, the necessity of such joint extraction should be obelized, given the experimental findings in this paper.